mediateq

README

mediateq

mediateq is a file storage REST API microservice that allows users to upload and download files, as well as resize images on the fly.

Prerequisites

• libvips 8.3+
• PostgreSQL 14+

Installation

To install mediateq, clone the project repository:

git clone https://github.com/behouba/mediateq.git

Mediateq depend on libvips.

Install libvips on Debian based Linux distributions:

sudo apt install -y libvips-dev

or run the following script from bimg as sudo (supports OSX, Debian/Ubuntu, Redhat, Fedora, Amazon Linux):

curl -s https://raw.githubusercontent.com/h2non/bimg/master/preinstall.sh | sudo bash -

The install script requires curl and pkg-config

If the above script is not working, please follow libvips installation instructions:

https://libvips.github.io/libvips/install.html

You can then build mediateq by running the build.sh script:

./build.sh

You will also need to setup a database. Currently mediateq only support postgreSQL.

Read the instructions to setup a postgreSQL database here

To run mediateq, use the following command:

./bin/mediateq -config=mediateq.yaml

You can also run tests for mediateq by using the run_tests.sh script:

./run_tests.sh

API specification

The API specification of mediateq can be found here

All endpoints should be prefixed with: /mediateq/{version}

Note: mediateq uses base64 hash of files as filename to avoid duplications.

/info (GET)

Retrieves information about the server.

Example Request

GET /info

Example Response

HTTP/1.1 200 OK
Content-Type: application/json

{
    "version": "v0",
    "domain": "http://localhost:8080",
    "port": 8080,
    "database": "postgres",
    "storage": "localdisk",
    "allowedContentTypes": [
    "image/jpeg",
    "image/png",
    "image/bimg",
    "image/webp"
    ],
    "uptime": 35
}

/upload (POST)

Uploads a new media file to the server.

Example Request

curl -X POST -H "Content-Type: multipart/form-data" -F "file=@herman.jpeg" http://localhost:8080/mediateq/v0/upload

Example Response

Copy code
HTTP/1.1 200 OK
Content-Type: application/json

{
    "media": {
        "id": "2K2dScz76xXsc0D3u29M3w1iKcw",
        "url": "http://localhost:8080/mediateq/v0/download/2K2dScz76xXsc0D3u29M3w1iKcw",
        "origin": "::1",
        "contentType": "image/jpeg",
        "sizeBytes": 103417,
        "tmestamp": 1673176612,
        "base64Hash": "zKZqFTxVdiIWGf-4otBjjBS46aPqs-Q6W0mefJmVhmo"
    }
}

/download/{base64Hash} (GET)

Downloads a media file from the server.

Example Request

GET /download/2K2dScz76xXsc0D3u29M3w1iKcw?width=400
Example Response

Copy code
HTTP/1.1 200 OK
Content-Type: video/mp4
[binary data]

/thumbnail/{base64Hash} (GET)

Downloads thumbnail version of images files.

The width and height of the image are required queries paramters.

Example Request

GET /download/2K2dScz76xXsc0D3u29M3w1iKcw?width=640&height=480
Example Response

Copy code
HTTP/1.1 200 OK
Content-Type: video/mp4
[binary data]

/media (GET)

Retrieves a paginated list of all media files on the server.

Use query parameters limit and offset to paginate the results.

Example Request

GET /media?limit=4&offset=1

Example Response

HTTP/1.1 200 OK
Content-Type: application/json

{
    "mediaList": [
        {
            "id": "2JVvl82araY9MfG9O2keI1dSZDy",
            "url": "http://localhost:8080/mediateq/v0/download/2JVvl82araY9MfG9O2keI1dSZDy",
            "origin": "::1",
            "contentType": "image/jpeg",
            "sizeBytes": 103417,
            "tmestamp": 1672176212,
            "base64Hash": "zKZqFTxVdiIWGf-4otBjjBS46aPqs-Q6W0mefJmVhmo"
        },
        {
            "id": "2K2dScz76xXsc0D3u29M3w1iKcw",
            "url": "http://localhost:8080/mediateq/v0/download/2K2dScz76xXsc0D3u29M3w1iKcw",
            "origin": "::1",
            "contentType": "image/jpeg",
            "sizeBytes": 103417,
            "tmestamp": 1673176612,
            "base64Hash": "zKZqFTxVdiIWGf-4otBjjBS46aPqs-Q6W0mefJmVhmo"
        }
    ]
}

/media/{base64Hash} (GET)

Retrieves information about a specific media file.

Example Request

GET /media/2K2dScz76xXsc0D3u29M3w1iKcw

Example Response

HTTP/1.1 200 OK
Content-Type: application/json

{
    "media": {
        "id": "2K2dScz76xXsc0D3u29M3w1iKcw",
        "url": "http://localhost:8080/mediateq/v0/download/2K2dScz76xXsc0D3u29M3w1iKcw",
        "origin": "::1",
        "contentType": "image/jpeg",
        "sizeBytes": 103417,
        "tmestamp": 1673176612,
        "base64Hash": "zKZqFTxVdiIWGf-4otBjjBS46aPqs-Q6W0mefJmVhmo"
    }
}

/media/{base64Hash} (DELETE)

Delete a specific media file by it base64 hash identifier.

Example Request

DELETE /media/4siOxL16rCSeWvEeGxBAtqMmF04HffW_qg8zWuOh2MY

Example Response

HTTP/1.1 200 OK
Content-Type: application/json

{
    "message": "media 4siOxL16rCSeWvEeGxBAtqMmF04HffW_qg8zWuOh2MY deleted"
}

Docker container

[TODO]

Documentation

Index

• Constants
• type ContentType
• type DBType
• type ImageProcessor
• type Media
  • func (m Media) GetFilePath(uploadPath string) (string, error)
  • func (m Media) IsImage() bool
• type Storage
• type StorageType
• type Thumbnail
• type ThumbnailSize

Constants

const (
	ContentTypeFormData ContentType = "multipart/form-data"

	// Supported content types for images
	ContentTypeJPEG ContentType = "image/jpeg"
	ContentTypePNG  ContentType = "image/png"
	ContentTypeGIF  ContentType = "image/gif"
	ContentTypeBIMP ContentType = "image/bimg"
	ContentTypeWEBP ContentType = "image/webp"

	// Storage type options
	StorageTypeLocalDisk StorageType = "localdisk"
	StorageTypeS3        StorageType = "s3"

	// Database options
	DBTypePostgres DBType = "postgres"
	DBTypeSQLite   DBType = "sqlite"
)

Types

type ContentType

type ContentType string

ContentType represents HTTP MIME types sent in Content-type header

type DBType

type DBType string

DB type represents the type of database used by the server

type ImageProcessor

type ImageProcessor interface {
	Resize(buff []byte, width, height int) ([]byte, error)
	Rotage(buff []byte, degree int) ([]byte, error)
	Grayscale(buff []byte) ([]byte, error)
}

ImageProcessor interface provides image processing methods

type Media

type Media struct {
	ID          string      `json:"id"`          // Numeric id (db primary key)
	URL         string      `json:"url"`         // url to access the file over internet
	Origin      string      `json:"origin"`      // Origin domain of the file
	ContentType ContentType `json:"contentType"` //
	SizeBytes   int64       `json:"sizeBytes"`   // Size of the file in bytes
	Timestamp   int64       `json:"tmestamp"`    // Media creation timestamp
	Base64Hash  string      `json:"base64Hash"`  // Base64 hash of the file used as a unique string identifier
}

Media is a representation of mediateq file.

func (Media) GetFilePath

func (m Media) GetFilePath(uploadPath string) (string, error)

GetFilePath return the path to a media file 2 subdirectories are created for more manageable browsing and use the remainder as the file name. For example, if Base64Hash is 'qwerty' and content type is 'image/png' the path will be 'q/w/erty'.

func (Media) IsImage

func (m Media) IsImage() bool

IsImage check if a media file is an image base on it content type

type Storage

type Storage interface {
	// Write method write file buffer to storage and
	// return the relative path and an error if the write operating fail
	Write(ctx context.Context, buff []byte, filePath string) error
	// Read method read a file from storage and return
	// the relative path and an error if the read operation fail
	Read(ctx context.Context, filePath string) ([]byte, error)
	// Remove method should remove a file from
	// the storage given the path to the target file
	Remove(ctx context.Context, path string) error
}

Storage is an abstration of place where files are stored

type StorageType

type StorageType string

StorageType represents type of storage where media are stored

type Thumbnail

type Thumbnail struct {
	Media
	ThumbnailSize
}

type ThumbnailSize

type ThumbnailSize struct {
	Width  int  `yaml:"width"`
	Height int  `yaml:"height"`
	Crop   bool `yaml:"method"`
}