objects

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2024 License: Apache-2.0 Imports: 20 Imported by: 0

README

Objects

Objects is a package designed to simplify the process of managing receipt of multipart/form-data requests and subsequently the uploading of files.

Usage

Assuming you have a HTML form like this:

<form action="/" method="post" enctype="multipart/form-data">
  <input type="file" name="form-field-1" />
  <input type="file" name="form-field-2" />
</form>

To create a new objects instance, you can do something like this:

objectManager, _ := objects.New(
 objects.WithMaxFileSize(10<<20),
 objects.WithKeys([]string{"form-field-1", "form-field-2"}),
 objects.WithUploaderFunc(
   func(ctx context.Context, u *Objects, files []FileUpload) ([]File, error) {
    // add your own custom uploader functionality here
    // or leave out to use the default uploader func
   }
 ),
 objects.WithValidationFunc(
  objects.ChainValidators(objects.MimeTypeValidator("image/jpeg", "image/png"),
   func(f objects.File) error {
    // Your own custom validation function on the file here
    return nil
   })),
 objects.WithStorage(s3Store),
)

The objectManager can be used with the provided middleware FileUploadMiddleware(objectManager) and added to the chain of middleware within the server. This is a generic http.Handler so it can be used with your router of choice. For example, to be used with echo:

echo.WrapMiddleware(objects.FileUploadMiddleware(objectManager))
Standard HTTP router
package main

import (
	"fmt"
	"net/http"

	"github.com/aws/aws-sdk-go-v2/aws"
	awsCreds "github.com/aws/aws-sdk-go-v2/credentials"

	"github.com/theopenlane/core/pkg/objects"
	"github.com/theopenlane/core/pkg/objects/storage"
)

func main() {
	s3Store, err := storage.NewS3FromConfig(aws.Config{
		Region: "us-west-2",
		Credentials: awsCreds.NewStaticCredentialsProvider(
			"accessKey", "secretKey", ""),
	}, storage.S3Options{
		Bucket: "std-router",
	})
	if err != nil {
		panic(err.Error())
	}

	objectsHandler, err := objects.New(
		objects.WithMaxFileSize(10<<20),
		objects.WithStorage(s3Store),
		objects.WithKeys([]string{"name", "mitb"}),
	)

	mux := http.NewServeMux()

	// upload all files with the "name" and "mitb" fields on this route
	mux.Handle("/", objects.FileUploadMiddleware(objectsHandler)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Println("Uploading file")

		ctx := r.Context()

		// return all uploaded files
		f, err := objects.FilesFromContext(ctx)
		if err != nil {
			fmt.Println(err)
			return
		}

		// return uploaded files with the form field "mitb"
		ff, err := objects.FilesFromContextWithKey(ctx, "mitb")
		if err != nil {
			fmt.Println(err)
			return
		}

		fmt.Printf("%+v", ff)

		for _, v := range f {
			fmt.Printf("%+v", v)
			fmt.Println()
		}
	})))

	http.ListenAndServe(":3300", mux)
}
Ignoring non existent keys in the multipart Request

Sometimes, the keys you have configured the middleware might get dropped from the frontend for some reason, ideally the middleware fails if it cannot find a configured key in the request. To disable this behavior and ignore the missing key, you can make use of the WithIgnoreNonExistentKey(true) option to prevent the middleware from causing an error when such keys do not exists

Customizing the error response

Since Objects could be used as a middleware, it returns an error to the client if found, this might not match your existing structure, so to configure the response, use the WithErrorResponseHandler. The default is shown below and can be used as a template to define yours.

var errHandler objects.ErrResponseHandler = func(err error, statusCode int) http.HandlerFunc {
  return func(w http.ResponseWriter, _ *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(statusCode)
    fmt.Fprintf(w, `{"message" : "could not upload file", "error" : "%s"}`, err.Error())
  }
}

// add the following to the objects options: 
// objects.WithErrorResponseHandler(errHandler)
Writing your own custom upload logic

The uploader function by default will just upload the file to the storage backend. In some cases you may want custom logic, e.g. update your local database with information. A custom UploaderFunc can be used to do this.

var uploader objects.UploaderFunc = func(ctx context.Context, u *objects.Objects, files []objects.FileUpload) ([]objects.File, error) {
	uploadedFiles := make([]objects.File, 0, len(files))

	for _, f := range files {
		// do things
	}

	return uploadedFiles, nil
}

// add the following to the objects options: 
// objects.WithUploaderFunc(uploader)
Writing your custom validator logic

Sometimes, you could have some custom logic to validate uploads, in this example below, we limit the size of the upload based on the mimeypes of the uploaded files

var customValidator objects.ValidationFunc = func(f objects.File) error {
 switch f.MimeType {
 case "image/png":
  if f.Size > 4096 {
   return errors.New("file size too large")
  }

  return nil

 case "application/pdf":
  if f.Size > (1024 * 10) {
   return errors.New("file size too large")
  }

  return nil
 default:
  return nil
 }
}

// add the following to the objects options: 
// objects.WithValidationFunc(customValidator)

Documentation

Overview

Package objects provides interfaces and helpers for management of objects (files) either received via our endpoints or created and stored by the system

Index

Constants

View Source 
const (
	ErrNoFilesUploaded = errorMsg("objects: no uploadable files found in request")
)

Variables

View Source 
var (
	// ErrFilesNotFound is returned when files could not be found in key from http request
	ErrFilesNotFound = errors.New("files could not be found in key from http request")
	// ErrFileOpenFailed is returned when a file could not be opened
	ErrFileOpenFailed = errors.New("could not open file")
	// ErrInvalidMimeType is returned when a file has an invalid mime type
	ErrInvalidMimeType = errors.New("invalid mimetype")
	// ErrValidationFailed is returned when a validation fails
	ErrValidationFailed = errors.New("validation failed")
	// ErrUnsupportedMimeType is returned when a file has an unsupported mime type
	ErrUnsupportedMimeType = errors.New("unsupported mime type uploaded")
	// ErrMustProvideStorageBackend is returned when a storage backend is not provided
	ErrMustProvideStorageBackend = errors.New("you must provide a storage backend")
	// ErrUnexpectedType is returned when an invalid type is provided
	ErrUnexpectedType = errors.New("unexpected type provided")
)
View Source 
var FileContextKey = &ContextKey{"files"}

FileContextKey is the context key for the files This is the key that is used to store the files in the context, which is then used to retrieve the files in subsequent parts of the request this is different than the `key` in the multipart form, which is the form field name that the file was uploaded with

Functions

func ComputeChecksum 

func ComputeChecksum(data io.ReadSeeker) (string, error)

ComputeChecksum calculates the MD5 checksum for the provided data. It expects that the passed io object will be seeked to its beginning and will seek back to the beginning after reading its content.

func CreateURI 

func CreateURI(scheme, destination, key string) string

CreateURI creates a URI for the file based on the scheme, destination and key

func DetectContentType 

func DetectContentType(data io.ReadSeeker) (string, error)

DetectContentType leverages http.DetectContentType to identify the content type of the provided data. It expects that the passed io object will be seeked to its beginning and will seek back to the beginning after reading its content.

func FileUploadMiddleware 

func FileUploadMiddleware(config *Objects) func(http.Handler) http.Handler

FileUploadMiddleware is a middleware that handles the file upload process this can be added to the middleware chain to handle file uploads prior to the main handler Since gqlgen handles file uploads differently, this middleware is not used in the graphql handler

func FormatFileSize 

func FormatFileSize(size int64) string

FormatFileSize converts a file size in bytes to a human-readable string in MB/GB notation.

func GetFileIDsFromContext 

func GetFileIDsFromContext(ctx context.Context) []string

GetFileIDsFromContext returns the file IDs from the context that are associated with the request

func ReaderToSeeker 

func ReaderToSeeker(r io.Reader) (io.ReadSeeker, error)

ReaderToSeeker function takes an io.Reader as input and returns an io.ReadSeeker which can be used to upload files to the object storage

func StreamToByte 

func StreamToByte(stream io.ReadSeeker) ([]byte, error)

StreamToByte function reads the content of the provided io.Reader and returns it as a byte slice

func WriteFilesToContext 

func WriteFilesToContext(ctx context.Context, f Files) context.Context

WriteFilesToContext retrieves any existing files from the context, appends the new files to the existing files map based on the form field name, then returns a new context with the updated files map stored in it

Types

type AppValidator 

type AppValidator struct {
	// contains filtered or unexported fields
}

AppValidator is a wrapper around the validator package

func NewValidator 

func NewValidator() *AppValidator

NewValidator creates a new instance of the AppValidator

func (*AppValidator) Validate 

func (cv *AppValidator) Validate(i interface{}) error

Validate runs the validation against the provided struct

type Config 

type Config struct {
	// Enabled indicates if the store is enabled
	Enabled bool `json:"enabled" koanf:"enabled" default:"true"`
	// Provider is the name of the provider
	Provider string `json:"provider" koanf:"provider"`
	// AccessKey is the access key for the storage provider
	AccessKey string `json:"accessKey" koanf:"accessKey"`
	// Region is the region for the storage provider
	Region string `json:"region" koanf:"region"`
	// SecretKey is the secret key for the storage provider
	SecretKey string `json:"secretKey" koanf:"secretKey"`
	// CredentialsJSON is the credentials JSON for the storage provider
	CredentialsJSON string `json:"credentialsJSON" koanf:"credentialsJSON"`
	// DefaultBucket is the default bucket name for the storage provider
	DefaultBucket string `json:"defaultBucket" koanf:"defaultBucket"`
	// Keys is a list of keys to look for in the multipart form on the REST request
	// if the keys are not found, the request upload will be skipped
	// this is not used by the graphql handler
	Keys []string `json:"keys" koanf:"keys" default:"[uploadFile]"`
}

type ContextKey 

type ContextKey struct {
	// contains filtered or unexported fields
}

ContextKey is the key name for the additional context

type DownloadFileMetadata 

type DownloadFileMetadata struct {
	// FolderDestination is the folder that holds the file
	FolderDestination string `json:"folder_destination,omitempty"`
	// Key is the unique identifier for the file
	Key string `json:"key,omitempty"`
	// Size in bytes of the downloaded file
	Size int64 `json:"size,omitempty"`
}

DownloadFileMetadata is a struct that holds information about a file that was successfully downloaded

type DownloadFileOptions 

type DownloadFileOptions struct {
	FileName string
}

DownloadFileOptions is a struct that holds the options for downloading a file

type ErrResponseHandler 

type ErrResponseHandler func(err error, statusCode int) http.HandlerFunc

ErrResponseHandler is a custom error that should be used to handle errors when an upload fails

type File 

type File struct {
	// ID is the unique identifier for the file
	ID string `json:"id"`
	// Name of the file
	Name string `json:"name"`
	// Path of the file
	Path string `json:"path"`
	// Type of file that was uploaded
	Type string `json:"type"`
	// Thumbnail is a URL to the thumbnail of the file
	Thumbnail *string `json:"thumbnail"`
	// MD5 hash of the file
	MD5 []byte `json:"md5"`
	// CreatedAt is the time the file was created
	CreatedAt time.Time `json:"created_at"`
	// UpdatedAt is the time the file was last updated
	UpdatedAt time.Time `json:"updated_at"`
	// OwnerID is the ID of the organization or user who created the file
	OwnerID string `json:"owner_id"`
	// FieldName denotes the field from the multipart form
	FieldName string `json:"field_name,omitempty"`

	// OriginalName of the file from the client side
	OriginalName string `json:"original_name,omitempty"`
	// UploadedFileName denotes the name of the file when it was ultimately
	// uploaded to the storage layer. The distinction is important because of
	// potential changes to the file name that may be done
	UploadedFileName string `json:"uploaded_file_name,omitempty"`
	// FolderDestination is the folder that holds the uploaded file
	FolderDestination string `json:"folder_destination,omitempty"`

	// StorageKey can be used to retrieve the file from the storage backend
	StorageKey string `json:"storage_key,omitempty"`

	// MimeType of the uploaded file
	MimeType string `json:"mime_type,omitempty"`

	// ContentType is the detected content type of the file
	ContentType string `json:"content_type,omitempty"`

	// Size in bytes of the uploaded file
	Size              int64 `json:"size,omitempty"`
	Metadata          map[string]string
	Bucket            string
	PresignedURL      string `json:"url"`
	ProvidedExtension string `json:"provided_extension"`
}

File is a struct that holds information about a file - there is no distinction between a File received in a multipart form request or used in a download

func FilesFromContextWithKey 

func FilesFromContextWithKey(ctx context.Context, key string) ([]File, error)

FilesFromContextWithKey returns all files that have been uploaded during the request and sorts by the provided form field

type FileUpload 

type FileUpload struct {
	// File is the file to be uploaded
	File io.ReadSeeker
	// Filename is the name of the file provided in the multipart form
	Filename string
	// Size is the size of the file in bytes
	Size int64
	// ContentType is the content type of the file from the header
	ContentType string
	// Key is the field name from the graph input or multipart form
	Key string
}

FileUpload is the object that holds the file information

func NewUploadFile 

func NewUploadFile(filePath string) (file FileUpload, err error)

NewUploadFile function reads the content of the provided file path and returns a FileUpload object

type Files 

type Files map[string][]File

Files is a map of field names to a slice of files

func FilesFromContext 

func FilesFromContext(ctx context.Context) (Files, error)

FilesFromContext returns all files that have been uploaded during the request

type NameGeneratorFunc 

type NameGeneratorFunc func(s string) string

NameGeneratorFunc allows you alter the name of the file before it is ultimately uploaded and stored 

var OrganizationNameFunc NameGeneratorFunc = func(s string) string {
	return s
}

type Objects 

type Objects struct {
	// Storage is the storage backend that will be used to store the uploaded files
	Storage Storage `json:"-" koanf:"-"`
	// MaxSize is the maximum size of file uploads to accept
	MaxSize int64 `json:"maxSize" koanf:"maxSize"`
	// MaxMemory is the maximum memory to use when parsing a multipart form
	MaxMemory int64 `json:"maxMemory" koanf:"maxMemory"`
	// IgnoreNonExistentKeys is a flag that indicates the handler should skip multipart form key values which do not match the configured
	IgnoreNonExistentKeys bool `json:"ignoreNonExistentKeys" koanf:"ignoreNonExistentKeys"`
	// Keys is a list of keys to look for in the multipart form on the REST request
	// if the keys are not found, the request upload will be skipped
	// this is not used by the graphql handler
	Keys []string `json:"keys" koanf:"keys" default:"[uploadFile]"`
	// ValidationFunc is a custom validation function
	ValidationFunc ValidationFunc `json:"-" koanf:"-"`
	// NameFuncGenerator is a function that allows you to rename your uploaded files
	NameFuncGenerator NameGeneratorFunc `json:"-" koanf:"-"`
	// Uploader is the func that handlers the file upload process and returns the files uploaded
	Uploader UploaderFunc `json:"-" koanf:"-"`
	// Skipper defines a function to skip middleware.
	Skipper SkipperFunc `json:"-" koanf:"-"`
	// ErrorResponseHandler is a custom error response handler
	ErrorResponseHandler ErrResponseHandler `json:"-" koanf:"-"`
}

Objects is the definition for handling objects and file uploads

func New 

func New(opts ...Option) (*Objects, error)

New creates a new instance of Objects

func (*Objects) FileUpload 

func (u *Objects) FileUpload(ctx context.Context, files []FileUpload) (context.Context, error)

FileUpload uploads the files to the storage and returns the the context with the uploaded files

type Option 

type Option func(*Objects)

Option is a function that configures the Objects

func WithErrorResponseHandler 

func WithErrorResponseHandler(errHandler ErrResponseHandler) Option

WithErrorResponseHandler allows you to provide a custom error response handler

func WithIgnoreNonExistentKey 

func WithIgnoreNonExistentKey(ignore bool) Option

WithIgnoreNonExistentKey allows you to configure the handler to skip multipart form key values which do not match the configured

func WithKeys 

func WithKeys(keys []string) Option

WithKeys allows you to configure the keys to look for in the multipart form on the REST request

func WithMaxFileSize 

func WithMaxFileSize(i int64) Option

WithMaxFileSize allows you limit the size of file uploads to accept

func WithMaxMemory 

func WithMaxMemory(i int64) Option

WithMaxMemory allows you limit the amount of memory to use when parsing a multipart form

func WithNameFuncGenerator 

func WithNameFuncGenerator(nameFunc NameGeneratorFunc) Option

WithNameFuncGenerator allows you configure how you'd like to rename your uploaded files

func WithSkipper 

func WithSkipper(skipper SkipperFunc) Option

WithSkipper allows you to provide a custom skipper function

func WithStorage 

func WithStorage(store Storage) Option

WithStorage allows you to provide a storage backend to the Objects

func WithUploaderFunc 

func WithUploaderFunc(uploader UploaderFunc) Option

WithUploaderFunc allows you to provide a custom uploader function

func WithValidationFunc 

func WithValidationFunc(validationFunc ValidationFunc) Option

WithValidationFunc allows you to provide a custom validation function

type Progress 

type Progress struct {
	// TotalSize is the total size of the file being uploaded
	TotalSize int64
	// BytesRead is the number of bytes that have been read so far
	BytesRead int64
}

Progress is used to track the progress of a file upload It implements the io.Writer interface so it can be passed to an io.TeeReader()

func (*Progress) Print 

func (pr *Progress) Print()

Print displays the current progress of the file upload

func (*Progress) Write 

func (pr *Progress) Write(p []byte) (n int, err error)

Write is used to satisfy the io.Writer interface Instead of writing somewhere, it simply aggregates the total bytes on each read

type SkipperFunc 

type SkipperFunc func(r *http.Request) bool

SKipperFunc is a function that defines whether to skip the middleware

type Storage 

type Storage interface {
	// Upload is used to upload a file to the storage backend
	Upload(context.Context, io.Reader, *UploadFileOptions) (*UploadedFileMetadata, error)
	// ManagerUpload is used to upload multiple files to the storage backend
	ManagerUpload(context.Context, [][]byte) error
	// Download is used to download a file from the storage backend
	Download(context.Context, string, *DownloadFileOptions) (*DownloadFileMetadata, io.ReadCloser, error)
	// GetPresignedURL is used to get a presigned URL for a file in the storage backend
	GetPresignedURL(context.Context, string) (string, error)
	// GetScheme returns the scheme of the storage backend
	GetScheme() *string
	io.Closer
}

Storage is the primary interface that must be implemented by any storage backend and for interacting with Objects

type UploadFileOptions 

type UploadFileOptions struct {
	FileName              string
	Metadata              map[string]string
	Progress              *pb.ProgressBar
	ProgressOutput        io.Writer
	ProgressFinishMessage string
	Bucket                string
	ContentType           string
}

UploadFileOptions is a struct that holds the options for uploading a file

type UploadOption 

type UploadOption func(*UploadFileOptions)

UploadOption is a function that configures the UploadFileOptions

func UploadMetadata 

func UploadMetadata(mp map[string]interface{}) UploadOption

UploadMetadata allows you to provide metadata for the upload

func UploadProgress 

func UploadProgress(p *pb.ProgressBar) UploadOption

UploadProgress allows you to provide a progress bar for the upload

func UploadProgressFinishMessage 

func UploadProgressFinishMessage(s string) UploadOption

UploadProgressFinishMessage allows you to provide a message to display when the upload is complete

func UploadProgressOutput 

func UploadProgressOutput(out io.Writer) UploadOption

UploadProgressOutput allows you to provide a writer for the progress bar

type UploadedFileMetadata 

type UploadedFileMetadata struct {
	// FolderDestination is the folder that holds the file
	FolderDestination string `json:"folder_destination,omitempty"`
	// Key is the unique identifier for the file
	Key string `json:"key,omitempty"`
	// Size in bytes of the uploaded file
	Size int64 `json:"size,omitempty"`
	// PresignedURL is the URL that can be used to download the file
	PresignedURL string `json:"presigned_url,omitempty"`
}

UploadedFileMetadata is a struct that holds information about a file that was successfully uploaded

type UploaderFunc 

type UploaderFunc func(ctx context.Context, u *Objects, files []FileUpload) ([]File, error)

UploaderFunc is a function that handles the file upload process and returns the files uploaded

type ValidationFunc 

type ValidationFunc func(f File) error

ValidationFunc is a type that can be used to dynamically validate a file

func ChainValidators 

func ChainValidators(validators ...ValidationFunc) ValidationFunc

ChainValidators returns a validator that accepts multiple validating criteria

func MimeTypeValidator 

func MimeTypeValidator(validMimeTypes ...string) ValidationFunc

MimeTypeValidator makes sure we only accept a valid mimetype. It takes in an array of supported mimes

Source Files

View all Source files

Directories

Path Synopsis
Package storage provides basic storage interfaces for storage providers to write / read objects to and from
 Package storage provides basic storage interfaces for storage providers to write / read objects to and from

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.