ots3

Documentation

Overview

Package ots3 provides a s3 uploader with opentracing capabilities.

Introduction

S3 is the de facto standard for cloud file systems.The transport of S3 is HTTP(s) which is pleasantly simple to trace. This package also features a go kit server and client for the upload service.

Simple Usage

Creating a s3 manager:

var manager = NewManager(accessKey, accessSecret, endpoint, region, bucket)
url, err := manager.Upload(context.Background(), "myfile", file)

Integration

Package ots3 exports the following configuration:

s3:
  default:
    accessKey:
    accessSecret:
    bucket:
    endpoint:
    region:
    cdnUrl:

To use package ots3 with package core:

var c *core.C = core.New()
c.Provide(ots3.Providers())
c.AddModuleFunc(ots3.New)
c.Invoke(function(manager *ots3.Manager) {
	// do something with manager
})

Adding the module created by mods3.New is optional. This module provides an "/upload" path for the http router. If this is not relevant, just leave it out.

Sometimes there are valid reasons to connect to more than one s3 server. Inject mods3.Maker to factory a *ots3.Manager with a specific configuration entry.

c.Invoke(function(maker ots3.Maker) {
	manager, err := maker.Make("default")
})

Future scope

Currently this package only focus on the file upload aspect of s3. Other s3 features can be incrementally implemented.

Index

• func Providers() []interface{}
• type Config
• type Factory
  • func (s Factory) Make(name string) (*Manager, error)
• type Maker
• type Manager
  • func NewManager(accessKey, accessSecret, endpoint, region, bucket string, opts ...Option) *Manager
  • func (m *Manager) CreateBucket(ctx context.Context, name string) error
  • func (m *Manager) Upload(ctx context.Context, name string, reader io.Reader) (newUrl string, err error)
  • func (m *Manager) UploadFromUrl(ctx context.Context, url string) (newUrl string, err error)
• type Option
  • func WithHttpClient(client contract.HttpDoer) Option
  • func WithKeyer(keyer contract.Keyer) Option
  • func WithLocationFunc(f func(location string) (url string)) Option
  • func WithPathPrefix(pathPrefix string) Option
  • func WithTracer(tracer opentracing.Tracer) Option
• type S3Config
• type Uploader

Functions

func Providers

func Providers() []interface{}

Providers returns a set of dependencies providers related to S3. It includes the s3 Manager, the Maker and exported configurations.

Depends On:
	log.Logger
	contract.ConfigAccessor
	opentracing.Tracer `optional:"true"`
Provide:
	Factory
	Maker
	*Manager
	Uploader

Types

type Config

type Config struct {
	// contains filtered or unexported fields
}

Config contains a various of configurations for Manager. It is mean to be modified by Option.

type Factory

type Factory struct {
	*di.Factory
}

Factory can be used to connect to multiple s3 servers.

func (Factory) Make

func (s Factory) Make(name string) (*Manager, error)

Make creates a s3 manager under the given name.

type Maker

type Maker interface {
	Make(name string) (*Manager, error)
}

Maker is an interface for *Factory. Used as a type hint for injection.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages S3 uploads.

func NewManager

func NewManager(accessKey, accessSecret, endpoint, region, bucket string, opts ...Option) *Manager

NewManager creates a new S3 manager

func (*Manager) CreateBucket

func (m *Manager) CreateBucket(ctx context.Context, name string) error

CreateBucket create a buckets in s3 server. TODO: handle acl

func (*Manager) Upload

func (m *Manager) Upload(ctx context.Context, name string, reader io.Reader) (newUrl string, err error)

Upload uploads an io.reader to the S3 server, and returns the url on S3. The extension of the uploaded file is auto detected.

func (*Manager) UploadFromUrl

func (m *Manager) UploadFromUrl(ctx context.Context, url string) (newUrl string, err error)

UploadFromUrl fetches a file from an external url, copy them to the S3 server, and generate a new, local url. It uses streams to relay files (instead of buffering the entire file in memory). it gives the file a random name using the global seed.

type Option

type Option func(*Config)

Option is the type of functional options to alter Config.

func WithHttpClient

func WithHttpClient(client contract.HttpDoer) Option

WithHttpClient is an option that replaces the default http client. Useful for interceptors like tracing and metrics.

func WithKeyer

func WithKeyer(keyer contract.Keyer) Option

WithKeyer is an option that changes the path of the uploaded file.

func WithLocationFunc

func WithLocationFunc(f func(location string) (url string)) Option

WithLocationFunc is an option that decides the how url is mapped to S3 bucket and path. Useful when not serving file directly from S3, but from a CDN.

func WithPathPrefix

func WithPathPrefix(pathPrefix string) Option

WithPathPrefix is an option that changes the path prefix of uploaded file.

func WithTracer

func WithTracer(tracer opentracing.Tracer) Option

WithTracer is an option that add opentracing.Tracer via the hook of S3 client.

type S3Config

type S3Config struct {
	AccessKey    string `json:"accessKey" yaml:"accessKey"`
	AccessSecret string `json:"accessSecret" yaml:"accessSecret"`
	Endpoint     string `json:"endpoint" yaml:"endpoint"`
	Region       string `json:"region" yaml:"region"`
	Bucket       string `json:"bucket" yaml:"bucket"`
	CdnUrl       string `json:"cdnUrl" yaml:"cdnUrl"`
}

S3Config contains credentials of S3 server

type Uploader

type Uploader interface {
	// Upload the bytes from io.Reader with a given filename to a server, and returns the url and error.
	Upload(ctx context.Context, name string, reader io.Reader) (string, error)
}

Uploader models UploadService