uploader

Documentation

Index

• type Backend
  • func NewCopyBackend(directory string) Backend
  • func NewNoopBackend() Backend
  • func NewS3Backend(m metrics.Reporter, bucket, prefix, taskRole, taskID, writerRole string, ...) (Backend, error)
  • func NewS3BackendWithSession(s *session.Session, m metrics.Reporter, bucket, prefix string) Backend
• type Config
• type ContentTypeInferenceFunction
• type CopyBackend
  • func (u *CopyBackend) Upload(ctx context.Context, local, remote string, ...) error
  • func (u *CopyBackend) UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, ...) error
• type NoopBackend
  • func (u *NoopBackend) Upload(ctx context.Context, local, remote string, ...) error
  • func (u *NoopBackend) UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, ...) error
• type S3Backend
  • func (u *S3Backend) Upload(ctx context.Context, local string, remote string, ...) error
  • func (u *S3Backend) UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, ...) error
• type Uploader
  • func NewUploader(config *config.Config, uploaderConfig *Config, iamRole string, taskID string, ...) (*Uploader, error)
  • func NewUploaderWithBackend(backend Backend) *Uploader
  • func (e *Uploader) Upload(ctx context.Context, local, remote string, ...) error
  • func (e *Uploader) UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, ...) error

Types

type Backend

type Backend interface {
	Upload(ctx context.Context, local, remote string, ctypeFunc ContentTypeInferenceFunction) error
	UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, remote, contentType string) error
}

Backend is a common interface for a service that can upload log files somewhere

func NewCopyBackend

func NewCopyBackend(directory string) Backend

NewCopyBackend creates a new instance of a copy uploader

func NewNoopBackend

func NewNoopBackend() Backend

NewNoopBackend creates a new instance of a NoopBackend

func NewS3Backend

func NewS3Backend(m metrics.Reporter, bucket, prefix, taskRole, taskID, writerRole string, useDefaultRole bool) (Backend, error)

NewS3Backend creates a new instance of an S3 manager which uploads to the specified location.

func NewS3BackendWithSession

func NewS3BackendWithSession(s *session.Session, m metrics.Reporter, bucket, prefix string) Backend

NewS3BackendWithSession creates a new instance of an S3 manager to upload to a given location given a configured session.

type Config

type Config struct {
	DisableUpload bool
	S3WriterRole  string
	S3BucketName  string
	S3PathPrefix  string
}

Config specifies the config for the uploader

type ContentTypeInferenceFunction

type ContentTypeInferenceFunction func(filename string) string

ContentTypeInferenceFunction is the callback that can be used to set the mime type of a file at upload time

type CopyBackend

type CopyBackend struct {
	Dir string `json:"directory"`
}

CopyBackend is an uploader that just copies files to another location on the same host

func (*CopyBackend) Upload

func (u *CopyBackend) Upload(ctx context.Context, local, remote string, ctypeFunc ContentTypeInferenceFunction) error

Upload copies a single file only!

func (*CopyBackend) UploadPartOfFile

func (u *CopyBackend) UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, remote, contentType string) error

UploadPartOfFile copies a single file only. It doesn't preserve the cursor location in the file.

type NoopBackend

type NoopBackend struct{}

NoopBackend is an uploader to be used when testing

func (*NoopBackend) Upload

func (u *NoopBackend) Upload(ctx context.Context, local, remote string, ctypeFunc ContentTypeInferenceFunction) error

Upload does nothing (i.e., noop)

func (*NoopBackend) UploadPartOfFile

func (u *NoopBackend) UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, remote, contentType string) error

UploadPartOfFile does nothing (i.e., noop)

type S3Backend

type S3Backend struct {
	// contains filtered or unexported fields
}

S3Backend uploads logs to S3

func (*S3Backend) Upload

func (u *S3Backend) Upload(ctx context.Context, local string, remote string, ctypeFunc ContentTypeInferenceFunction) error

func (*S3Backend) UploadPartOfFile

func (u *S3Backend) UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, remote, contentType string) error

UploadPartOfFile copies a single file only. It doesn't preserve the cursor location in the file.

type Uploader

type Uploader struct {
	// contains filtered or unexported fields
}

Wraps the backends used to do log uploading.

func NewUploader

func NewUploader(config *config.Config, uploaderConfig *Config, iamRole string, taskID string, m metrics.Reporter) (*Uploader, error)

The upload always prefers s3, then copy, and will use a black hole sink if nothing else is configured. The first s3 location or copy destination specified is used. Rest are ignored.

func NewUploaderWithBackend

func NewUploaderWithBackend(backend Backend) *Uploader

func (*Uploader) Upload

func (e *Uploader) Upload(ctx context.Context, local, remote string, ctypeFunc ContentTypeInferenceFunction) error

Upload uploads the file, or all the files in a directory to the remote location.

func (*Uploader) UploadPartOfFile

func (e *Uploader) UploadPartOfFile(ctx context.Context, local io.ReadSeeker, start, length int64, remote, contentType string) error

UploadPartOfFile logs the call and forwards to the backend. It can upload a subset of a file. Offsets are not preserved.