s3

Documentation

Overview

Package s3 provides types and methods for consuming an s3 message archive

Index

• type Archive
  • func NewArchive(c *ArchiveConfig) (*Archive, error)
  • func (a *Archive) Scan() chan *replay.Object
• type ArchiveConfig
  • func NewArchiveConfig() *ArchiveConfig
• type Downloader

Types

type Archive

type Archive struct {
	// contains filtered or unexported fields
}

Archive implements an s3 archive that utilizes an s3 download manager to download objects concurrently

func NewArchive

func NewArchive(c *ArchiveConfig) (*Archive, error)

NewArchive returns a new s3 archive

func (*Archive) Scan

func (a *Archive) Scan() chan *replay.Object

Scan implements logic for scanning some or all of an s3 message archive, downloading archive objects concurrently, and emitting them the returned channel

type ArchiveConfig

type ArchiveConfig struct {
	// The S3 bucket that contains the archive
	Bucket string `validate:"required"`
	// A configured s3 client
	Client s3iface.S3API `validate:"required"`
	// Number of objects to download in parallel
	Concurrency int `validate:"required,min=1"`
	// A configured s3 download manager
	Downloader Downloader `validate:"required"`
	// An optional archive scoped logger
	Log logrus.FieldLogger `valdiate:"required"`
	// An optional prefix that contains the relevant archive portion
	Prefix string `validate:"-"`
	// An optional s3 key to begin replay after
	StartAfter string `validate:"-"`
	// An optional s3 key to stop replay at
	StopAt string `validate:"-"`
}

ArchiveConfig defines an archive configuration

func NewArchiveConfig

func NewArchiveConfig() *ArchiveConfig

NewArchiveConfig returns an archive config value with appropriate defaults

type Downloader

type Downloader interface {
	Download(io.WriterAt, *s3.GetObjectInput, ...func(*s3manager.Downloader)) (int64, error)
}

Downloader represents an s3 download manager