archiver

Documentation

Index

• Constants
• Variables
• type Archiver
  • func (a *Archiver) Cleanup() error
  • func (a *Archiver) Collect(ctx context.Context) error
  • func (a *Archiver) Prepare(ctx context.Context, date time.Time) error
  • func (a *Archiver) WriterCh() chan interface{}

Constants

const (
	FileExtJsonlGzip       = ".jsonl.gz"
	FileExtParquet         = ".parquet"
	LocalTempDirPattern    = "penguin_stats-archiver-*"
	ArchiverChanBufferSize = 1000
)

Variables

var ErrFileAlreadyExists = errors.New("file already exists")

Types

type Archiver

type Archiver struct {
	S3Client *s3.Client
	S3Bucket string

	// S3Prefix is for the files in the bucket with no leading slash but optionally (typically) with trailing slash
	// e.g. "v1/" or simply "" (empty string)
	S3Prefix string

	RealmName string
	// contains filtered or unexported fields
}

func (*Archiver) Cleanup

func (a *Archiver) Cleanup() error

func (*Archiver) Collect

func (a *Archiver) Collect(ctx context.Context) error

Caller MUST use WriterCh() to get the channel and ensure necessary data is sent to the channel before calling this function. Moreover, caller should ensure that Collect runs only once and runs on a different goroutine from the one that sends data to the channel to avoid deadlocks.

func (*Archiver) Prepare

func (a *Archiver) Prepare(ctx context.Context, date time.Time) error

func (*Archiver) WriterCh

func (a *Archiver) WriterCh() chan interface{}

Caller MUST close the channel when it's done