archiver

package
v0.0.0-...-f43e618 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 15, 2024 License: MIT Imports: 15 Imported by: 0

Documentation

Index

Constants

View Source
const (
	FileExtJsonlGzip       = ".jsonl.gz"
	FileExtParquet         = ".parquet"
	LocalTempDirPattern    = "penguin_stats-archiver-*"
	ArchiverChanBufferSize = 1000
)

Variables

View Source
var ErrFileAlreadyExists = errors.New("file already exists")

Functions

This section is empty.

Types

type Archiver

type Archiver struct {
	S3Client *s3.Client
	S3Bucket string

	// S3Prefix is for the files in the bucket with no leading slash but optionally (typically) with trailing slash
	// e.g. "v1/" or simply "" (empty string)
	S3Prefix string

	RealmName string
	// contains filtered or unexported fields
}

func (*Archiver) Cleanup

func (a *Archiver) Cleanup() error

func (*Archiver) Collect

func (a *Archiver) Collect(ctx context.Context) error

Caller MUST use WriterCh() to get the channel and ensure necessary data is sent to the channel before calling this function. Moreover, caller should ensure that Collect runs only once and runs on a different goroutine from the one that sends data to the channel to avoid deadlocks.

func (*Archiver) Prepare

func (a *Archiver) Prepare(ctx context.Context, date time.Time) error

func (*Archiver) WriterCh

func (a *Archiver) WriterCh() chan interface{}

Caller MUST close the channel when it's done

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL