destination

Documentation

Index

• Constants
• type Config
  • func Parse(cfg map[string]string) (Config, error)
• type Destination
  • func (s *Destination) Open(ctx context.Context, cfg plugins.Config) error
  • func (s *Destination) Teardown() error
  • func (s *Destination) Validate(cfg plugins.Config) error
  • func (s *Destination) Write(ctx context.Context, r record.Record) (record.Position, error)

Constants

const (
	// ConfigKeyBufferSize is the config name for buffer size.
	ConfigKeyBufferSize = "buffer-size"

	// ConfigKeyFormat is the config name for destination format.
	ConfigKeyFormat = "format"

	// ConfigKeyPrefix is the config name for S3 destination key prefix.
	ConfigKeyPrefix = "prefix"

	// MaxBufferSize determines maximum buffer size a config can accept.
	// When config with bigger buffer size is parsed, an error is returned.
	MaxBufferSize uint64 = 100000

	// DefaultBufferSize is the value BufferSize assumes when the config omits
	// the buffer size parameter
	DefaultBufferSize uint64 = 1000
)

Types

type Config

type Config struct {
	config.Config
	BufferSize uint64
	Format     format.Format
	Prefix     string
}

Config represents S3 configuration with Destination specific configurations

func Parse

func Parse(cfg map[string]string) (Config, error)

Parse attempts to parse plugins.Config into a Config struct that Destination could utilize

type Destination

type Destination struct {
	Buffer []record.Record
	Config Config
	Error  error
	Writer writer.Writer
	Mutex  *sync.Mutex
}

Destination S3 Connector persists records to an S3 storage. The records are usually buffered and written in batches for performance reasons. The buffer size is determined by config.

func (*Destination) Open

func (s *Destination) Open(ctx context.Context, cfg plugins.Config) error

Open parses and initializes the config and makes sure everything is prepared to receive records.

func (*Destination) Teardown

func (s *Destination) Teardown() error

Teardown gracefully disconnects the client

func (*Destination) Validate

func (s *Destination) Validate(cfg plugins.Config) error

Validate takes config and returns an error if some values are missing or incorrect.

func (*Destination) Write

func (s *Destination) Write(ctx context.Context, r record.Record) (record.Position, error)

Write writes a record into a Destination. Typically Destination maintains an in-memory buffer and doesn't actually perform a write until the buffer has enough records in it. This is done for performance reasons.