writer

Documentation

Index

• type SnowflakeCSV
  • func NewCSV(ctx context.Context, cfg SnowflakeConfig) (*SnowflakeCSV, error)
  • func (s *SnowflakeCSV) CheckTable(ctx context.Context, operation opencdc.Operation, primaryKey string, ...) error
  • func (s *SnowflakeCSV) Close(ctx context.Context) error
  • func (s *SnowflakeCSV) Merge(ctx context.Context, insertsFilename, updatesFilename string, ...) error
  • func (s *SnowflakeCSV) SetupTables(ctx context.Context, schema map[string]string, columnOrder []string) error
  • func (s *SnowflakeCSV) Write(ctx context.Context, records []opencdc.Record) (int, error)
• type SnowflakeConfig
• type Writer

Types

type SnowflakeCSV

type SnowflakeCSV struct {
	// contains filtered or unexported fields
}

SnowflakeCSV writer stores batch bytes into an SnowflakeCSV bucket as a file.

func NewCSV

func NewCSV(ctx context.Context, cfg SnowflakeConfig) (*SnowflakeCSV, error)

NewCSV takes an SnowflakeConfig reference and produces an SnowflakeCSV Writer.

func (*SnowflakeCSV) CheckTable

func (s *SnowflakeCSV) CheckTable(ctx context.Context, operation opencdc.Operation,
	primaryKey string, schema map[string]string,
) error

func (*SnowflakeCSV) Close

func (s *SnowflakeCSV) Close(ctx context.Context) error

func (*SnowflakeCSV) Merge

func (s *SnowflakeCSV) Merge(
	ctx context.Context,
	insertsFilename,
	updatesFilename string,
	colOrder []string,
	meroxaColumns *format.ConnectorColumns,
) error

func (*SnowflakeCSV) SetupTables

func (s *SnowflakeCSV) SetupTables(ctx context.Context, schema map[string]string, columnOrder []string) error

creates temporary, and destination table if they don't exist already.

func (*SnowflakeCSV) Write

func (s *SnowflakeCSV) Write(ctx context.Context, records []opencdc.Record) (int, error)

type SnowflakeConfig

type SnowflakeConfig struct {
	DSN               string
	Prefix            string
	PrimaryKey        string
	Stage             string
	Table             string
	ProcessingWorkers int
	FileThreads       int
	Compression       string
	CleanStageFiles   bool
}

SnowflakeConfig is a type used to initialize an Snowflake Writer.

type Writer

type Writer interface {
	Write(context.Context, []opencdc.Record) (int, error)
	Close(context.Context) error
}

Writer is an interface that is responsible for persisting record that Destination has accumulated in its buffers. The default writer the Destination would use is SnowflakeWriter, others exists to test local behavior.