clickhouse

README

Clickhouse transformer

Based on Clickhouse Local

This tool accept clickhouse SQL dialect and allow to produce SQL-like in memory data transformation.

Source table inside CH Local named as table, clickhouse table structure mimic source table structure.

Since each source change item (row) contains extra metadata we must match source and target data together. There for each row must have a key defined. All of this key should be uniq in every batch (for this we call collapse function). If we can't match source keys with transformed data we will mark such row as errored.

Documentation

Index

• Constants
• type ClickhouseTransformer
  • func New(config Config, lgr log.Logger) (*ClickhouseTransformer, error)
  • func (s *ClickhouseTransformer) Apply(input []abstract.ChangeItem) abstract.TransformerResult
  • func (s *ClickhouseTransformer) Description() string
  • func (s *ClickhouseTransformer) ResultSchema(schema *abstract.TableSchema) (*abstract.TableSchema, error)
  • func (s *ClickhouseTransformer) Suitable(table abstract.TableID, schema *abstract.TableSchema) bool
  • func (s *ClickhouseTransformer) Type() abstract.TransformerType
• type Config

Constants

const Type = abstract.TransformerType("sql")

Types

type ClickhouseTransformer

type ClickhouseTransformer struct {
	// contains filtered or unexported fields
}

func New

func New(config Config, lgr log.Logger) (*ClickhouseTransformer, error)

func (*ClickhouseTransformer) Apply

func (s *ClickhouseTransformer) Apply(input []abstract.ChangeItem) abstract.TransformerResult

func (*ClickhouseTransformer) Description

func (s *ClickhouseTransformer) Description() string

func (*ClickhouseTransformer) ResultSchema

func (s *ClickhouseTransformer) ResultSchema(schema *abstract.TableSchema) (*abstract.TableSchema, error)

func (*ClickhouseTransformer) Suitable

func (s *ClickhouseTransformer) Suitable(table abstract.TableID, schema *abstract.TableSchema) bool

func (*ClickhouseTransformer) Type

func (s *ClickhouseTransformer) Type() abstract.TransformerType

type Config

type Config struct {
	Tables filter.Tables `json:"tables"`
	Query  string        `json:"query"`
}