clickhouse

README

clickhouse

Usage

source:
  name: clickhouse
  config:
    connection_url: clickhouse://username:password@clickhouse-server:9000
    exclude:
      databases: [database_a, database_b]
      tables: [database_c.table_a]

Inputs

Key	Value	Example	Description
connection_url	string	tcp://localhost:3306?username=admin&password=pass123&debug=true	URL to access the clickhouse server	required
exclude.databases	[]string	[database_a, database_b]	List of databases to be excluded	optional
exclude.tables	[]string	[database_c.table_a, database_c.table_b]	List of tables to be excluded	optional

Outputs

Field	Sample Value
resource.urn	my_database.my_table
resource.name	my_table
resource.service	clickhouse
description	table description
profile.total_rows	2100
schema	[]column

Column

Field	Sample Value
name	total_price
description	item's total price
data_type	String

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation

Index

• type Config
• type Exclude
• type Extractor
  • func New(logger log.Logger) *Extractor
  • func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error
  • func (e *Extractor) Init(ctx context.Context, config plugins.Config) (err error)

Types

type Config

type Config struct {
	ConnectionURL string  `json:"connection_url" yaml:"connection_url" mapstructure:"connection_url" validate:"required"`
	Exclude       Exclude `json:"exclude" yaml:"exclude" mapstructure:"exclude"`
}

Config holds the connection URL for the extractor

type Exclude

type Exclude struct {
	Databases []string `json:"databases" yaml:"databases" mapstructure:"databases"`
	Tables    []string `json:"tables" yaml:"tables" mapstructure:"tables"`
}

type Extractor

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the output stream and logger interface for the extractor

func New

func New(logger log.Logger) *Extractor

New returns a pointer to an initialized Extractor Object

func (*Extractor) Extract

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

Extract checks if the extractor is configured and if the connection to the DB is successful and then starts the extraction process

func (*Extractor) Init

func (e *Extractor) Init(ctx context.Context, config plugins.Config) (err error)

Init initializes the extractor