clickhouse

package module
v2.6.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 3, 2023 License: Apache-2.0 Imports: 40 Imported by: 0

README

ClickHouse run-tests Go Reference

Golang SQL database client for ClickHouse.

Versions

There are two version of this client, v1 and v2, available as separate branches.

v1 is now in a state of a maintenance - we will only accept PRs for bug and security fixes.

Users should use v2 which is production ready and significantly faster than v1.

v2 has breaking changes for users migrating from v1. These were not properly tracked prior to this client being officially supported. We endeavour to track known differences here and resolve where possible.

Supported ClickHouse Versions

The client is tested against the currently supported versions of ClickHouse

Supported Golang Versions

Client Version Golang Versions
=> 2.0 <= 2.2 1.17, 1.18
>= 2.3 1.18.4+, 1.19

Key features

  • Uses ClickHouse native format for optimal performance. Utilises low level ch-go client for encoding/decoding and compression (versions >= 2.3.0).
  • Supports native ClickHouse TCP client-server protocol
  • Compatibility with database/sql (slower than native interface!)
  • database/sql supports http protocol for transport. (Experimental)
  • Marshal rows into structs (ScanStruct, Select)
  • Unmarshal struct to row (AppendStruct)
  • Connection pool
  • Failover and load balancing
  • Bulk write support (for database/sql use begin->prepare->(in loop exec)->commit)
  • AsyncInsert
  • Named and numeric placeholders support
  • LZ4/ZSTD compression support
  • External data
  • Query parameters

Support for the ClickHouse protocol advanced features using Context:

  • Query ID
  • Quota Key
  • Settings
  • Query parameters
  • OpenTelemetry
  • Execution events:
    • Logs
    • Progress
    • Profile info
    • Profile events

Documentation

https://clickhouse.com/docs/en/integrations/go

clickhouse interface (formally native interface)

	conn, err := clickhouse.Open(&clickhouse.Options{
		Addr: []string{"127.0.0.1:9000"},
		Auth: clickhouse.Auth{
			Database: "default",
			Username: "default",
			Password: "",
		},
		DialContext: func(ctx context.Context, addr string) (net.Conn, error) {
			dialCount++
			var d net.Dialer
			return d.DialContext(ctx, "tcp", addr)
		},
		Debug: true,
		Debugf: func(format string, v ...interface{}) {
			fmt.Printf(format, v)
		},
		Settings: clickhouse.Settings{
			"max_execution_time": 60,
		},
		Compression: &clickhouse.Compression{
			Method: clickhouse.CompressionLZ4,
		},
		DialTimeout:      time.Second * 30,
		MaxOpenConns:     5,
		MaxIdleConns:     5,
		ConnMaxLifetime:  time.Duration(10) * time.Minute,
		ConnOpenStrategy: clickhouse.ConnOpenInOrder,
		BlockBufferSize: 10,
		MaxCompressionBuffer: 10240,
		ClientInfo: clickhouse.ClientInfo{ // optional, please see Client info section in the README.md
			Products: []struct {
				Name    string
				Version string
			}{
				{Name: "my-app", Version: "0.1"},
			},
		},
	})
	if err != nil {
		return err
	}
	return conn.Ping(context.Background())

database/sql interface

OpenDB

conn := clickhouse.OpenDB(&clickhouse.Options{
	Addr: []string{"127.0.0.1:9999"},
	Auth: clickhouse.Auth{
		Database: "default",
		Username: "default",
		Password: "",
	},
	TLS: &tls.Config{
		InsecureSkipVerify: true,
	},
	Settings: clickhouse.Settings{
		"max_execution_time": 60,
	},
	DialTimeout: time.Second * 30,
	Compression: &clickhouse.Compression{
		clickhouse.CompressionLZ4,
	},
	Debug: true,
	BlockBufferSize: 10,
	MaxCompressionBuffer: 10240,
	ClientInfo: clickhouse.ClientInfo{ // optional, please see Client info section in the README.md
		Products: []struct {
			Name    string
			Version string
		}{
			{Name: "my-app", Version: "0.1"},
		},
	},
})
conn.SetMaxIdleConns(5)
conn.SetMaxOpenConns(10)
conn.SetConnMaxLifetime(time.Hour)

DSN

  • hosts - comma-separated list of single address hosts for load-balancing and failover
  • username/password - auth credentials
  • database - select the current default database
  • dial_timeout - a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix such as "300ms", "1s". Valid time units are "ms", "s", "m". (default 30s)
  • connection_open_strategy - round_robin/in_order (default in_order).
    • round_robin - choose a round-robin server from the set
    • in_order - first live server is chosen in specified order
  • debug - enable debug output (boolean value)
  • compress - compress - specify the compression algorithm - “none” (default), zstd, lz4, gzip, deflate, br. If set to true, lz4 will be used.
  • compress_level - Level of compression (default is 0). This is algorithm specific:
    • gzip - -2 (Best Speed) to 9 (Best Compression)
    • deflate - -2 (Best Speed) to 9 (Best Compression)
    • br - 0 (Best Speed) to 11 (Best Compression)
    • zstd, lz4 - ignored
  • block_buffer_size - size of block buffer (default 2)
  • read_timeout - a duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix such as "300ms", "1s". Valid time units are "ms", "s", "m" (default 5m).
  • max_compression_buffer - max size (bytes) of compression buffer during column by column compression (default 10MiB)
  • client_info_product - optional list (comma separated) of product name and version pair separated with /. This value will be pass a part of client info. e.g. client_info_product=my_app/1.0,my_module/0.1 More details in Client info section.

SSL/TLS parameters:

  • secure - establish secure connection (default is false)
  • skip_verify - skip certificate verification (default is false)

Example:

clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60
HTTP Support (Experimental)

The native format can be used over the HTTP protocol. This is useful in scenarios where users need to proxy traffic e.g. using ChProxy or via load balancers.

This can be achieved by modifying the DSN to specify the http protocol.

http://host1:8123,host2:8123/database?dial_timeout=200ms&max_execution_time=60

Alternatively, use OpenDB and specify the interface type.

conn := clickhouse.OpenDB(&clickhouse.Options{
	Addr: []string{"127.0.0.1:8123"},
	Auth: clickhouse.Auth{
		Database: "default",
		Username: "default",
		Password: "",
	},
	Settings: clickhouse.Settings{
		"max_execution_time": 60,
	},
	DialTimeout: 30 * time.Second,
	Compression: &clickhouse.Compression{
		Method: clickhouse.CompressionLZ4,
	},
	Interface: clickhouse.HttpInterface,
})

Compression

ZSTD/LZ4 compression is supported over native and http protocols. This is performed column by column at a block level and is only used for inserts. Compression buffer size is set as MaxCompressionBuffer option.

If using Open via the std interface and specifying a DSN, compression can be enabled via the compress flag. Currently, this is a boolean flag which enables LZ4 compression.

Other compression methods will be added in future PRs.

TLS/SSL

At a low level all client connect methods (DSN/OpenDB/Open) will use the Go tls package to establish a secure connection. The client knows to use TLS if the Options struct contains a non-nil tls.Config pointer.

Setting secure in the DSN creates a minimal tls.Config struct with only the InsecureSkipVerify field set (either true or false). It is equivalent to this code:

conn := clickhouse.OpenDB(&clickhouse.Options{
	...
    TLS: &tls.Config{
            InsecureSkipVerify: false
	}
	...
    })

This minimal tls.Config is normally all that is necessary to connect to the secure native port (normally 9440) on a ClickHouse server. If the ClickHouse server does not have a valid certificate (expired, wrong host name, not signed by a publicly recognized root Certificate Authority), InsecureSkipVerify can be to true, but that is strongly discouraged.

If additional TLS parameters are necessary the application code should set the desired fields in the tls.Config struct. That can include specific cipher suites, forcing a particular TLS version (like 1.2 or 1.3), adding an internal CA certificate chain, adding a client certificate (and private key) if required by the ClickHouse server, and most of the other options that come with a more specialized security setup.

HTTPS (Experimental)

To connect using HTTPS either:

  • Use https in your dsn string e.g.

    https://host1:8443,host2:8443/database?dial_timeout=200ms&max_execution_time=60

  • Specify the interface type as HttpsInterface e.g.

conn := clickhouse.OpenDB(&clickhouse.Options{
	Addr: []string{"127.0.0.1:8443"},
	Auth: clickhouse.Auth{
		Database: "default",
		Username: "default",
		Password: "",
	},
	Interface: clickhouse.HttpsInterface,
})

Client info

Clickhouse-go implements client info as a part of language client specification. client_name for native protocol and HTTP User-Agent header values are provided with the exact client info string.

Users can extend client options with additional product information included in client info. This might be useful for analysis on a server side.

Order is the highest abstraction to the lowest level implementation left to right.

Usage examples for native API and database/sql are provided.

Benchmark

V1 (READ) V2 (READ) std V2 (READ) clickhouse API
1.218s 924.390ms 675.721ms
V1 (WRITE) V2 (WRITE) std V2 (WRITE) clickhouse API V2 (WRITE) by column
1.899s 1.177s 699.203ms 661.973ms

Install

go get -u github.com/v4run/clickhouse-go/v2

Examples

native interface
std database/sql interface

ClickHouse alternatives - ch-go

Versions of this client >=2.3.x utilise ch-go for their low level encoding/decoding. This low lever client provides a high performance columnar interface and should be used in performance critical use cases. This client provides more familar row orientated and database/sql semantics at the cost of some performance.

Both clients are supported by ClickHouse.

Third-party alternatives

Documentation

Index

Constants

View Source 
const (
	CompressionNone    = CompressionMethod(compress.None)
	CompressionLZ4     = CompressionMethod(compress.LZ4)
	CompressionZSTD    = CompressionMethod(compress.ZSTD)
	CompressionGZIP    = CompressionMethod(0x95)
	CompressionDeflate = CompressionMethod(0x96)
	CompressionBrotli  = CompressionMethod(0x97)
)
View Source 
const (
	ClientVersionMajor       = 2
	ClientVersionMinor       = 5
	ClientVersionPatch       = 1
	ClientTCPProtocolVersion = proto.DBMS_TCP_PROTOCOL_VERSION
)
View Source 
const ClientName = "clickhouse-go"

Variables

View Source 
var (
	ErrBatchInvalid              = errors.New("clickhouse: batch is invalid. check appended data is correct")
	ErrBatchAlreadySent          = errors.New("clickhouse: batch has already been sent")
	ErrAcquireConnTimeout        = errors.New("clickhouse: acquire conn timeout. you can increase the number of max open conn or the dial timeout")
	ErrUnsupportedServerRevision = errors.New("clickhouse: unsupported server revision")
	ErrBindMixedParamsFormats    = errors.New("clickhouse [bind]: mixed named, numeric or positional parameters")
	ErrAcquireConnNoAddress      = errors.New("clickhouse: no valid address supplied")
)
View Source 
var (
	ErrExpectedStringValueInNamedValueForQueryParameter = errors.New("expected string value in NamedValue for query parameter")
)

Functions

func Connector 

func Connector(opt *Options) driver.Connector

func Context 

func Context(parent context.Context, options ...QueryOption) context.Context

func DateNamed 

func DateNamed(name string, value time.Time, scale TimeUnit) driver.NamedDateValue

func Named 

func Named(name string, value interface{}) driver.NamedValue

func Open 

func Open(opt *Options) (driver.Conn, error)

func OpenDB 

func OpenDB(opt *Options) *sql.DB

Types

type ArraySet 

type ArraySet []interface{}

type Auth 

type Auth struct {
	Database string
	Username string
	Password string
}

type ClientInfo 

type ClientInfo struct {
	Products []struct {
		Name    string
		Version string
	}
	// contains filtered or unexported fields
}

func (ClientInfo) String 

func (o ClientInfo) String() string

type Compression 

type Compression struct {
	Method CompressionMethod
	// this only applies to zlib and brotli compression algorithms
	Level int
}

type CompressionMethod 

type CompressionMethod byte

func (CompressionMethod) String 

func (c CompressionMethod) String() string

type Conn 

type Conn = driver.Conn

type ConnOpenStrategy 

type ConnOpenStrategy uint8
const (
	ConnOpenInOrder ConnOpenStrategy = iota
	ConnOpenRoundRobin
)

type Dial 

type Dial func(ctx context.Context, addr string, opt *Options) (DialResult, error)

type DialResult 

type DialResult struct {
	// contains filtered or unexported fields
}

func DefaultDialStrategy 

func DefaultDialStrategy(ctx context.Context, connID int, opt *Options, dial Dial) (r DialResult, err error)

type Exception 

type Exception = proto.Exception

type GroupSet 

type GroupSet struct {
	Value []interface{}
}

type HTTPReaderWriter 

type HTTPReaderWriter struct {
	// contains filtered or unexported fields
}

type Log 

type Log struct {
	Time      time.Time
	TimeMicro uint32
	Hostname  string
	QueryID   string
	ThreadID  uint64
	Priority  int8
	Source    string
	Text      string
}

type OpError 

type OpError struct {
	Op         string
	ColumnName string
	Err        error
}

func (*OpError) Error 

func (e *OpError) Error() string

type Options 

type Options struct {
	Protocol   Protocol
	ClientInfo ClientInfo

	TLS                  *tls.Config
	Addr                 []string
	Auth                 Auth
	DialContext          func(ctx context.Context, addr string) (net.Conn, error)
	DialStrategy         func(ctx context.Context, connID int, options *Options, dial Dial) (DialResult, error)
	Debug                bool
	Debugf               func(format string, v ...interface{}) // only works when Debug is true
	Settings             Settings
	Compression          *Compression
	DialTimeout          time.Duration // default 30 second
	MaxOpenConns         int           // default MaxIdleConns + 5
	MaxIdleConns         int           // default 5
	ConnMaxLifetime      time.Duration // default 1 hour
	ConnOpenStrategy     ConnOpenStrategy
	HttpHeaders          map[string]string // set additional headers on HTTP requests
	BlockBufferSize      uint8             // default 2 - can be overwritten on query
	MaxCompressionBuffer int               // default 10485760 - measured in bytes  i.e. 10MiB
	AllowEmptyDatabase   bool              // if set, Auth.Database will not be set to default if empty

	ReadTimeout time.Duration
	// contains filtered or unexported fields
}

func ParseDSN 

func ParseDSN(dsn string) (*Options, error)

type Parameters 

type Parameters map[string]string

type Pool 

type Pool[T any] struct {
	// contains filtered or unexported fields
}

func NewPool 

func NewPool[T any](fn func() T) Pool[T]

func (*Pool[T]) Get 

func (p *Pool[T]) Get() T

func (*Pool[T]) Put 

func (p *Pool[T]) Put(x T)

type ProfileEvent 

type ProfileEvent struct {
	Hostname    string
	CurrentTime time.Time
	ThreadID    uint64
	Type        string
	Name        string
	Value       int64
}

type ProfileInfo 

type ProfileInfo = proto.ProfileInfo

type Progress 

type Progress = proto.Progress

type Protocol 

type Protocol int
const (
	Native Protocol = iota
	HTTP
)

func (Protocol) String 

func (p Protocol) String() string

type QueryOption 

type QueryOption func(*QueryOptions) error

func WithBlockBufferSize 

func WithBlockBufferSize(size uint8) QueryOption

func WithExternalTable 

func WithExternalTable(t ...*ext.Table) QueryOption

func WithLogs 

func WithLogs(fn func(*Log)) QueryOption

func WithParameters 

func WithParameters(params Parameters) QueryOption

func WithProfileEvents 

func WithProfileEvents(fn func([]ProfileEvent)) QueryOption

func WithProfileInfo 

func WithProfileInfo(fn func(*ProfileInfo)) QueryOption

func WithProgress 

func WithProgress(fn func(*Progress)) QueryOption

func WithQueryID 

func WithQueryID(queryID string) QueryOption

func WithQuotaKey 

func WithQuotaKey(quotaKey string) QueryOption

func WithSettings 

func WithSettings(settings Settings) QueryOption

func WithSpan 

func WithSpan(span trace.SpanContext) QueryOption

func WithStdAsync 

func WithStdAsync(wait bool) QueryOption

type QueryOptions 

type QueryOptions struct {
	// contains filtered or unexported fields
}

type ServerVersion 

type ServerVersion = proto.ServerHandshake

type Settings 

type Settings map[string]interface{}

type TimeUnit 

type TimeUnit uint8
const (
	Seconds TimeUnit = iota
	MilliSeconds
	MicroSeconds
	NanoSeconds
)

Source Files

View all Source files

Directories

Path Synopsis
benchmark
v1/read
v1/write
v2/read
v2/read-native
v2/write
v2/write-async
v2/write-async-std
v2/write-native
v2/write-native-columnar
v2/write-native-struct
examples
clickhouse_api
std
lib
binary
cityhash102
* COPY from https://github.com/zentures/cityhash/
 * COPY from https://github.com/zentures/cityhash/
column
column/codegen
driver
proto
timezone
issues/209
issues/360
issues/470
issues/476
issues/484
issues/485
Licensed to ClickHouse, Inc.
 Licensed to ClickHouse, Inc.
std
stress

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.