README ¶
ch
TCP ClickHouse client in Go. Designed for very fast data block streaming with low network, cpu and memory overhead.
Work in progress, please leave feedback on package API or features. Also, see benchmarks and protocol reference.
ClickHouse is an open-source, high performance columnar OLAP database management system for real-time analytics using SQL.
go get github.com/go-faster/ch@latest
Example
package main
import (
"context"
"fmt"
"github.com/go-faster/ch"
"github.com/go-faster/ch/proto"
)
func main() {
ctx := context.Background()
c, err := ch.Dial(ctx, "localhost:9000", ch.Options{})
if err != nil {
panic(err)
}
var (
numbers int
data proto.ColUInt64
)
if err := c.Do(ctx, ch.Query{
Body: "SELECT number FROM system.numbers LIMIT 500000000",
// OnResult will be called on next received data block.
OnResult: func(ctx context.Context, b proto.Block) error {
numbers += len(data)
return nil
},
Result: proto.Results{
{Name: "number", Data: &data},
},
}); err != nil {
panic(err)
}
fmt.Println("numbers:", numbers)
}
393ms 0.5B rows 4GB 10GB/s 1 job
874ms 2.0B rows 16GB 18GB/s 4 jobs
Results
To stream query results, set Result
and OnResult
fields of Query.
The OnResult
will be called after Result
is filled with received data block.
The OnResult
is optional, but query will fail if more than single block is received, so it is ok to solely set the Result
if only one row is expected.
Automatic result inference
var result proto.Results
q := ch.Query{
Body: "SELECT * FROM table",
Result: result.Auto(),
}
Single result with column name inference
var res proto.ColBool
q := ch.Query{
Body: "SELECT v FROM test_table",
Result: proto.ResultColumn{Data: &res},
}
Features
- OpenTelemetry support
- No reflection or
interface{}
- Generics (go1.18) for
ArrayOf[T]
,LowCardinaliyOf[T]
,EnumOf[T]
- Column-oriented design that operates with blocks
- Dramatically more efficient
- Up to 100x faster than row-first design around
sql
- Up to 700x faster than HTTP API
- Low memory overhead (data blocks are slices, i.e. continuous memory)
- Highly efficient input and output block streaming
- As close to ClickHouse as possible
- Structured query execution telemetry streaming
- Query progress
- Profiles
- Logs
- Profile events
- LZ4, ZSTD or None (just checksums) compression
- External data support
- Rigorously tested
- ARM64, Windows, Mac, Linux (also x86)
- Unit tests for encoding and decoding
- ClickHouse Server in Go for faster tests
- Golden files for all packets, columns
- Both server and client structures
- Ensuring that partial read leads to failure
- End-to-end tests
- 21.8.13.6-lts
- 21.9.6.24-stable
- 21.10.4.26-stable
- 21.11.10.1-stable
- 21.12.3.32-stable
- 22.1.3.7-stable
- Fuzzing
Supported types
- UInt8, UInt16, UInt32, UInt64, UInt128, UInt256
- Int8, Int16, Int32, Int64, Int128, Int256
- Date, Date32, DateTime, DateTime64
- Decimal32, Decimal64, Decimal128, Decimal256
- IPv4, IPv6
- String, FixedString(N)
- UUID
- Array(T)
- Enum8, Enum16
- LowCardinality(T)
- Map(K, V)
- Bool
- Tuple(T1, T2, ..., Tn)
- Nullable(T)
Generics
ArrayOf
Generic for Array(T)
// Array(String)
arr := proto.ArrayOf[string](new(proto.ColStr))
// Or
arr := new(proto.ColStr).Array()
q := ch.Query{
Body: "SELECT ['foo', 'bar', 'baz']::Array(String) as v",
Result: arr.Results("v"),
}
// Do ...
arr.Row(0) // ["foo", "bar", "baz"]
TODO
- Connection pools
- Improved i/o timeout handling
- Close connection on context cancellation in all cases
- Ensure that reads can't block forever
- TLS
- API UX Improvements (with 1.18 generics?)
- Enum
- LowCardinality
- Array(T)
- FixedString(N)
- Map(K, V)
- Decimal(P, S)
- Nullable(T)
- Tuple?
- Code generation from DDL
- Parser
- Code generator for SELECT/INSERT
- Query builder
- DSL for DDL
-
database/sql
integration - Reading and writing Native format dumps
Reference
License
Apache License 2.0
Documentation ¶
Overview ¶
Package ch implements ClickHouse client.
Index ¶
- Constants
- func CompressionStrings() []string
- func IsErr(err error, codes ...proto.Error) bool
- func IsException(err error) bool
- type Client
- type Compression
- type CorruptedDataErr
- type Dialer
- type Exception
- type Log
- type Options
- type ProfileEvent
- type ProfileEventType
- type Query
- type Server
- type ServerConn
- type ServerOptions
- type Setting
Examples ¶
Constants ¶
const ( DefaultDatabase = "default" DefaultUser = "default" DefaultHost = "127.0.0.1" DefaultPort = 9000 )
Defaults for connection.
Variables ¶
This section is empty.
Functions ¶
func CompressionStrings ¶
func CompressionStrings() []string
CompressionStrings returns a slice of all String values of the enum
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client implements ClickHouse binary protocol client on top of single TCP connection.
func Connect ¶
Connect performs handshake with ClickHouse server and initializes application level connection.
func Dial ¶
Dial dials requested address and establishes TCP connection to ClickHouse server, performing handshake.
func (*Client) Close ¶
Close closes underlying connection and frees all resources, rendering Client to unusable state.
func (*Client) ServerInfo ¶ added in v0.19.0
func (c *Client) ServerInfo() proto.ServerHello
ServerInfo returns server information.
type Compression ¶
type Compression byte
Compression setting.
Trade bandwidth for CPU.
const ( // CompressionDisabled disables compression. Lowest CPU overhead. CompressionDisabled Compression = iota // CompressionLZ4 enables LZ4 compression for data. Medium CPU overhead. CompressionLZ4 // CompressionZSTD enables ZStandard compression. High CPU overhead. CompressionZSTD // CompressionNone uses no compression but data has checksums. CompressionNone )
func CompressionString ¶
func CompressionString(s string) (Compression, error)
CompressionString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.
func CompressionValues ¶
func CompressionValues() []Compression
CompressionValues returns all values of the enum
func (Compression) IsACompression ¶
func (i Compression) IsACompression() bool
IsACompression returns "true" if the value is listed in the enum definition. "false" otherwise
func (Compression) String ¶
func (i Compression) String() string
type CorruptedDataErr ¶ added in v0.5.0
CorruptedDataErr means that provided hash mismatch with calculated.
func (*CorruptedDataErr) Error ¶ added in v0.5.0
func (c *CorruptedDataErr) Error() string
type Dialer ¶ added in v0.17.1
type Dialer interface {
DialContext(ctx context.Context, network, address string) (net.Conn, error)
}
A Dialer dials using a context.
type Exception ¶
type Exception struct { Code proto.Error Name string Message string Stack string Next []Exception // non-nil only for top exception }
Exception is server-side error.
func AsException ¶
AsException finds first *Exception in err chain.
type Options ¶
type Options struct { Logger *zap.Logger // defaults to Nop. Address string // 127.0.0.1:9000 Database string // "default" User string // "default" Password string // blank string by default Compression Compression // disabled by default Dialer Dialer // defaults to net.Dialer Settings []Setting // none by default // Additional OpenTelemetry instrumentation that will capture query body // and other parameters. // // Note: OpenTelemetry context propagation works without this option too. OpenTelemetryInstrumentation bool TracerProvider trace.TracerProvider MeterProvider metric.MeterProvider // contains filtered or unexported fields }
Options for Client. Zero value is valid.
type ProfileEvent ¶ added in v0.5.0
type ProfileEvent = proto.ProfileEvent
type ProfileEventType ¶ added in v0.5.0
type ProfileEventType = proto.ProfileEventType
type Query ¶
type Query struct { // Body of query, like "SELECT 1". Body string // QueryID is ID of query, defaults to new UUIDv4. QueryID string // QuotaKey of query, optional. QuotaKey string // Input columns for INSERT operations. Input proto.Input // OnInput is called to allow ingesting more data to Input. // // The io.EOF reports that no more input should be ingested. // // Optional, single block is ingested from Input if not provided, // but query will fail if Input is set but has zero rows. OnInput func(ctx context.Context) error // Result columns for SELECT operations. Result proto.Result // OnResult is called when Result is filled with result block. // // Optional, but query will fail of more than one block is received // and no OnResult is provided. OnResult func(ctx context.Context, block proto.Block) error // OnProgress is optional progress handler. The progress value contain // difference, so progress should be accumulated if needed. OnProgress func(ctx context.Context, p proto.Progress) error // OnProfile is optional handler for profiling data. OnProfile func(ctx context.Context, p proto.Profile) error // OnProfileEvent is optional handler for profiling event stream data. OnProfileEvent func(ctx context.Context, e ProfileEvent) error // OnLog is optional handler for server log entry. OnLog func(ctx context.Context, l Log) error // Settings are optional query-scoped settings. Can override client settings. Settings []Setting // Secret is optional inter-server per-cluster secret for Distributed queries. // // See https://clickhouse.com/docs/en/engines/table-engines/special/distributed/#distributed-clusters Secret string // InitialUser is optional initial user for Distributed queries. InitialUser string // ExternalData is optional data for server to load. // // https://clickhouse.com/docs/en/engines/table-engines/special/external-data/ ExternalData []proto.InputColumn // ExternalTable name. Defaults to _data. ExternalTable string }
Query to ClickHouse.
Example (MultipleInputColumns) ¶
Output: INSERT INTO logs (timestamp, severity_text, severity_number, body, name, arr) VALUES
type Server ¶ added in v0.4.0
type Server struct {
// contains filtered or unexported fields
}
Server is basic ClickHouse server.
func NewServer ¶ added in v0.4.0
func NewServer(opt ServerOptions) *Server
NewServer returns new ClickHouse Server.
type ServerConn ¶ added in v0.4.0
type ServerConn struct {
// contains filtered or unexported fields
}
ServerConn wraps Server connection.
type ServerOptions ¶ added in v0.4.0
ServerOptions wraps possible Server configuration.
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
Package cht implements ClickHouse testing utilities, primarily end to end.
|
Package cht implements ClickHouse testing utilities, primarily end to end. |
internal
|
|
Package otelch provide OpenTelemetry instrumentation for go-faster/ch.
|
Package otelch provide OpenTelemetry instrumentation for go-faster/ch. |
Package proto implements ClickHouse wire protocol.
|
Package proto implements ClickHouse wire protocol. |