config

package
v1.13.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 16, 2019 License: MIT Imports: 9 Imported by: 0

README

Possible types used in configuration:

  • <bool>: a boolean value true or false
  • <addr>: string value consisting of a hostname or IP followed by an optional port number
  • <scheme>: a string that can take the values http or https
  • <duration>: a duration matching the regular expression ^([0-9]+)(w|d|h|m|s|ms|µs|ns)
  • <networks>: string value consisting of IP, IP mask or named group, for example "127.0.0.1" or "127.0.0.1/24".
  • <host_name>: string value consisting of host name, for example "example.com"
  • <byte_size>: string value matching the regular expression /^\d+(\.\d+)?[KMGTP]?B?$/i, for example "100MB"

Global configuration consist of:

# Whether to print debug logs
log_debug: <bool> | default = false [optional]

# Whether to ignore security warnings
hack_me_please <bool> | default = false [optional]

# Named list of cache configurations
caches:
  - <cache_config> ...

# Named list of parameters to apply to each query
param_groups:
  - <param_groups_config> ...

# Named network lists
network_groups: <network_groups_config> ... [optional]

server:
  <server_config> [optional]

# List of allowed users
# which requests will be proxied to ClickHouse
users:
  - <user_config> ...

clusters:
  - <cluster_config> ...

<network_groups_config>

# Name of group
name: "office"

# List of networks access is allowed from
# Each list item could be IP address or subnet mask
networks: <networks> ...

<cache_config>

# Cache name, which may be passed into `cache` option on the `user` level.
#
# Multiple users may share the same cache.
name: <string>

# Path to directory where cached responses will be stored.
dir: <string>

# Maximum cache size.
max_size: <byte_size>

# Expiration time for cached responses.
expire: <duration>

# When multiple requests with identical query simultaneously hit `chproxy`
# and there is no cached response for the query, then only a single
# request will be proxied to clickhouse. Other requests will wait
# for the cached response during this grace duration.
# This is known as protection from `thundering herd` problem.
#
# By default `grace_time` is 5s. Negative value disables the protection
# from `thundering herd` problem.
grace_time: <duration>

<param_groups_config>

# Group name, which may be passed into `params` option on the `user` level.
- name: <string>
# List of key-value params to send
params:
  - key: <string>
    value: <string>

<server_config>

# HTTP server configuration
http: <http_config> [optional]

# HTTPS server configuration
https: <https_config> [optional]

# Metrics handler configuration
metrics: <metrics_config> [optional]

<http_config>

# TCP address to listen to for http
listen_addr: <addr>

# List of networks or network_groups access is allowed from
# Each list item could be IP address or subnet mask
allowed_networks: <network_groups>, <networks> ... | optional

# ReadTimeout is the maximum duration for reading the entire
# request, including the body.
read_timeout: <duration> | optional | default = 1m

# WriteTimeout is the maximum duration before timing out writes of the response.
# Default is largest MaxExecutionTime + MaxQueueTime value from Users or Clusters
write_timeout: <duration> | optional

// IdleTimeout is the maximum amount of time to wait for the next request.
idle_timeout: <duration> | optional | default = 10m

<https_config>

# TCP address to listen to for https
listen_addr: <addr> | optional | default = `:443`

# List of networks or network_groups access is allowed from
# Each list item could be IP address or subnet mask
allowed_networks: <network_groups>, <networks> ... | optional

# ReadTimeout is the maximum duration for proxy to reading the entire
# request, including the body.
read_timeout: <duration> | optional | default = 1m

# WriteTimeout is the maximum duration for proxy before timing out writes of the response.
# Default is largest MaxExecutionTime + MaxQueueTime value from Users or Clusters
write_timeout: <duration> | optional

// IdleTimeout is the maximum amount of time for proxy to wait for the next request.
idle_timeout: <duration> | optional | default = 10m

# Certificate and key files for client cert authentication to the server
cert_file: <string> | optional
key_file: <string> | optional

# Autocert configuration via letsencrypt
autocert: <autocert_config> | optional

<autocert_config>

# Path to the directory where autocert certs are cached
cache_dir: <string>

# List of host names to which proxy is allowed to respond to
# see https://godoc.org/golang.org/x/crypto/acme/autocert#HostPolicy
allowed_hosts: <host_name> ... | optional

<metrics_config>

# List of networks or network_groups access is allowed from
# Each list item could be IP address or subnet mask
allowed_networks: <network_groups>, <networks> ... | optional

<user_config>

# User name, will be taken from BasicAuth or from URL `user`-param
name: <string>

# User password, will be taken from BasicAuth or from URL `password`-param
password: <string> | optional

# Must match with name of `cluster` config,
# where requests will be proxied
to_cluster: <string>

# Must match with name of `user` from `cluster` config,
# whom credentials will be used for proxying request to CH
to_user: <string>

# Maximum number of concurrently running queries for user.
# By default there is no limit on the number of concurrently
# running queries.
max_concurrent_queries: <int> | optional | default = 0

# Maximum duration of query execution for user
# By default there is no limit on the query duration.
max_execution_time: <duration> | optional | default = 0

# Maximum number of requests per minute for user.
# By default there are no per-minute limits
requests_per_minute: <int> | optional | default = 0

# Maximum number of requests waiting for execution in the queue.
# By default requests are executed without waiting in the queue
max_queue_size: <int> | optional | default = 0

# Maximum duration the request may wait in the queue.
# By default 10s duration is used
max_queue_time: <duration> | optional | default = 10s

# Whether to deny http connections for this user
deny_http: <bool> | optional | default = false

# Whether to deny https connections for this user
deny_https: <bool> | optional | default = false

# Whether to allow `CORS` requests for this user.
# Such requests are needed for `tabix`.
allow_cors: <bool> | optional | default = false

# List of networks or network_groups access is allowed from
# Each list item could be IP address or subnet mask
allowed_networks: <network_groups>, <networks> ... | optional

# Optional response cache name from <cache_config>
# By default responses aren't cached.
cache: <string> | optional

# Optional group of params name to send to ClickHouse with each proxied request from <param_groups_config>
# By default no additional params are sent to ClickHouse.
params: <string> | optional

<cluster_config>

# Name of CH cluster, must match with `to_cluster`
name: <string>

# Scheme: `http` or `https`; would be applied to all nodes
scheme: <scheme> | optional | default = "http"

# Node addresses. Requests would be balanced among them.
#
# Either nodes or replicas may be configured, but not both.
nodes: <addr> ...

# The cluster may contain multiple replicas instead of flat nodes.
#
# Chproxy selects the least loaded node among the least loaded replicas.
#
# Either nodes or replicas may be configured, but not both.
replicas:
    - <replica_config>

# List of ClickHouse cluster users
users:
    - <cluster_user_config> ...

# KillQueryUser - user configuration for killing timed out queries.
# By default timed out queries are killed from `default` user.
kill_query_user: <kill_query_user_config> | optional

# An interval for checking all cluster nodes for availability
heartbeat_interval: <duration> | optional | default = 5s

<replica_config>

# Replica name
name: <string>

# Node addresses in the replica. Requests are balanced among them.
nodes: <addr> ...

<cluster_user_config>

# User name in ClickHouse `users.xml` config
name: <string>

# User password in ClickHouse `users.xml` config
password: <string> | optional 

# Maximum number of concurrently running queries for user
# By default there is no limit on the number of concurrently
# running queries.
max_concurrent_queries: <int> | optional | default = 0

# Maximum duration of query execution for user
# By default there is no limit on the query duration.
max_execution_time: <duration> | optional | default = 0

# Maximum number of requests per minute for user.
# By default there are no per-minute limits
requests_per_minute: <int> | optional | default = 0

# Maximum number of requests waiting for execution in the queue.
# By default requests are executed without waiting in the queue
max_queue_size: <int> | optional | default = 0

# Maximum duration the request may wait in the queue.
# By default 10s duration is used
max_queue_time: <duration> | optional | default = 10s

<kill_query_user_config>

# User name to access CH with basic auth
name: <string>

# User password to access CH with basic auth
password: <string> | optional

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Autocert 

type Autocert struct {
	// Path to the directory where autocert certs are cached
	CacheDir string `yaml:"cache_dir,omitempty"`

	// List of host names to which proxy is allowed to respond to
	// see https://godoc.org/golang.org/x/crypto/acme/autocert#HostPolicy
	AllowedHosts []string `yaml:"allowed_hosts,omitempty"`

	// Catches all undefined fields and must be empty after parsing.
	XXX map[string]interface{} `yaml:",inline"`
}

Autocert configuration via letsencrypt It requires port :80 to be open see https://community.letsencrypt.org/t/2018-01-11-update-regarding-acme-tls-sni-and-shared-hosting-infrastructure/50188

func (*Autocert) UnmarshalYAML 

func (c *Autocert) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type ByteSize 

type ByteSize uint64

ByteSize holds size in bytes.

May be used in yaml for parsing byte size values.

func (*ByteSize) UnmarshalYAML 

func (bs *ByteSize) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Cache 

type Cache struct {
	// Name of configuration for further assign
	Name string `yaml:"name"`

	// Path to directory where cached files will be saved
	Dir string `yaml:"dir"`

	// Maximum total size of all cached to Dir files
	// If size is exceeded - the oldest files in Dir will be deleted
	// until total size becomes normal
	MaxSize ByteSize `yaml:"max_size"`

	// Expiration period for cached response
	// Files which are older than expiration period will be deleted
	// on new request and re-cached
	Expire Duration `yaml:"expire,omitempty"`

	// Grace duration before the expired entry is deleted from the cache.
	GraceTime Duration `yaml:"grace_time,omitempty"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
}

Cache describes configuration options for caching responses from CH clusters

func (*Cache) UnmarshalYAML 

func (c *Cache) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Cluster 

type Cluster struct {
	// Name of ClickHouse cluster
	Name string `yaml:"name"`

	// Scheme: `http` or `https`; would be applied to all nodes
	// default value is `http`
	Scheme string `yaml:"scheme,omitempty"`

	// Nodes contains cluster nodes.
	//
	// Either Nodes or Replicas must be set, but not both.
	Nodes []string `yaml:"nodes,omitempty"`

	// Replicas contains replicas.
	//
	// Either Replicas or Nodes must be set, but not both.
	Replicas []Replica `yaml:"replicas,omitempty"`

	// ClusterUsers - list of ClickHouse users
	ClusterUsers []ClusterUser `yaml:"users"`

	// KillQueryUser - user configuration for killing timed out queries.
	// By default timed out queries are killed under `default` user.
	KillQueryUser KillQueryUser `yaml:"kill_query_user,omitempty"`

	// HeartBeatInterval is an interval of checking
	// all cluster nodes for availability
	// if omitted or zero - interval will be set to 5s
	HeartBeatInterval Duration `yaml:"heartbeat_interval,omitempty"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
}

Cluster describes CH cluster configuration The simplest configuration consists of: 

cluster description - see <remote_servers> section in CH config.xml
and users - see <users> section in CH users.xml

func (*Cluster) UnmarshalYAML 

func (c *Cluster) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type ClusterUser 

type ClusterUser struct {
	// User name in ClickHouse users.xml config
	Name string `yaml:"name"`

	// User password in ClickHouse users.xml config
	Password string `yaml:"password,omitempty"`

	// Maximum number of concurrently running queries for user
	// if omitted or zero - no limits would be applied
	MaxConcurrentQueries uint32 `yaml:"max_concurrent_queries,omitempty"`

	// Maximum duration of query execution for user
	// if omitted or zero - no limits would be applied
	MaxExecutionTime Duration `yaml:"max_execution_time,omitempty"`

	// Maximum number of requests per minute for user
	// if omitted or zero - no limits would be applied
	ReqPerMin uint32 `yaml:"requests_per_minute,omitempty"`

	// Maximum number of queries waiting for execution in the queue
	// if omitted or zero - queries are executed without waiting
	// in the queue
	MaxQueueSize uint32 `yaml:"max_queue_size,omitempty"`

	// Maximum duration the query may wait in the queue
	// if omitted or zero - 10s duration is used
	MaxQueueTime Duration `yaml:"max_queue_time,omitempty"`

	NetworksOrGroups NetworksOrGroups `yaml:"allowed_networks,omitempty"`

	// List of networks that access is allowed from
	// Each list item could be IP address or subnet mask
	// if omitted or zero - no limits would be applied
	AllowedNetworks Networks `yaml:"-"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
}

ClusterUser describes simplest <users> configuration

func (*ClusterUser) UnmarshalYAML 

func (cu *ClusterUser) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Config 

type Config struct {
	Server Server `yaml:"server,omitempty"`

	Clusters []Cluster `yaml:"clusters"`

	Users []User `yaml:"users"`

	// Whether to print debug logs
	LogDebug bool `yaml:"log_debug,omitempty"`

	// Whether to ignore security warnings
	HackMePlease bool `yaml:"hack_me_please,omitempty"`

	NetworkGroups []NetworkGroups `yaml:"network_groups,omitempty"`

	Caches []Cache `yaml:"caches,omitempty"`

	ParamGroups []ParamGroup `yaml:"param_groups,omitempty"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
	// contains filtered or unexported fields
}

Config describes server configuration, access and proxy rules

func LoadFile 

func LoadFile(filename string) (*Config, error)

LoadFile loads and validates configuration from provided .yml file

func (*Config) String 

func (c *Config) String() string

String implements the Stringer interface

func (*Config) UnmarshalYAML 

func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Duration 

type Duration time.Duration

Duration wraps time.Duration. It is used to parse the custom duration format

func (Duration) MarshalYAML 

func (d Duration) MarshalYAML() (interface{}, error)

MarshalYAML implements the yaml.Marshaler interface.

func (Duration) String 

func (d Duration) String() string

String implements the Stringer interface.

func (*Duration) UnmarshalYAML 

func (d *Duration) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type HTTP 

type HTTP struct {
	// TCP address to listen to for http
	ListenAddr string `yaml:"listen_addr"`

	NetworksOrGroups NetworksOrGroups `yaml:"allowed_networks,omitempty"`

	// List of networks that access is allowed from
	// Each list item could be IP address or subnet mask
	// if omitted or zero - no limits would be applied
	AllowedNetworks Networks `yaml:"-"`

	// Whether to support Autocert handler for http-01 challenge
	ForceAutocertHandler bool

	TimeoutCfg `yaml:",inline"`

	// Catches all undefined fields and must be empty after parsing.
	XXX map[string]interface{} `yaml:",inline"`
}

HTTP describes configuration for server to listen HTTP connections

func (*HTTP) UnmarshalYAML 

func (c *HTTP) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type HTTPS 

type HTTPS struct {
	// TCP address to listen to for https
	// Default is `:443`
	ListenAddr string `yaml:"listen_addr,omitempty"`

	// Certificate and key files for client cert authentication to the server
	CertFile string `yaml:"cert_file,omitempty"`
	KeyFile  string `yaml:"key_file,omitempty"`

	Autocert Autocert `yaml:"autocert,omitempty"`

	NetworksOrGroups NetworksOrGroups `yaml:"allowed_networks,omitempty"`

	// List of networks that access is allowed from
	// Each list item could be IP address or subnet mask
	// if omitted or zero - no limits would be applied
	AllowedNetworks Networks `yaml:"-"`

	TimeoutCfg `yaml:",inline"`

	// Catches all undefined fields and must be empty after parsing.
	XXX map[string]interface{} `yaml:",inline"`
}

HTTPS describes configuration for server to listen HTTPS connections It can be autocert with letsencrypt or custom certificate

func (*HTTPS) UnmarshalYAML 

func (c *HTTPS) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type KillQueryUser 

type KillQueryUser struct {
	// User name
	Name string `yaml:"name"`

	// User password to access CH with basic auth
	Password string `yaml:"password,omitempty"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
}

KillQueryUser - user configuration for killing timed out queries.

func (*KillQueryUser) UnmarshalYAML 

func (u *KillQueryUser) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Metrics 

type Metrics struct {
	NetworksOrGroups NetworksOrGroups `yaml:"allowed_networks,omitempty"`

	// List of networks that access is allowed from
	// Each list item could be IP address or subnet mask
	// if omitted or zero - no limits would be applied
	AllowedNetworks Networks `yaml:"-"`

	// Catches all undefined fields and must be empty after parsing.
	XXX map[string]interface{} `yaml:",inline"`
}

Metrics describes configuration to access metrics endpoint

func (*Metrics) UnmarshalYAML 

func (c *Metrics) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type NetworkGroups 

type NetworkGroups struct {
	// Name of the group
	Name string `yaml:"name"`

	// List of networks
	// Each list item could be IP address or subnet mask
	Networks Networks `yaml:"networks"`

	// Catches all undefined fields and must be empty after parsing.
	XXX map[string]interface{} `yaml:",inline"`
}

NetworkGroups describes a named Networks lists

func (*NetworkGroups) UnmarshalYAML 

func (ng *NetworkGroups) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Networks 

type Networks []*net.IPNet

Networks is a list of IPNet entities

func (Networks) Contains 

func (n Networks) Contains(addr string) bool

Contains checks whether passed addr is in the range of networks

func (Networks) MarshalYAML 

func (n Networks) MarshalYAML() (interface{}, error)

MarshalYAML implements yaml.Marshaler interface.

It prettifies yaml output for Networks.

func (*Networks) UnmarshalYAML 

func (n *Networks) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type NetworksOrGroups 

type NetworksOrGroups []string

NetworksOrGroups is a list of strings with names of NetworkGroups or just Networks

type Param 

type Param struct {
	// Key is a name of params
	Key string `yaml:"key"`
	// Value is a value of param
	Value string `yaml:"value"`
}

Param describes URL param value

type ParamGroup 

type ParamGroup struct {
	// Name of configuration for further assign
	Name string `yaml:"name"`

	// Params contains a list of GET params
	Params []Param `yaml:"params"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
}

ParamGroup describes named group of GET params for sending with each query

func (*ParamGroup) UnmarshalYAML 

func (pg *ParamGroup) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Replica 

type Replica struct {
	// Name is replica name.
	Name string `yaml:"name"`

	// Nodes contains replica nodes.
	Nodes []string `yaml:"nodes"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
}

Replica contains ClickHouse replica configuration.

func (*Replica) UnmarshalYAML 

func (r *Replica) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type Server 

type Server struct {
	// Optional HTTP configuration
	HTTP HTTP `yaml:"http,omitempty"`

	// Optional TLS configuration
	HTTPS HTTPS `yaml:"https,omitempty"`

	// Optional metrics handler configuration
	Metrics Metrics `yaml:"metrics,omitempty"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
}

Server describes configuration of proxy server These settings are immutable and can't be reloaded without restart

func (*Server) UnmarshalYAML 

func (s *Server) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

type TimeoutCfg 

type TimeoutCfg struct {
	// ReadTimeout is the maximum duration for reading the entire
	// request, including the body.
	// Default value is 1m
	ReadTimeout Duration `yaml:"read_timeout,omitempty"`

	// WriteTimeout is the maximum duration before timing out writes of the response.
	// Default is largest MaxExecutionTime + MaxQueueTime value from Users or Clusters
	WriteTimeout Duration `yaml:"write_timeout,omitempty"`

	// IdleTimeout is the maximum amount of time to wait for the next request.
	// Default is 10m
	IdleTimeout Duration `yaml:"idle_timeout,omitempty"`
}

TimeoutCfg contains configurable http.Server timeouts

type User 

type User struct {
	// User name
	Name string `yaml:"name"`

	// User password to access proxy with basic auth
	Password string `yaml:"password,omitempty"`

	// ToCluster is the name of cluster where requests
	// will be proxied
	ToCluster string `yaml:"to_cluster"`

	// ToUser is the name of cluster_user from cluster's ToCluster
	// whom credentials will be used for proxying request to CH
	ToUser string `yaml:"to_user"`

	// Maximum number of concurrently running queries for user
	// if omitted or zero - no limits would be applied
	MaxConcurrentQueries uint32 `yaml:"max_concurrent_queries,omitempty"`

	// Maximum duration of query execution for user
	// if omitted or zero - no limits would be applied
	MaxExecutionTime Duration `yaml:"max_execution_time,omitempty"`

	// Maximum number of requests per minute for user
	// if omitted or zero - no limits would be applied
	ReqPerMin uint32 `yaml:"requests_per_minute,omitempty"`

	// Maximum number of queries waiting for execution in the queue
	// if omitted or zero - queries are executed without waiting
	// in the queue
	MaxQueueSize uint32 `yaml:"max_queue_size,omitempty"`

	// Maximum duration the query may wait in the queue
	// if omitted or zero - 10s duration is used
	MaxQueueTime Duration `yaml:"max_queue_time,omitempty"`

	NetworksOrGroups NetworksOrGroups `yaml:"allowed_networks,omitempty"`

	// List of networks that access is allowed from
	// Each list item could be IP address or subnet mask
	// if omitted or zero - no limits would be applied
	AllowedNetworks Networks `yaml:"-"`

	// Whether to deny http connections for this user
	DenyHTTP bool `yaml:"deny_http,omitempty"`

	// Whether to deny https connections for this user
	DenyHTTPS bool `yaml:"deny_https,omitempty"`

	// Whether to allow CORS requests for this user
	AllowCORS bool `yaml:"allow_cors,omitempty"`

	// Name of Cache configuration to use for responses of this user
	Cache string `yaml:"cache,omitempty"`

	// Name of ParamGroup to use
	Params string `yaml:"params,omitempty"`

	// Catches all undefined fields
	XXX map[string]interface{} `yaml:",inline"`
}

User describes list of allowed users which requests will be proxied to ClickHouse

func (*User) UnmarshalYAML 

func (u *User) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.