cluster

package
v0.10.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 3, 2016 License: MIT Imports: 23 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// DefaultWriteTimeout is the default timeout for a complete write to succeed.
	DefaultWriteTimeout = 5 * time.Second

	// DefaultShardWriterTimeout is the default timeout set on shard writers.
	DefaultShardWriterTimeout = 5 * time.Second

	// DefaultShardMapperTimeout is the default timeout set on shard mappers.
	DefaultShardMapperTimeout = 5 * time.Second

	// DefaultMaxRemoteWriteConnections is the maximum number of open connections
	// that will be available for remote writes to another host.
	DefaultMaxRemoteWriteConnections = 3
)
View Source 
const MaxMessageSize = 1024 * 1024 * 1024 // 1GB

MaxMessageSize defines how large a message can be before we reject it

View Source 
const MuxHeader = 2

MuxHeader is the header byte used in the TCP mux.

Variables

View Source 
var (
	// ErrTimeout is returned when a write times out.
	ErrTimeout = errors.New("timeout")

	// ErrPartialWrite is returned when a write partially succeeds but does
	// not meet the requested consistency level.
	ErrPartialWrite = errors.New("partial write")

	// ErrWriteFailed is returned when no writes succeeded.
	ErrWriteFailed = errors.New("write failed")

	// ErrInvalidConsistencyLevel is returned when parsing the string version
	// of a consistency level.
	ErrInvalidConsistencyLevel = errors.New("invalid consistency level")
)

Functions

func NewBoundedPool added in v0.10.0 

func NewBoundedPool(initialCap, maxCap int, timeout time.Duration, factory Factory) (pool.Pool, error)

NewBoundedPool returns a new pool based on buffered channels with an initial capacity, maximum capacity and timeout to wait for a connection from the pool. Factory is used when initial capacity is greater than zero to fill the pool. A zero initialCap doesn't fill the Pool until a new Get() is called. During a Get(), If there is no new connection available in the pool and total connections is less than the max, a new connection will be created via the Factory() method. Othewise, the call will block until a connection is available or the timeout is reached.

func ReadTLV added in v0.9.0 

func ReadTLV(r io.Reader) (byte, []byte, error)

ReadTLV reads a type-length-value record from r.

func WriteTLV added in v0.9.0 

func WriteTLV(w io.Writer, typ byte, buf []byte) error

WriteTLV writes a type-length-value record to w.

Types

type Balancer added in v0.9.5 

type Balancer interface {
	// Next returns the next Node according to the balancing method
	// or nil if there are no nodes available
	Next() *meta.NodeInfo
}

Balancer represents a load-balancing algorithm for a set of nodes

func NewNodeBalancer added in v0.9.5 

func NewNodeBalancer(nodes []meta.NodeInfo) Balancer

NewNodeBalancer create a shuffled, round-robin balancer so that multiple instances will return nodes in randomized order and each each returned node will be repeated in a cycle

type Config added in v0.9.0 

type Config struct {
	ForceRemoteShardMapping   bool          `toml:"force-remote-mapping"`
	WriteTimeout              toml.Duration `toml:"write-timeout"`
	ShardWriterTimeout        toml.Duration `toml:"shard-writer-timeout"`
	MaxRemoteWriteConnections int           `toml:"max-remote-write-connections"`
	ShardMapperTimeout        toml.Duration `toml:"shard-mapper-timeout"`
}

Config represents the configuration for the clustering service.

func NewConfig added in v0.9.0 

func NewConfig() Config

NewConfig returns an instance of Config with defaults.

type ConsistencyLevel added in v0.9.0 

type ConsistencyLevel int

ConsistencyLevel represent a required replication criteria before a write can be returned as successful 

const (
	// ConsistencyLevelAny allows for hinted hand off, potentially no write happened yet
	ConsistencyLevelAny ConsistencyLevel = iota

	// ConsistencyLevelOne requires at least one data node acknowledged a write
	ConsistencyLevelOne

	// ConsistencyLevelQuorum requires a quorum of data nodes to acknowledge a write
	ConsistencyLevelQuorum

	// ConsistencyLevelAll requires all data nodes to acknowledge a write
	ConsistencyLevelAll
)

func ParseConsistencyLevel added in v0.9.0 

func ParseConsistencyLevel(level string) (ConsistencyLevel, error)

ParseConsistencyLevel converts a consistency level string to the corresponding ConsistencyLevel const

type Factory added in v0.10.0 

type Factory func() (net.Conn, error)

Factory is a function to create new connections.

type MapShardRequest added in v0.9.3 

type MapShardRequest struct {
	// contains filtered or unexported fields
}

MapShardRequest represents the request to map a remote shard for a query.

func (*MapShardRequest) ChunkSize added in v0.9.3 

func (m *MapShardRequest) ChunkSize() int32

ChunkSize returns Shard map request's chunk size

func (*MapShardRequest) MarshalBinary added in v0.9.3 

func (m *MapShardRequest) MarshalBinary() ([]byte, error)

MarshalBinary encodes the object to a binary format.

func (*MapShardRequest) Query added in v0.9.3 

func (m *MapShardRequest) Query() string

Query returns the Shard map request's query

func (*MapShardRequest) SetChunkSize added in v0.9.3 

func (m *MapShardRequest) SetChunkSize(chunkSize int32)

SetChunkSize sets the Shard map request's chunk size

func (*MapShardRequest) SetQuery added in v0.9.3 

func (m *MapShardRequest) SetQuery(query string)

SetQuery sets the Shard map request's Query

func (*MapShardRequest) SetShardID added in v0.9.3 

func (m *MapShardRequest) SetShardID(id uint64)

SetShardID sets the map request's shard id

func (*MapShardRequest) ShardID added in v0.9.3 

func (m *MapShardRequest) ShardID() uint64

ShardID of the map request

func (*MapShardRequest) UnmarshalBinary added in v0.9.3 

func (m *MapShardRequest) UnmarshalBinary(buf []byte) error

UnmarshalBinary populates MapShardRequest from a binary format.

type MapShardResponse added in v0.9.3 

type MapShardResponse struct {
	// contains filtered or unexported fields
}

MapShardResponse represents the response returned from a remote MapShardRequest call

func NewMapShardResponse added in v0.9.3 

func NewMapShardResponse(code int, message string) *MapShardResponse

NewMapShardResponse returns the response returned from a remote MapShardRequest call

func (*MapShardResponse) Code added in v0.9.3 

func (r *MapShardResponse) Code() int

Code returns the Shard map response's code

func (*MapShardResponse) Data added in v0.9.3 

func (r *MapShardResponse) Data() []byte

Data returns the Shard map response's Data

func (*MapShardResponse) Fields added in v0.9.3 

func (r *MapShardResponse) Fields() []string

Fields returns the Shard map response's Fields

func (*MapShardResponse) MarshalBinary added in v0.9.3 

func (r *MapShardResponse) MarshalBinary() ([]byte, error)

MarshalBinary encodes the object to a binary format.

func (*MapShardResponse) Message added in v0.9.3 

func (r *MapShardResponse) Message() string

Message returns the the Shard map response's Message

func (*MapShardResponse) SetCode added in v0.9.3 

func (r *MapShardResponse) SetCode(code int)

SetCode sets the Shard map response's code

func (*MapShardResponse) SetData added in v0.9.3 

func (r *MapShardResponse) SetData(data []byte)

SetData sets the Shard map response's Data

func (*MapShardResponse) SetFields added in v0.9.3 

func (r *MapShardResponse) SetFields(fields []string)

SetFields sets the Shard map response's Fields

func (*MapShardResponse) SetMessage added in v0.9.3 

func (r *MapShardResponse) SetMessage(message string)

SetMessage sets Shard map response's message

func (*MapShardResponse) SetTagSets added in v0.9.3 

func (r *MapShardResponse) SetTagSets(tagsets []string)

SetTagSets sets Shard map response's tagsets

func (*MapShardResponse) TagSets added in v0.9.3 

func (r *MapShardResponse) TagSets() []string

TagSets returns Shard map response's tag sets

func (*MapShardResponse) UnmarshalBinary added in v0.9.3 

func (r *MapShardResponse) UnmarshalBinary(buf []byte) error

UnmarshalBinary populates WritePointRequest from a binary format.

type PointsWriter added in v0.9.0 

type PointsWriter struct {
	WriteTimeout time.Duration
	Logger       *log.Logger

	Node *influxdb.Node

	MetaClient interface {
		Database(name string) (di *meta.DatabaseInfo, err error)
		RetentionPolicy(database, policy string) (*meta.RetentionPolicyInfo, error)
		CreateShardGroup(database, policy string, timestamp time.Time) (*meta.ShardGroupInfo, error)
		ShardOwner(shardID uint64) (string, string, *meta.ShardGroupInfo)
	}

	TSDBStore interface {
		CreateShard(database, retentionPolicy string, shardID uint64) error
		WriteToShard(shardID uint64, points []models.Point) error
	}

	ShardWriter interface {
		WriteShard(shardID, ownerID uint64, points []models.Point) error
	}

	HintedHandoff interface {
		WriteShard(shardID, ownerID uint64, points []models.Point) error
	}

	Subscriber interface {
		Points() chan<- *WritePointsRequest
	}
	// contains filtered or unexported fields
}

PointsWriter handles writes across multiple local and remote data nodes.

func NewPointsWriter added in v0.9.0 

func NewPointsWriter() *PointsWriter

NewPointsWriter returns a new instance of PointsWriter for a node.

func (*PointsWriter) Close added in v0.9.0 

func (w *PointsWriter) Close() error

Close closes the communication channel with the point writer

func (*PointsWriter) MapShards added in v0.9.0 

func (w *PointsWriter) MapShards(wp *WritePointsRequest) (*ShardMapping, error)

MapShards maps the points contained in wp to a ShardMapping. If a point maps to a shard group or shard that does not currently exist, it will be created before returning the mapping.

func (*PointsWriter) Open added in v0.9.0 

func (w *PointsWriter) Open() error

Open opens the communication channel with the point writer

func (*PointsWriter) WritePoints added in v0.9.0 

func (w *PointsWriter) WritePoints(p *WritePointsRequest) error

WritePoints writes across multiple local and remote data nodes according the consistency level.

func (*PointsWriter) WritePointsInto added in v0.9.5 

func (w *PointsWriter) WritePointsInto(p *tsdb.IntoWriteRequest) error

WritePointsInto is a copy of WritePoints that uses a tsdb structure instead of a cluster structure for information. This is to avoid a circular dependency

type RemoteMapper added in v0.9.2 

type RemoteMapper struct {
	// contains filtered or unexported fields
}

RemoteMapper implements the tsdb.Mapper interface. It connects to a remote node, sends a query, and interprets the stream of data that comes back.

func NewRemoteMapper added in v0.9.3 

func NewRemoteMapper(c net.Conn, shardID uint64, stmt influxql.Statement, chunkSize int) *RemoteMapper

NewRemoteMapper returns a new remote mapper using the given connection.

func (*RemoteMapper) Close added in v0.9.2 

func (r *RemoteMapper) Close()

Close the Mapper

func (*RemoteMapper) Fields added in v0.9.3 

func (r *RemoteMapper) Fields() []string

Fields returns RemoteMapper's Fields

func (*RemoteMapper) NextChunk added in v0.9.2 

func (r *RemoteMapper) NextChunk() (chunk interface{}, err error)

NextChunk returns the next chunk read from the remote node to the client.

func (*RemoteMapper) Open added in v0.9.2 

func (r *RemoteMapper) Open() (err error)

Open connects to the remote node and starts receiving data.

func (*RemoteMapper) TagSets added in v0.9.2 

func (r *RemoteMapper) TagSets() []string

TagSets returns the TagSets

type Service added in v0.9.0 

type Service struct {
	Listener net.Listener

	MetaClient interface {
		ShardOwner(shardID uint64) (string, string, *meta.ShardGroupInfo)
	}

	TSDBStore interface {
		CreateShard(database, policy string, shardID uint64) error
		WriteToShard(shardID uint64, points []models.Point) error
		CreateMapper(shardID uint64, stmt influxql.Statement, chunkSize int) (tsdb.Mapper, error)
	}

	Logger *log.Logger
	// contains filtered or unexported fields
}

Service processes data received over raw TCP connections.

func NewService added in v0.9.0 

func NewService(c Config) *Service

NewService returns a new instance of Service.

func (*Service) Close added in v0.9.0 

func (s *Service) Close() error

Close shuts down the listener and waits for all connections to finish.

func (*Service) Open added in v0.9.0 

func (s *Service) Open() error

Open opens the network listener and begins serving requests.

func (*Service) SetLogger added in v0.9.0 

func (s *Service) SetLogger(l *log.Logger)

SetLogger sets the internal logger to the logger passed in.

type ShardMapper added in v0.9.2 

type ShardMapper struct {
	ForceRemoteMapping bool // All shards treated as remote. Useful for testing.

	Node *influxdb.Node

	MetaClient interface {
		DataNode(id uint64) (ni *meta.NodeInfo, err error)
	}

	TSDBStore interface {
		CreateMapper(shardID uint64, stmt influxql.Statement, chunkSize int) (tsdb.Mapper, error)
	}
	// contains filtered or unexported fields
}

ShardMapper is responsible for providing mappers for requested shards. It is responsible for creating those mappers from the local store, or reaching out to another node on the cluster.

func NewShardMapper added in v0.9.2 

func NewShardMapper(timeout time.Duration) *ShardMapper

NewShardMapper returns a mapper of local and remote shards.

func (*ShardMapper) CreateMapper added in v0.9.2 

func (s *ShardMapper) CreateMapper(sh meta.ShardInfo, stmt influxql.Statement, chunkSize int) (tsdb.Mapper, error)

CreateMapper returns a Mapper for the given shard ID.

type ShardMapping added in v0.9.0 

type ShardMapping struct {
	Points map[uint64][]models.Point  // The points associated with a shard ID
	Shards map[uint64]*meta.ShardInfo // The shards that have been mapped, keyed by shard ID
}

ShardMapping contains a mapping of a shards to a points.

func NewShardMapping added in v0.9.0 

func NewShardMapping() *ShardMapping

NewShardMapping creates an empty ShardMapping

func (*ShardMapping) MapPoint added in v0.9.0 

func (s *ShardMapping) MapPoint(shardInfo *meta.ShardInfo, p models.Point)

MapPoint maps a point to shard

type ShardWriter added in v0.9.0 

type ShardWriter struct {
	MetaClient interface {
		DataNode(id uint64) (ni *meta.NodeInfo, err error)
	}
	// contains filtered or unexported fields
}

ShardWriter writes a set of points to a shard.

func NewShardWriter added in v0.9.0 

func NewShardWriter(timeout time.Duration, maxConnections int) *ShardWriter

NewShardWriter returns a new instance of ShardWriter.

func (*ShardWriter) Close added in v0.9.0 

func (w *ShardWriter) Close() error

Close closes ShardWriter's pool

func (*ShardWriter) WriteShard added in v0.9.0 

func (w *ShardWriter) WriteShard(shardID, ownerID uint64, points []models.Point) error

WriteShard writes time series points to a shard

type WritePointsRequest added in v0.9.0 

type WritePointsRequest struct {
	Database         string
	RetentionPolicy  string
	ConsistencyLevel ConsistencyLevel
	Points           []models.Point
}

WritePointsRequest represents a request to write point data to the cluster

func (*WritePointsRequest) AddPoint added in v0.9.0 

func (w *WritePointsRequest) AddPoint(name string, value interface{}, timestamp time.Time, tags map[string]string)

AddPoint adds a point to the WritePointRequest with field key 'value'

type WriteShardRequest added in v0.9.0 

type WriteShardRequest struct {
	// contains filtered or unexported fields
}

WriteShardRequest represents the a request to write a slice of points to a shard

func (*WriteShardRequest) AddPoint added in v0.9.0 

func (w *WriteShardRequest) AddPoint(name string, value interface{}, timestamp time.Time, tags map[string]string)

AddPoint adds a new time series point

func (*WriteShardRequest) AddPoints added in v0.9.0 

func (w *WriteShardRequest) AddPoints(points []models.Point)

AddPoints adds a new time series point

func (*WriteShardRequest) MarshalBinary added in v0.9.0 

func (w *WriteShardRequest) MarshalBinary() ([]byte, error)

MarshalBinary encodes the object to a binary format.

func (*WriteShardRequest) Points added in v0.9.0 

func (w *WriteShardRequest) Points() []models.Point

Points returns the time series Points

func (*WriteShardRequest) SetShardID added in v0.9.0 

func (w *WriteShardRequest) SetShardID(id uint64)

SetShardID sets the ShardID

func (*WriteShardRequest) ShardID added in v0.9.0 

func (w *WriteShardRequest) ShardID() uint64

ShardID gets the ShardID

func (*WriteShardRequest) UnmarshalBinary added in v0.9.0 

func (w *WriteShardRequest) UnmarshalBinary(buf []byte) error

UnmarshalBinary populates WritePointRequest from a binary format.

type WriteShardResponse added in v0.9.0 

type WriteShardResponse struct {
	// contains filtered or unexported fields
}

WriteShardResponse represents the response returned from a remote WriteShardRequest call

func (*WriteShardResponse) Code added in v0.9.0 

func (w *WriteShardResponse) Code() int

Code returns the Code

func (*WriteShardResponse) MarshalBinary added in v0.9.0 

func (w *WriteShardResponse) MarshalBinary() ([]byte, error)

MarshalBinary encodes the object to a binary format.

func (*WriteShardResponse) Message added in v0.9.0 

func (w *WriteShardResponse) Message() string

Message returns the Message

func (*WriteShardResponse) SetCode added in v0.9.0 

func (w *WriteShardResponse) SetCode(code int)

SetCode sets the Code

func (*WriteShardResponse) SetMessage added in v0.9.0 

func (w *WriteShardResponse) SetMessage(message string)

SetMessage sets the Message

func (*WriteShardResponse) UnmarshalBinary added in v0.9.0 

func (w *WriteShardResponse) UnmarshalBinary(buf []byte) error

UnmarshalBinary populates WritePointRequest from a binary format.

Source Files

View all Source files

Directories

Path Synopsis
Package internal is a generated protocol buffer package.
 Package internal is a generated protocol buffer package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.