statsgod

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 13, 2015 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package statsgod - this library manages authentication methods. All auth currently happens at the metric level. This means that all authentication parameters must be included in the metric string itself before it is parsed.

Package statsgod - This library handles the file-based runtime configuration.

Package statsgod - this library handles the logging during runtime.

Package statsgod - This library is responsible for parsing and defining what a "metric" is that we are going to be aggregating.

Package statsgod - This library handles the relaying of data to a backend storage. This backend could be a webservice like carbon or a filesystem or a mock for testing. All backend implementations should conform to the MetricRelay interface.

Package statsgod - This library is responsible for parsing and defining what a "metric" is that we are going to be aggregating.

Package statsgod - this library manages the different socket listeners that we use to collect metrics.

Package statsgod - This library handles the statistics calculations.

Index

Constants

View Source 
const (
	// AuthTypeNone defines a runtime with no authentication.
	AuthTypeNone = "none"
	// AuthTypeConfigToken is a token-based auth which is retrieved from
	// the main configuration file.
	AuthTypeConfigToken = "token"
)
View Source 
const (
	// ConnPoolTypeTcp is an enum describing a TCP connection pool.
	ConnPoolTypeTcp = iota
	// ConnPoolTypeUnix is an enum describing a Unix Socket connection pool.
	ConnPoolTypeUnix
	// ConnPoolTypeNone is for testing.
	ConnPoolTypeNone
)
View Source 
const (
	// SeparatorNamespaceValue is the character separating the namespace and value
	// in the metric string.
	SeparatorNamespaceValue = ":"
	// SeparatorValueType is the character separating the value and metric type in
	// the metric string.
	SeparatorValueType = "|"
	// SeparatorTypeSample is the character separating the type and an optional
	// sample rate.
	SeparatorTypeSample = "@"
)

Metric strings look like my.namespaced.value:123|g|@0.9

View Source 
const (
	// RuneColon is the rune value for a colon (:).
	RuneColon rune = 58
	// RunePipe is the rune value for a pipe (|).
	RunePipe rune = 124
	// RuneAt is the rune value for an "at" sign (@).
	RuneAt rune = 64
	// RuneSpace is the rune value for a space ( ).
	RuneSpace rune = 32
	// RuneNull is the rune value for an null byte.
	RuneNull rune = 0
)
View Source 
const (
	// MetricTypeCounter describes a counter, sent as "c"
	MetricTypeCounter = 0
	// MetricTypeGauge describes a gauge, sent as "g"
	MetricTypeGauge = 1
	// MetricTypeSet describes a set, set as "s"
	MetricTypeSet = 2
	// MetricTypeTimer describes a timer, set as "ms"
	MetricTypeTimer = 3
	// MetricTypeUnknown describes a malformed metric type
	MetricTypeUnknown = 4
)
View Source 
const (
	// RelayTypeCarbon is an enum describing a carbon backend relay.
	RelayTypeCarbon = "carbon"
	// RelayTypeMock is an enum describing a mock backend relay.
	RelayTypeMock = "mock"
	// NamespaceTypeCounter is an enum for counter namespacing.
	NamespaceTypeCounter = iota
	// NamespaceTypeGauge is an enum for gauge namespacing.
	NamespaceTypeGauge
	// NamespaceTypeRate is an enum for rate namespacing.
	NamespaceTypeRate
	// NamespaceTypeSet is an enum for rate namespacing.
	NamespaceTypeSet
	// NamespaceTypeTimer is an enum for timer namespacing.
	NamespaceTypeTimer
	// Megabyte represents the number of bytes in a megabyte.
	Megabyte = 1048576
)
View Source 
const (
	SocketTypeUdp = iota
	SocketTypeTcp
	SocketTypeUnix
)

Enumeration of the socket types.

View Source 
const (
	// MaximumMetricLength is the number of runes allowed in a metric string.
	MaximumMetricLength = 512
)
View Source 
const (
	// MinimumLengthMessage is the shortest message (or fragment) we can parse.
	MinimumLengthMessage = 1
)
View Source 
const (
	NilConnPanicMsg = "Use of a nil connection."
)

NilConnPanicMsg is the message that the NilConn implementation sends in panic.

Variables

This section is empty.

Functions

func AggregateMetric 

func AggregateMetric(metrics map[string]Metric, metric Metric)

AggregateMetric adds the metric to the specified storage map or aggregates it with an existing metric which has the same namespace.

func BlockForSocket 

func BlockForSocket(socket Socket, timeout time.Duration)

BlockForSocket blocks until the specified socket is active.

func GetHostname 

func GetHostname(defaultValue string) (hostname string)

GetHostname determines the current hostname if the provided default is empty.

func ListenForSignals 

func ListenForSignals(finishChannel chan int, config *ConfigValues, configFile *string, logger Logger)

ListenForSignals will listen for quit/reload signals and respond accordingly.

func ParseMetrics 

func ParseMetrics(parseChannel chan string, relayChannel chan *Metric, auth Auth, logger Logger, quit *bool)

ParseMetrics parses the strings received from clients and creates Metric structures.

func PrepareFlushMetrics 

func PrepareFlushMetrics(metrics map[string]Metric, config *ConfigValues, flushStart time.Time, flushStop time.Time, flushCount int)

PrepareFlushMetrics creates metrics that represent the speed and size of the flushes.

func PrepareRuntimeMetrics 

func PrepareRuntimeMetrics(metrics map[string]Metric, config *ConfigValues)

PrepareRuntimeMetrics creates key runtime metrics to monitor the health of the system.

func ProcessMetric 

func ProcessMetric(metric *Metric, flushDuration time.Duration, quantiles []int, logger Logger)

ProcessMetric will create additional calculations based on the type of metric.

func RelayAllMetrics 

func RelayAllMetrics(relay MetricRelay, metrics map[string]Metric, logger Logger)

RelayAllMetrics is a helper to iterate over a Metric map and flush all to the relay.

func RelayMetrics 

func RelayMetrics(relay MetricRelay, relayChannel chan *Metric, logger Logger, config *ConfigValues, quit *bool)

RelayMetrics relays the metrics in-memory to the permanent storage facility. At this point we are receiving Metric structures from a channel that need to be aggregated by the specified namespace. We do this immediately, then when the specified flush interval passes, we send aggregated metrics to storage.

Types

type Auth 

type Auth interface {
	// Authenticate takes a metric string and authenticates it. The metric
	// is passed by reference in case the Auth object needs to manipulate
	// it, such as removing a token.
	Authenticate(metric *string) (bool, error)
}

Auth is an interface describing statsgod authentication objects.

func CreateAuth 

func CreateAuth(config ConfigValues) Auth

CreateAuth is a factory to create an Auth object.

type AuthConfigToken 

type AuthConfigToken struct {
	// Tokens contains a list of valid auth tokens with a token as the key
	// which maps to a boolean defining the validity of the token.
	Tokens map[string]bool
}

AuthConfigToken checks the configuration file for a valid auth token.

func (AuthConfigToken) Authenticate 

func (a AuthConfigToken) Authenticate(metric *string) (bool, error)

Authenticate conforms to Auth.Authenticate()

type AuthNone 

type AuthNone struct {
}

AuthNone does nothing and allows all traffic.

func (AuthNone) Authenticate 

func (a AuthNone) Authenticate(metric *string) (bool, error)

Authenticate conforms to Auth.Authenticate()

type CarbonRelay 

type CarbonRelay struct {
	FlushInterval  time.Duration
	Percentile     []int
	ConnectionPool *ConnectionPool
	Prefixes       map[int]string
	Suffixes       map[int]string
}

CarbonRelay implements MetricRelay.

func (CarbonRelay) ApplyPrefixAndSuffix 

func (c CarbonRelay) ApplyPrefixAndSuffix(namespace string, metricType int) string

ApplyPrefixAndSuffix interpolates the configured prefix and suffix with the metric namespace string.

func (CarbonRelay) Relay 

func (c CarbonRelay) Relay(metric Metric, logger Logger) bool

Relay implements MetricRelay::Relay().

func (*CarbonRelay) SetPrefixesAndSuffixes 

func (c *CarbonRelay) SetPrefixesAndSuffixes(config ConfigValues)

SetPrefixesAndSuffixes is a helper to set the prefixes and suffixes from config when relaying data.

type ConfigValues 

type ConfigValues struct {
	Service struct {
		Name     string
		Debug    bool
		Auth     string
		Tokens   map[string]bool
		Hostname string
	}
	Connection struct {
		Tcp struct {
			Enabled bool
			Host    string
			Port    int
		}
		Udp struct {
			Enabled   bool
			Host      string
			Port      int
			Maxpacket int
		}
		Unix struct {
			Enabled bool
			File    string
		}
	}
	Relay struct {
		Type        string
		Concurrency int
		Timeout     time.Duration
		Flush       time.Duration
	}
	Namespace struct {
		Prefix   string
		Prefixes struct {
			Counters string
			Gauges   string
			Rates    string
			Sets     string
			Timers   string
		}
		Suffix   string
		Suffixes struct {
			Counters string
			Gauges   string
			Rates    string
			Sets     string
			Timers   string
		}
	}
	Carbon struct {
		Host string
		Port int
	}
	Stats struct {
		Percentile []int
	}
	Debug struct {
		Verbose bool
		Receipt bool
		Profile bool
		Relay   bool
	}
}

ConfigValues describes the data type that configuration is loaded into. The values from the YAML config file map directly to these values. e.g.

service: 

name: statsgod
debug: false

Map to: config.Service.Name = "statsgod" config.Service.Debug = false

All values specified in the ConfigValues struct should also have a default value set in LoadFile() to ensure a safe runtime environment.

func CreateConfig 

func CreateConfig(filePath string) (ConfigValues, error)

CreateConfig is a factory for creating ConfigValues.

func (*ConfigValues) LoadFile 

func (config *ConfigValues) LoadFile(filePath string) error

LoadFile will read configuration from a specified file.

type ConnectionPool 

type ConnectionPool struct {
	// Size indicates the number of connections to keep open.
	Size int
	// Connections is the channel to push new/reused connections onto.
	Connections chan net.Conn
	// Addr is the string representing the address of the socket.
	Addr string
	// Type is the type of connection the pool will make.
	Type int
	// Timeout is the amount of time to wait for a connection.
	Timeout time.Duration
	// ErrorCount tracks the number of connection errors that have occured.
	ErrorCount int
}

ConnectionPool maintains a channel of connections to a remote host.

func CreateConnectionPool 

func CreateConnectionPool(size int, addr string, connType int, timeout time.Duration, logger Logger) (*ConnectionPool, error)

CreateConnectionPool creates instances of ConnectionPool.

func (*ConnectionPool) CreateConnection 

func (pool *ConnectionPool) CreateConnection(logger Logger) (bool, error)

CreateConnection attempts to contact the remote relay host.

func (*ConnectionPool) GetConnection 

func (pool *ConnectionPool) GetConnection(logger Logger) (net.Conn, error)

GetConnection retrieves a connection from the pool.

func (*ConnectionPool) ReleaseConnection 

func (pool *ConnectionPool) ReleaseConnection(conn net.Conn, recreate bool, logger Logger) (bool, error)

ReleaseConnection releases a connection back to the pool.

type Logger 

type Logger struct {
	// Trace log level.
	Trace *log.Logger
	// Info log level.
	Info *log.Logger
	// Warning log level.
	Warning *log.Logger
	// Error log level.
	Error *log.Logger
}

Logger is a container for our log levels.

func CreateLogger 

func CreateLogger(
	traceHandle io.Writer,
	infoHandle io.Writer,
	warningHandle io.Writer,
	errorHandle io.Writer) *Logger

CreateLogger is a factory to instantiate a Logger struct.

type Metric 

type Metric struct {
	Key             string           // Name of the metric.
	MetricType      int              // What type of metric is it (gauge, counter, timer)
	TotalHits       float64          // Number of times it has been used.
	LastValue       float64          // The last value stored.
	ValuesPerSecond float64          // The number of values per second.
	MinValue        float64          // The min value.
	MaxValue        float64          // The max value.
	MeanValue       float64          // The cumulative mean.
	MedianValue     float64          // The cumulative median.
	Quantiles       []MetricQuantile // A list of quantile calculations.
	AllValues       ValueSlice       // All of the values.
	FlushTime       int              // What time are we sending Graphite?
	LastFlushed     int              // When did we last flush this out?
	SampleRate      float64          // The sample rate of the metric.
}

Metric is our main data type.

func CreateSimpleMetric 

func CreateSimpleMetric(key string, value float64, metricType int) *Metric

CreateSimpleMetric is a helper to quickly create a metric with the minimum information.

func ParseMetricString 

func ParseMetricString(metricString string) (*Metric, error)

ParseMetricString parses a metric string, and if it is properly constructed, create a Metric structure. Expects the format [namespace]:[value]|[type]

type MetricQuantile 

type MetricQuantile struct {
	Quantile  int        // The specified percentile.
	Boundary  float64    // The calculated quantile value.
	AllValues ValueSlice // All of the values.
	Mean      float64    // The mean value within the quantile.
	Median    float64    // The median value within the quantile.
	Max       float64    // The maxumum value within the quantile.
	Sum       float64    // The sum value within the quantile.
}

MetricQuantile tracks a specified quantile measurement.

type MetricRelay 

type MetricRelay interface {
	Relay(metric Metric, logger Logger) bool
}

MetricRelay defines the interface for a back end implementation.

func CreateRelay 

func CreateRelay(config ConfigValues, logger Logger) MetricRelay

CreateRelay is a factory for instantiating remote relays.

type MockRelay 

type MockRelay struct {
	FlushInterval time.Duration
	Percentile    []int
}

MockRelay implements MetricRelay.

func (MockRelay) Relay 

func (c MockRelay) Relay(metric Metric, logger Logger) bool

Relay implements MetricRelay::Relay().

type NilConn 

type NilConn struct{}

NilConn is an implementation of the net.Conn interface. We use it so that we can return a value from GetConnection. Without the NilConn, it would attempt to convert nil into a net.Conn which causes an error. In this case we can handle it gracefully and panic for any access instead of attemting to access a nil pointer.

func (NilConn) Close 

func (c NilConn) Close() error

Close implements net.Conn.Close()

func (NilConn) LocalAddr 

func (c NilConn) LocalAddr() net.Addr

LocalAddr implements net.Conn.LocalAddr()

func (NilConn) Read 

func (c NilConn) Read(b []byte) (n int, err error)

Read implements net.Conn.Read()

func (NilConn) RemoteAddr 

func (c NilConn) RemoteAddr() net.Addr

RemoteAddr implements net.Conn.RemoteAddr()

func (NilConn) SetDeadline 

func (c NilConn) SetDeadline(t time.Time) error

SetDeadline implements net.Conn.SetDeadline()

func (NilConn) SetReadDeadline 

func (c NilConn) SetReadDeadline(t time.Time) error

SetReadDeadline implements net.Conn.SetReadDeadline()

func (NilConn) SetWriteDeadline 

func (c NilConn) SetWriteDeadline(t time.Time) error

SetWriteDeadline implements net.Conn.SetWriteDeadline()

func (NilConn) Write 

func (c NilConn) Write(b []byte) (n int, err error)

Write implements net.Conn.Write()

type Socket 

type Socket interface {
	Listen(parseChannel chan string, logger Logger, config *ConfigValues)
	Close(logger Logger)
	GetAddr() string
	SocketIsActive() bool
}

Socket is the interface for all of our socket types.

func CreateSocket 

func CreateSocket(socketType int, addr string) Socket

CreateSocket is a factory to create Socket structs.

type SocketTcp 

type SocketTcp struct {
	Addr     string
	Listener net.Listener
}

SocketTcp contains the required fields to start a TCP socket.

func (*SocketTcp) Close 

func (l *SocketTcp) Close(logger Logger)

Close closes an open socket. Conforms to Socket.Close().

func (*SocketTcp) GetAddr 

func (l *SocketTcp) GetAddr() string

GetAddr retrieves a net compatible address string. Conforms to Socket.GetAddr().

func (*SocketTcp) Listen 

func (l *SocketTcp) Listen(parseChannel chan string, logger Logger, config *ConfigValues)

Listen listens on a socket and populates a channel with received messages. Conforms to Socket.Listen().

func (*SocketTcp) SocketIsActive 

func (l *SocketTcp) SocketIsActive() bool

SocketIsActive determines if the socket is listening. Conforms to Socket.SocketIsActive()

type SocketUdp 

type SocketUdp struct {
	Addr     string
	Listener *net.UDPConn
}

SocketUdp contains the fields required to start a UDP socket.

func (*SocketUdp) Close 

func (l *SocketUdp) Close(logger Logger)

Close closes an open socket. Conforms to Socket.Close().

func (*SocketUdp) GetAddr 

func (l *SocketUdp) GetAddr() string

GetAddr retrieves a net compatible address string. Conforms to Socket.GetAddr().

func (*SocketUdp) Listen 

func (l *SocketUdp) Listen(parseChannel chan string, logger Logger, config *ConfigValues)

Listen listens on a socket and populates a channel with received messages. Conforms to Socket.Listen().

func (*SocketUdp) SocketIsActive 

func (l *SocketUdp) SocketIsActive() bool

SocketIsActive determines if the socket is listening. Conforms to Socket.SocketIsActive()

type SocketUnix 

type SocketUnix struct {
	Addr     string
	Listener net.Listener
}

SocketUnix contains the fields required to start a Unix socket.

func (*SocketUnix) Close 

func (l *SocketUnix) Close(logger Logger)

Close closes an open socket. Conforms to Socket.Close().

func (*SocketUnix) GetAddr 

func (l *SocketUnix) GetAddr() string

GetAddr retrieves a net compatible address string. Conforms to Socket.GetAddr().

func (*SocketUnix) Listen 

func (l *SocketUnix) Listen(parseChannel chan string, logger Logger, config *ConfigValues)

Listen listens on a socket and populates a channel with received messages. Conforms to Socket.Listen().

func (*SocketUnix) SocketIsActive 

func (l *SocketUnix) SocketIsActive() bool

SocketIsActive determines if the socket is listening. Conforms to Socket.SocketIsActive()

type ValueSlice 

type ValueSlice []float64

ValueSlice provides a storage for float64 values.

func (ValueSlice) Get 

func (values ValueSlice) Get(i int) float64

Get is a getter for internal values.

func (ValueSlice) Len 

func (values ValueSlice) Len() int

Len gets the length of the internal values.

func (ValueSlice) Less 

func (values ValueSlice) Less(i, j int) bool

Less is used for the sorting interface.

func (ValueSlice) Mean 

func (values ValueSlice) Mean() (mean float64)

Mean finds the mean value from a ValueSlice.

func (ValueSlice) Median 

func (values ValueSlice) Median() (median float64)

Median finds the median value from a ValueSlice.

func (ValueSlice) Minmax 

func (values ValueSlice) Minmax() (min float64, max float64, err error)

Minmax provides the minimum and maximum values for a ValueSlice structure.

func (ValueSlice) Quantile 

func (values ValueSlice) Quantile(quantile float64) float64

Quantile finds a specified quantile value from a ValueSlice.

func (ValueSlice) Sum 

func (values ValueSlice) Sum() float64

Sum finds the total of all values.

func (ValueSlice) Swap 

func (values ValueSlice) Swap(i, j int)

Swap is used for the sorting interface.

func (ValueSlice) UniqueCount 

func (values ValueSlice) UniqueCount() int

UniqueCount provides the number of unique values sent during this period.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.