remote

package
v1.29.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 8, 2021 License: Apache-2.0 Imports: 52 Imported by: 0

Documentation

Index

Constants

View Source 
const DefaultChunkedReadLimit = 5e+7

DefaultChunkedReadLimit is the default value for the maximum size of the protobuf frame client allows. 50MB is the default. This is equivalent to ~100k full XOR chunks and average labelset.

Variables

View Source 
var UserAgent = fmt.Sprintf("Prometheus/%s", version.Version)

Functions

func DecodeReadRequest added in v1.8.0 

func DecodeReadRequest(r *http.Request) (*prompb.ReadRequest, error)

DecodeReadRequest reads a remote.Request from a http.Request.

func DecodeWriteRequest added in v1.29.3 

func DecodeWriteRequest(r io.Reader) (*prompb.WriteRequest, error)

DecodeWriteRequest from an io.Reader into a prompb.WriteRequest, handling snappy decompression.

func EncodeReadResponse added in v1.8.0 

func EncodeReadResponse(resp *prompb.ReadResponse, w http.ResponseWriter) error

EncodeReadResponse writes a remote.Response to a http.ResponseWriter.

func FromLabelMatchers added in v1.29.3 

func FromLabelMatchers(matchers []*prompb.LabelMatcher) ([]*labels.Matcher, error)

FromLabelMatchers parses protobuf label matchers to Prometheus label matchers.

func FromQueryResult added in v1.8.0 

func FromQueryResult(sortSeries bool, res *prompb.QueryResult) storage.SeriesSet

FromQueryResult unpacks and sorts a QueryResult proto.

func LabelProtosToMetric added in v1.29.3 

func LabelProtosToMetric(labelPairs []*prompb.Label) model.Metric

LabelProtosToMetric unpack a []*prompb.Label to a model.Metric

func MergeLabels added in v1.29.3 

func MergeLabels(primary, secondary []prompb.Label) []prompb.Label

MergeLabels merges two sets of sorted proto labels, preferring those in primary to those in secondary when there is an overlap.

func NegotiateResponseType added in v1.29.3 

func NegotiateResponseType(accepted []prompb.ReadRequest_ResponseType) (prompb.ReadRequest_ResponseType, error)

NegotiateResponseType returns first accepted response type that this server supports. On the empty accepted list we assume that the SAMPLES response type was requested. This is to maintain backward compatibility.

func NewReadHandler added in v1.29.3 

func NewReadHandler(logger log.Logger, r prometheus.Registerer, queryable storage.SampleAndChunkQueryable, config func() config.Config, remoteReadSampleLimit, remoteReadConcurrencyLimit, remoteReadMaxBytesInFrame int) http.Handler

NewReadHandler creates a http.Handler that accepts remote read requests and writes them to the provided queryable.

func NewSampleAndChunkQueryableClient added in v1.29.3 

func NewSampleAndChunkQueryableClient(
	c ReadClient,
	externalLabels labels.Labels,
	requiredMatchers []*labels.Matcher,
	readRecent bool,
	callback startTimeCallback,
) storage.SampleAndChunkQueryable

NewSampleAndChunkQueryableClient returns a storage.SampleAndChunkQueryable which queries the given client to select series sets.

func NewWriteHandler added in v1.29.3 

func NewWriteHandler(logger log.Logger, appendable storage.Appendable) http.Handler

NewWriteHandler creates a http.Handler that accepts remote write requests and writes them to the provided appendable.

func StreamChunkedReadResponses added in v1.29.3 

func StreamChunkedReadResponses(
	stream io.Writer,
	queryIndex int64,
	ss storage.ChunkSeriesSet,
	sortedExternalLabels []prompb.Label,
	maxBytesInFrame int,
) (storage.Warnings, error)

StreamChunkedReadResponses iterates over series, builds chunks and streams those to the caller. It expects Series set with populated chunks.

func ToQuery added in v1.8.0 

func ToQuery(from, to int64, matchers []*labels.Matcher, hints *storage.SelectHints) (*prompb.Query, error)

ToQuery builds a Query proto.

func ToQueryResult added in v1.8.0 

func ToQueryResult(ss storage.SeriesSet, sampleLimit int) (*prompb.QueryResult, storage.Warnings, error)

ToQueryResult builds a QueryResult proto.

Types

type ChunkedReader added in v1.29.3 

type ChunkedReader struct {
	// contains filtered or unexported fields
}

ChunkedReader is a buffered reader that expects uvarint delimiter and checksum before each message. It will allocate as much as the biggest frame defined by delimiter (on top of bufio.Reader allocations).

func NewChunkedReader added in v1.29.3 

func NewChunkedReader(r io.Reader, sizeLimit uint64, data []byte) *ChunkedReader

NewChunkedReader constructs a ChunkedReader. It allows passing data slice for byte slice reuse, which will be increased to needed size if smaller.

func (*ChunkedReader) Next added in v1.29.3 

func (r *ChunkedReader) Next() ([]byte, error)

Next returns the next length-delimited record from the input, or io.EOF if there are no more records available. Returns io.ErrUnexpectedEOF if a short record is found, with a length of n but fewer than n bytes of data. Next also verifies the given checksum with Castagnoli polynomial CRC-32 checksum.

NOTE: The slice returned is valid only until a subsequent call to Next. It's a caller's responsibility to copy the returned slice if needed.

func (*ChunkedReader) NextProto added in v1.29.3 

func (r *ChunkedReader) NextProto(pb proto.Message) error

NextProto consumes the next available record by calling r.Next, and decodes it into the protobuf with proto.Unmarshal.

type ChunkedWriter added in v1.29.3 

type ChunkedWriter struct {
	// contains filtered or unexported fields
}

ChunkedWriter is an io.Writer wrapper that allows streaming by adding uvarint delimiter before each write in a form of length of the corresponded byte array.

func NewChunkedWriter added in v1.29.3 

func NewChunkedWriter(w io.Writer, f http.Flusher) *ChunkedWriter

NewChunkedWriter constructs a ChunkedWriter.

func (*ChunkedWriter) Write added in v1.29.3 

func (w *ChunkedWriter) Write(b []byte) (int, error)

Write writes given bytes to the stream and flushes it. Each frame includes:

1. uvarint for the size of the data frame. 2. big-endian uint32 for the Castagnoli polynomial CRC-32 checksum of the data frame. 3. the bytes of the given data.

Write returns number of sent bytes for a given buffer. The number does not include delimiter and checksum bytes.

type Client added in v1.1.0 

type Client struct {
	Client *http.Client
	// contains filtered or unexported fields
}

Client allows reading and writing from/to a remote HTTP endpoint.

func (Client) Endpoint added in v1.29.3 

func (c Client) Endpoint() string

Endpoint is the remote read or write endpoint.

func (Client) Name added in v1.1.0 

func (c Client) Name() string

Name uniquely identifies the client.

func (*Client) Read added in v1.6.0 

func (c *Client) Read(ctx context.Context, query *prompb.Query) (*prompb.QueryResult, error)

Read reads from a remote endpoint.

func (*Client) Store added in v1.1.0 

func (c *Client) Store(ctx context.Context, req []byte) error

Store sends a batch of samples to the HTTP endpoint, the request is the proto marshalled and encoded bytes from codec.go.

type ClientConfig added in v1.8.0 

type ClientConfig struct {
	URL              *config_util.URL
	Timeout          model.Duration
	HTTPClientConfig config_util.HTTPClientConfig
	SigV4Config      *config.SigV4Config
	Headers          map[string]string
	RetryOnRateLimit bool
}

ClientConfig configures a client.

type HTTPError added in v1.29.3 

type HTTPError struct {
	// contains filtered or unexported fields
}

func (HTTPError) Error added in v1.29.3 

func (e HTTPError) Error() string

func (HTTPError) Status added in v1.29.3 

func (e HTTPError) Status() int

type MetadataAppender added in v1.29.3 

type MetadataAppender interface {
	AppendMetadata(context.Context, []scrape.MetricMetadata)
}

MetadataAppender is an interface used by the Metadata Watcher to send metadata, It is read from the scrape manager, on to somewhere else.

type MetadataWatcher added in v1.29.3 

type MetadataWatcher struct {
	// contains filtered or unexported fields
}

MetadataWatcher watches the Scrape Manager for a given WriteMetadataTo.

func NewMetadataWatcher added in v1.29.3 

func NewMetadataWatcher(l log.Logger, mg ReadyScrapeManager, name string, w MetadataAppender, interval model.Duration, deadline time.Duration) *MetadataWatcher

NewMetadataWatcher builds a new MetadataWatcher.

func (*MetadataWatcher) Start added in v1.29.3 

func (mw *MetadataWatcher) Start()

Start the MetadataWatcher.

func (*MetadataWatcher) Stop added in v1.29.3 

func (mw *MetadataWatcher) Stop()

Stop the MetadataWatcher.

type QueueManager added in v1.6.0 

type QueueManager struct {
	// contains filtered or unexported fields
}

QueueManager manages a queue of samples to be sent to the Storage indicated by the provided WriteClient. Implements writeTo interface used by WAL Watcher.

func NewQueueManager added in v1.6.0 

func NewQueueManager(
	metrics *queueManagerMetrics,
	watcherMetrics *wal.WatcherMetrics,
	readerMetrics *wal.LiveReaderMetrics,
	logger log.Logger,
	walDir string,
	samplesIn *ewmaRate,
	cfg config.QueueConfig,
	mCfg config.MetadataConfig,
	externalLabels labels.Labels,
	relabelConfigs []*relabel.Config,
	client WriteClient,
	flushDeadline time.Duration,
	interner *pool,
	highestRecvTimestamp *maxTimestamp,
	sm ReadyScrapeManager,
	enableExemplarRemoteWrite bool,
) *QueueManager

NewQueueManager builds a new QueueManager.

func (*QueueManager) Append added in v1.6.0 

func (t *QueueManager) Append(samples []record.RefSample) bool

Append queues a sample to be sent to the remote storage. Blocks until all samples are enqueued on their shards or a shutdown signal is received.

func (*QueueManager) AppendExemplars added in v1.29.3 

func (t *QueueManager) AppendExemplars(exemplars []record.RefExemplar) bool

func (*QueueManager) AppendMetadata added in v1.29.3 

func (t *QueueManager) AppendMetadata(ctx context.Context, metadata []scrape.MetricMetadata)

AppendMetadata sends metadata the remote storage. Metadata is sent all at once and is not parallelized.

func (*QueueManager) SeriesReset added in v1.29.3 

func (t *QueueManager) SeriesReset(index int)

SeriesReset is used when reading a checkpoint. WAL Watcher should have stored series records with the checkpoints index number, so we can now delete any ref ID's lower than that # from the two maps.

func (*QueueManager) SetClient added in v1.29.3 

func (t *QueueManager) SetClient(c WriteClient)

SetClient updates the client used by a queue. Used when only client specific fields are updated to avoid restarting the queue.

func (*QueueManager) Start added in v1.6.0 

func (t *QueueManager) Start()

Start the queue manager sending samples to the remote storage. Does not block.

func (*QueueManager) Stop added in v1.6.0 

func (t *QueueManager) Stop()

Stop stops sending samples to the remote storage and waits for pending sends to complete.

func (*QueueManager) StoreSeries added in v1.29.3 

func (t *QueueManager) StoreSeries(series []record.RefSeries, index int)

StoreSeries keeps track of which series we know about for lookups when sending samples to remote.

func (*QueueManager) UpdateSeriesSegment added in v1.29.3 

func (t *QueueManager) UpdateSeriesSegment(series []record.RefSeries, index int)

Update the segment number held against the series, so we can trim older ones in SeriesReset.

type ReadClient added in v1.29.3 

type ReadClient interface {
	Read(ctx context.Context, query *prompb.Query) (*prompb.QueryResult, error)
}

ReadClient uses the SAMPLES method of remote read to read series samples from remote server. TODO(bwplotka): Add streamed chunked remote read method as well (https://github.com/mkby/prometheus/issues/5926).

func NewReadClient added in v1.29.3 

func NewReadClient(name string, conf *ClientConfig) (ReadClient, error)

NewReadClient creates a new client for remote read.

type ReadyScrapeManager added in v1.29.3 

type ReadyScrapeManager interface {
	Get() (*scrape.Manager, error)
}

type RecoverableError added in v1.29.3 

type RecoverableError struct {
	// contains filtered or unexported fields
}

type Storage 

type Storage struct {
	// contains filtered or unexported fields
}

Storage represents all the remote read and write endpoints. It implements storage.Storage.

func NewStorage added in v1.29.3 

func NewStorage(l log.Logger, reg prometheus.Registerer, stCallback startTimeCallback, walDir string, flushDeadline time.Duration, sm ReadyScrapeManager) *Storage

NewStorage returns a remote.Storage.

func (*Storage) Appender added in v1.29.3 

func (s *Storage) Appender(ctx context.Context) storage.Appender

Appender implements storage.Storage.

func (*Storage) ApplyConfig 

func (s *Storage) ApplyConfig(conf *config.Config) error

ApplyConfig updates the state as the new config requires.

func (*Storage) ChunkQuerier added in v1.29.3 

func (s *Storage) ChunkQuerier(ctx context.Context, mint, maxt int64) (storage.ChunkQuerier, error)

ChunkQuerier returns a storage.MergeQuerier combining the remote client queriers of each configured remote read endpoint.

func (*Storage) Close added in v1.29.3 

func (s *Storage) Close() error

Close the background processing of the storage queues.

func (*Storage) Querier added in v1.29.3 

func (s *Storage) Querier(ctx context.Context, mint, maxt int64) (storage.Querier, error)

Querier returns a storage.MergeQuerier combining the remote client queriers of each configured remote read endpoint. Returned querier will never return error as all queryables are assumed best effort. Additionally all returned queriers ensure that its Select's SeriesSets have ready data after first `Next` invoke. This is because Prometheus (fanout and secondary queries) can't handle the stream failing half way through by design.

func (*Storage) StartTime added in v1.29.3 

func (s *Storage) StartTime() (int64, error)

StartTime implements the Storage interface.

type Watchable added in v1.29.3 

type Watchable interface {
	TargetsActive() map[string][]*scrape.Target
}

Watchable represents from where we fetch active targets for metadata.

type WriteClient added in v1.1.0 

type WriteClient interface {
	// Store stores the given samples in the remote storage.
	Store(context.Context, []byte) error
	// Name uniquely identifies the remote storage.
	Name() string
	// Endpoint is the remote read or write endpoint for the storage client.
	Endpoint() string
}

WriteClient defines an interface for sending a batch of samples to an external timeseries database.

func NewWriteClient added in v1.1.0 

func NewWriteClient(name string, conf *ClientConfig) (WriteClient, error)

NewWriteClient creates a new client for remote write.

type WriteStorage added in v1.29.3 

type WriteStorage struct {
	// contains filtered or unexported fields
}

WriteStorage represents all the remote write storage.

func NewWriteStorage added in v1.29.3 

func NewWriteStorage(logger log.Logger, reg prometheus.Registerer, walDir string, flushDeadline time.Duration, sm ReadyScrapeManager) *WriteStorage

NewWriteStorage creates and runs a WriteStorage.

func (*WriteStorage) Appender added in v1.29.3 

func (rws *WriteStorage) Appender(_ context.Context) storage.Appender

Appender implements storage.Storage.

func (*WriteStorage) ApplyConfig added in v1.29.3 

func (rws *WriteStorage) ApplyConfig(conf *config.Config) error

ApplyConfig updates the state as the new config requires. Only stop & create queues which have changes.

func (*WriteStorage) Close added in v1.29.3 

func (rws *WriteStorage) Close() error

Close closes the WriteStorage.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.