shipper

Documentation

Overview

Package shipper detects directories on the local file system and uploads them to a block storage.

Index

• Constants
• func WriteMetaFile(logger log.Logger, path string, meta *Meta) error
• type Meta
  • func ReadMetaFile(path string) (*Meta, error)
• type Shipper
  • func New(logger log.Logger, r prometheus.Registerer, dir string, bucket objstore.Bucket, ...) *Shipper
  • func (s *Shipper) SetLabels(lbls labels.Labels)
  • func (s *Shipper) Sync(ctx context.Context) (uploaded int, err error)
  • func (s *Shipper) Timestamps() (minTime, maxSyncTime int64, err error)
  • func (s *Shipper) UploadedBlocks() map[ulid.ULID]struct{}

Constants

const (
	// DefaultMetaFilename is the default JSON filename for meta information.
	DefaultMetaFilename = "thanos.shipper.json"

	// MetaVersion1 represents 1 version of meta.
	MetaVersion1 = 1
)

Functions

func WriteMetaFile

func WriteMetaFile(logger log.Logger, path string, meta *Meta) error

WriteMetaFile writes the given meta into <dir>/thanos.shipper.json.

Types

type Meta

type Meta struct {
	Version  int         `json:"version"`
	Uploaded []ulid.ULID `json:"uploaded"`
}

Meta defines the format thanos.shipper.json file that the shipper places in the data directory.

func ReadMetaFile

func ReadMetaFile(path string) (*Meta, error)

ReadMetaFile reads the given meta from <dir>/thanos.shipper.json.

type Shipper

type Shipper struct {
	// contains filtered or unexported fields
}

Shipper watches a directory for matching files and directories and uploads them to a remote data store.

func New

func New(
	logger log.Logger,
	r prometheus.Registerer,
	dir string,
	bucket objstore.Bucket,
	lbls func() labels.Labels,
	source metadata.SourceType,
	uploadCompactedFunc func() bool,
	allowOutOfOrderUploads bool,
	hashFunc metadata.HashFunc,
	metaFileName string,
) *Shipper

New creates a new shipper that detects new TSDB blocks in dir and uploads them to remote if necessary. It attaches the Thanos metadata section in each meta JSON file. If uploadCompacted is enabled, it also uploads compacted blocks which are already in filesystem.

func (*Shipper) SetLabels

func (s *Shipper) SetLabels(lbls labels.Labels)

func (*Shipper) Sync

func (s *Shipper) Sync(ctx context.Context) (uploaded int, err error)

Sync performs a single synchronization, which ensures all non-compacted local blocks have been uploaded to the object bucket once.

If uploaded.

It is not concurrency-safe, however it is compactor-safe (running concurrently with compactor is ok).

func (*Shipper) Timestamps

func (s *Shipper) Timestamps() (minTime, maxSyncTime int64, err error)

Timestamps returns the minimum timestamp for which data is available and the highest timestamp of blocks that were successfully uploaded.

func (*Shipper) UploadedBlocks

func (s *Shipper) UploadedBlocks() map[ulid.ULID]struct{}