block

package
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 2, 2020 License: Apache-2.0 Imports: 31 Imported by: 23

Documentation

Overview

Package block contains common functionality for interacting with TSDB blocks in the context of Thanos.

Index

Constants

View Source 
const (
	// MetaFilename is the known JSON filename for meta information.
	MetaFilename = "meta.json"
	// IndexFilename is the known index file for block index.
	IndexFilename = "index"
	// IndexCacheFilename is the canonical name for json index cache file that stores essential information.
	IndexCacheFilename = "index.cache.json"
	// IndexHeaderFilename is the canonical name for binary index header file that stores essential information.
	IndexHeaderFilename = "index-header"
	// ChunksDirname is the known dir name for chunks with compressed samples.
	ChunksDirname = "chunks"

	// DebugMetas is a directory for debug meta files that happen in the past. Useful for debugging.
	DebugMetas = "debug/metas"
)

Variables

View Source 
var (
	ErrorSyncMetaNotFound  = errors.New("meta.json not found")
	ErrorSyncMetaCorrupted = errors.New("meta.json corrupted")
)

Functions

func Delete 

func Delete(ctx context.Context, logger log.Logger, bkt objstore.Bucket, id ulid.ULID) error

Delete removes directory that is meant to be block directory. NOTE: Always prefer this method for deleting blocks.

  • We have to delete block's files in the certain order (meta.json first) to ensure we don't end up with malformed partial blocks. Thanos system handles well partial blocks only if they don't have meta.json. If meta.json is present Thanos assumes valid block.
  • This avoids deleting empty dir (whole bucket) by mistake.

func Download 

func Download(ctx context.Context, logger log.Logger, bucket objstore.Bucket, id ulid.ULID, dst string) error

Download downloads directory that is mean to be block directory.

func DownloadMeta 

func DownloadMeta(ctx context.Context, logger log.Logger, bkt objstore.Bucket, id ulid.ULID) (metadata.Meta, error)

DownloadMeta downloads only meta file from bucket by block ID. TODO(bwplotka): Differentiate between network error & partial upload.

func IgnoreCompleteOutsideChunk 

func IgnoreCompleteOutsideChunk(mint int64, maxt int64, _ *chunks.Meta, curr *chunks.Meta) (bool, error)

func IgnoreDuplicateOutsideChunk 

func IgnoreDuplicateOutsideChunk(_ int64, _ int64, last *chunks.Meta, curr *chunks.Meta) (bool, error)

func IgnoreIssue347OutsideChunk 

func IgnoreIssue347OutsideChunk(_ int64, maxt int64, _ *chunks.Meta, curr *chunks.Meta) (bool, error)

func IsBlockDir 

func IsBlockDir(path string) (id ulid.ULID, ok bool)

func Repair 

func Repair(logger log.Logger, dir string, id ulid.ULID, source metadata.SourceType, ignoreChkFns ...ignoreFnType) (resid ulid.ULID, err error)

Repair open the block with given id in dir and creates a new one with fixed data. It: - removes out of order duplicates - all "complete" outsiders (they will not accessed anyway) - removes all near "complete" outside chunks introduced by https://github.com/prometheus/tsdb/issues/347. Fixable inconsistencies are resolved in the new block. TODO(bplotka): https://github.com/thanos-io/thanos/issues/378.

func Upload 

func Upload(ctx context.Context, logger log.Logger, bkt objstore.Bucket, bdir string) error

Upload uploads block from given block dir that ends with block id. It makes sure cleanup is done on error to avoid partial block uploads. It also verifies basic features of Thanos block. TODO(bplotka): Ensure bucket operations have reasonable backoff retries.

func VerifyIndex 

func VerifyIndex(logger log.Logger, fn string, minTime int64, maxTime int64) error

VerifyIndex does a full run over a block index and verifies that it fulfills the order invariants.

Types

type ConsistencyDelayMetaFilter added in v0.11.0 

type ConsistencyDelayMetaFilter struct {
	// contains filtered or unexported fields
}

ConsistencyDelayMetaFilter is a MetaFetcher filter that filters out blocks that are created before a specified consistency delay.

func NewConsistencyDelayMetaFilter added in v0.11.0 

func NewConsistencyDelayMetaFilter(logger log.Logger, consistencyDelay time.Duration, reg prometheus.Registerer) *ConsistencyDelayMetaFilter

NewConsistencyDelayMetaFilter creates ConsistencyDelayMetaFilter.

func (*ConsistencyDelayMetaFilter) Filter added in v0.11.0 

func (f *ConsistencyDelayMetaFilter) Filter(metas map[ulid.ULID]*metadata.Meta, synced GaugeLabeled, _ bool)

Filter filters out blocks that filters blocks that have are created before a specified consistency delay.

type DeduplicateFilter added in v0.11.0 

type DeduplicateFilter struct {
	// contains filtered or unexported fields
}

DeduplicateFilter is a MetaFetcher filter that filters out older blocks that have exactly the same data.

func NewDeduplicateFilter added in v0.11.0 

func NewDeduplicateFilter() *DeduplicateFilter

NewDeduplicateFilter creates DeduplicateFilter.

func (*DeduplicateFilter) DuplicateIDs added in v0.11.0 

func (f *DeduplicateFilter) DuplicateIDs() []ulid.ULID

DuplicateIDs returns slice of block ids that are filtered out by DeduplicateFilter.

func (*DeduplicateFilter) Filter added in v0.11.0 

func (f *DeduplicateFilter) Filter(metas map[ulid.ULID]*metadata.Meta, synced GaugeLabeled, _ bool)

Filter filters out duplicate blocks that can be formed from two or more overlapping blocks that fully submatches the source blocks of the older blocks.

type GaugeLabeled added in v0.10.0 

type GaugeLabeled interface {
	WithLabelValues(lvs ...string) prometheus.Gauge
}

type LabelShardedMetaFilter added in v0.10.0 

type LabelShardedMetaFilter struct {
	// contains filtered or unexported fields
}

LabelShardedMetaFilter is a MetaFetcher filter that filters out blocks that have no labels after relabelling.

func NewLabelShardedMetaFilter added in v0.10.0 

func NewLabelShardedMetaFilter(relabelConfig []*relabel.Config) *LabelShardedMetaFilter

NewLabelShardedMetaFilter creates LabelShardedMetaFilter.

func (*LabelShardedMetaFilter) Filter added in v0.10.0 

func (f *LabelShardedMetaFilter) Filter(metas map[ulid.ULID]*metadata.Meta, synced GaugeLabeled, _ bool)

Filter filters out blocks that filters blocks that have no labels after relabelling.

type MetaFetcher added in v0.10.0 

type MetaFetcher struct {
	// contains filtered or unexported fields
}

MetaFetcher is a struct that synchronizes filtered metadata of all block in the object storage with the local state.

func NewMetaFetcher added in v0.10.0 

func NewMetaFetcher(logger log.Logger, concurrency int, bkt objstore.BucketReader, dir string, r prometheus.Registerer, filters ...MetaFetcherFilter) (*MetaFetcher, error)

NewMetaFetcher constructs MetaFetcher.

func (*MetaFetcher) Fetch added in v0.10.0 

func (s *MetaFetcher) Fetch(ctx context.Context) (metas map[ulid.ULID]*metadata.Meta, partial map[ulid.ULID]error, err error)

Fetch returns all block metas as well as partial blocks (blocks without or with corrupted meta file) from the bucket. It's caller responsibility to not change the returned metadata files. Maps can be modified.

Returned error indicates a failure in fetching metadata. Returned meta can be assumed as correct, with some blocks missing.

type MetaFetcherFilter added in v0.10.0 

type MetaFetcherFilter func(metas map[ulid.ULID]*metadata.Meta, synced GaugeLabeled, incompleteView bool)

type MetadataFetcher added in v0.10.0 

type MetadataFetcher interface {
	Fetch(ctx context.Context) (metas map[ulid.ULID]*metadata.Meta, partial map[ulid.ULID]error, err error)
}

type Node added in v0.11.0 

type Node struct {
	metadata.Meta
	Children []*Node
}

Node type represents a node of a tree.

func NewNode added in v0.11.0 

func NewNode(meta *metadata.Meta) *Node

NewNode creates a new node with children as empty slice.

type Stats 

type Stats struct {
	// TotalSeries represents total number of series in block.
	TotalSeries int
	// OutOfOrderSeries represents number of series that have out of order chunks.
	OutOfOrderSeries int

	// OutOfOrderChunks represents number of chunks that are out of order (older time range is after younger one).
	OutOfOrderChunks int
	// DuplicatedChunks represents number of chunks with same time ranges within same series, potential duplicates.
	DuplicatedChunks int
	// OutsideChunks represents number of all chunks that are before or after time range specified in block meta.
	OutsideChunks int
	// CompleteOutsideChunks is subset of OutsideChunks that will be never accessed. They are completely out of time range specified in block meta.
	CompleteOutsideChunks int
	// Issue347OutsideChunks represents subset of OutsideChunks that are outsiders caused by https://github.com/prometheus/tsdb/issues/347
	// and is something that Thanos handle.
	//
	// Specifically we mean here chunks with minTime == block.maxTime and maxTime > block.MaxTime. These are
	// are segregated into separate counters. These chunks are safe to be deleted, since they are duplicated across 2 blocks.
	Issue347OutsideChunks int
	// OutOfOrderLabels represents the number of postings that contained out
	// of order labels, a bug present in Prometheus 2.8.0 and below.
	OutOfOrderLabels int
}

func GatherIndexIssueStats 

func GatherIndexIssueStats(logger log.Logger, fn string, minTime int64, maxTime int64) (stats Stats, err error)

GatherIndexIssueStats returns useful counters as well as outsider chunks (chunks outside of block time range) that helps to assess index health. It considers https://github.com/prometheus/tsdb/issues/347 as something that Thanos can handle. See Stats.Issue347OutsideChunks for details.

func (Stats) AnyErr 

func (i Stats) AnyErr() error

AnyErr returns error if stats indicates any block issue.

func (Stats) CriticalErr 

func (i Stats) CriticalErr() error

CriticalErr returns error if stats indicates critical block issue, that might solved only by manual repair procedure.

func (Stats) Issue347OutsideChunksErr 

func (i Stats) Issue347OutsideChunksErr() error

Issue347OutsideChunksErr returns error if stats indicates issue347 block issue, that is repaired explicitly before compaction (on plan block).

func (Stats) PrometheusIssue5372Err added in v0.7.0 

func (i Stats) PrometheusIssue5372Err() error

PrometheusIssue5372Err returns an error if the Stats object indicates postings with out of order labels. This is corrected by Prometheus Issue #5372 and affects Prometheus versions 2.8.0 and below.

type TimePartitionMetaFilter added in v0.10.0 

type TimePartitionMetaFilter struct {
	// contains filtered or unexported fields
}

TimePartitionMetaFilter is a MetaFetcher filter that filters out blocks that are outside of specified time range.

func NewTimePartitionMetaFilter added in v0.10.0 

func NewTimePartitionMetaFilter(MinTime, MaxTime model.TimeOrDurationValue) *TimePartitionMetaFilter

NewTimePartitionMetaFilter creates TimePartitionMetaFilter.

func (*TimePartitionMetaFilter) Filter added in v0.10.0 

func (f *TimePartitionMetaFilter) Filter(metas map[ulid.ULID]*metadata.Meta, synced GaugeLabeled, _ bool)

Filter filters out blocks that are outside of specified time range.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.