epoch

package
v0.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 16, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package epoch manages repository epochs. It implements protocol described https://github.com/kopia/kopia/issues/1090 and is intentionally separate from 'content' package to be able to test in isolation.

Index

Constants

View Source 
const (
	EpochManagerIndexUberPrefix = "x"

	EpochMarkerIndexBlobPrefix      blob.ID = EpochManagerIndexUberPrefix + "e"
	UncompactedIndexBlobPrefix      blob.ID = EpochManagerIndexUberPrefix + "n"
	SingleEpochCompactionBlobPrefix blob.ID = EpochManagerIndexUberPrefix + "s"
	RangeCheckpointIndexBlobPrefix  blob.ID = EpochManagerIndexUberPrefix + "r"
	DeletionWatermarkBlobPrefix     blob.ID = EpochManagerIndexUberPrefix + "w"
)

Index blob prefixes.

View Source 
const FirstEpoch = 0

FirstEpoch is the number of the first epoch in a repository.

View Source 
const LatestEpoch = -1

LatestEpoch represents the current epoch number in GetCompleteIndexSet.

Variables

View Source 
var ErrVerySlowIndexWrite = errors.Errorf("extremely slow index write - index write took more than two epochs")

ErrVerySlowIndexWrite is returned by WriteIndex if a write takes more than 2 epochs (usually >48h). This is theoretically possible with laptops going to sleep, etc.

Functions

func UncompactedEpochBlobPrefix 

func UncompactedEpochBlobPrefix(epoch int) blob.ID

UncompactedEpochBlobPrefix returns the prefix for uncompacted blobs of a given epoch.

Types

type CompactionFunc 

type CompactionFunc func(ctx context.Context, blobIDs []blob.ID, outputPrefix blob.ID) error

CompactionFunc merges the given set of index blobs into a new index blob set with a given prefix and writes them out as a set following naming convention established in 'complete_set.go'.

type CurrentSnapshot 

type CurrentSnapshot struct {
	WriteEpoch                 int                     `json:"writeEpoch"`
	UncompactedEpochSets       map[int][]blob.Metadata `json:"unsettled"`
	LongestRangeCheckpointSets []*RangeMetadata        `json:"longestRangeCheckpointSets"`
	SingleEpochCompactionSets  map[int][]blob.Metadata `json:"singleEpochCompactionSets"`
	EpochStartTime             map[int]time.Time       `json:"epochStartTimes"`
	DeletionWatermark          time.Time               `json:"deletionWatermark"`
	ValidUntil                 time.Time               `json:"validUntil"`             // time after which the contents of this struct are no longer valid
	EpochMarkerBlobs           []blob.Metadata         `json:"epochMarkers"`           // list of epoch markers
	DeletionWatermarkBlobs     []blob.Metadata         `json:"deletionWatermarkBlobs"` // list of deletion watermark blobs
}

CurrentSnapshot captures a point-in time snapshot of a repository indexes, including current epoch information and compaction set.

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages repository epochs.

func NewManager 

func NewManager(st blob.Storage, paramProvider ParametersProvider, compactor CompactionFunc, log logging.Logger, timeNow func() time.Time) *Manager

NewManager creates new epoch manager.

func (*Manager) AdvanceDeletionWatermark 

func (e *Manager) AdvanceDeletionWatermark(ctx context.Context, ts time.Time) error

AdvanceDeletionWatermark moves the deletion watermark time to a given timestamp this causes all deleted content entries before given time to be treated as non-existent.

func (*Manager) CleanupMarkers added in v0.16.0 

func (e *Manager) CleanupMarkers(ctx context.Context) error

CleanupMarkers removes superseded watermarks and epoch markers.

func (*Manager) CleanupSupersededIndexes added in v0.9.8 

func (e *Manager) CleanupSupersededIndexes(ctx context.Context) error

CleanupSupersededIndexes cleans up the indexes which have been superseded by compacted ones.

func (*Manager) Current 

func (e *Manager) Current(ctx context.Context) (CurrentSnapshot, error)

Current retrieves current snapshot.

func (*Manager) Flush 

func (e *Manager) Flush()

Flush waits for all in-process compaction work to complete.

func (*Manager) GetCompleteIndexSet 

func (e *Manager) GetCompleteIndexSet(ctx context.Context, maxEpoch int) ([]blob.Metadata, time.Time, error)

GetCompleteIndexSet returns the set of blobs forming a complete index set up to the provided epoch number.

func (*Manager) Invalidate 

func (e *Manager) Invalidate()

Invalidate ensures that all cached index information is discarded.

func (*Manager) MaybeAdvanceWriteEpoch added in v0.16.0 

func (e *Manager) MaybeAdvanceWriteEpoch(ctx context.Context) error

MaybeAdvanceWriteEpoch writes a new write epoch marker when a new write epoch should be started, otherwise it does not do anything.

func (*Manager) MaybeCompactSingleEpoch added in v0.16.0 

func (e *Manager) MaybeCompactSingleEpoch(ctx context.Context) error

MaybeCompactSingleEpoch compacts the oldest epoch that is eligible for compaction if there is one.

func (*Manager) MaybeGenerateRangeCheckpoint added in v0.16.0 

func (e *Manager) MaybeGenerateRangeCheckpoint(ctx context.Context) error

MaybeGenerateRangeCheckpoint may create a new range index for all the individual epochs covered by the new range. If there are not enough epochs to create a new range, then a range index is not created.

func (*Manager) Refresh 

func (e *Manager) Refresh(ctx context.Context) error

Refresh refreshes information about current epoch.

func (*Manager) WriteIndex 

func (e *Manager) WriteIndex(ctx context.Context, dataShards map[blob.ID]blob.Bytes) ([]blob.Metadata, error)

WriteIndex writes new index blob by picking the appropriate prefix based on current epoch.

type Parameters 

type Parameters struct {
	// whether epoch manager is enabled, must be true.
	Enabled bool `json:"Enabled"`

	// how frequently each client will list blobs to determine the current epoch.
	EpochRefreshFrequency time.Duration `json:"EpochRefreshFrequency"`

	// number of epochs between full checkpoints.
	FullCheckpointFrequency int `json:"FullCheckpointFrequency"`

	// do not delete uncompacted blobs if the corresponding compacted blob age is less than this.
	CleanupSafetyMargin time.Duration `json:"CleanupSafetyMargin"`

	// minimum duration of an epoch
	MinEpochDuration time.Duration `json:"MinEpochDuration"`

	// advance epoch if number of files exceeds this
	EpochAdvanceOnCountThreshold int `json:"EpochAdvanceOnCountThreshold"`

	// advance epoch if total size of files exceeds this.
	EpochAdvanceOnTotalSizeBytesThreshold int64 `json:"EpochAdvanceOnTotalSizeBytesThreshold"`

	// number of blobs to delete in parallel during cleanup
	DeleteParallelism int `json:"DeleteParallelism"`
}

Parameters encapsulates all parameters that influence the behavior of epoch manager.

Note as a historical mistake, JSON tags are not camelCase, but rather PascalCase. We can't change that since the parameters are stored in a repository.

func DefaultParameters 

func DefaultParameters() Parameters

DefaultParameters contains default epoch manager parameters.

func (*Parameters) GetEpochAdvanceOnCountThreshold added in v0.12.0 

func (p *Parameters) GetEpochAdvanceOnCountThreshold() int

GetEpochAdvanceOnCountThreshold returns the number of files above which epoch should be advanced.

func (*Parameters) GetEpochAdvanceOnTotalSizeBytesThreshold added in v0.12.0 

func (p *Parameters) GetEpochAdvanceOnTotalSizeBytesThreshold() int64

GetEpochAdvanceOnTotalSizeBytesThreshold returns the total size of files above which the epoch should be advanced.

func (*Parameters) GetEpochCleanupSafetyMargin added in v0.12.0 

func (p *Parameters) GetEpochCleanupSafetyMargin() time.Duration

GetEpochCleanupSafetyMargin returns safety margin to prevent uncompacted blobs from being deleted if the corresponding compacted blob age is less than this.

func (*Parameters) GetEpochDeleteParallelism added in v0.12.0 

func (p *Parameters) GetEpochDeleteParallelism() int

GetEpochDeleteParallelism returns the number of blobs to delete in parallel during cleanup.

func (*Parameters) GetEpochFullCheckpointFrequency added in v0.12.0 

func (p *Parameters) GetEpochFullCheckpointFrequency() int

GetEpochFullCheckpointFrequency returns the number of epochs between full checkpoints.

func (*Parameters) GetEpochManagerEnabled added in v0.12.0 

func (p *Parameters) GetEpochManagerEnabled() bool

GetEpochManagerEnabled returns whether epoch manager is enabled, must be true.

func (*Parameters) GetEpochRefreshFrequency added in v0.12.0 

func (p *Parameters) GetEpochRefreshFrequency() time.Duration

GetEpochRefreshFrequency determines how frequently each client will list blobs to determine the current epoch.

func (*Parameters) GetMinEpochDuration added in v0.12.0 

func (p *Parameters) GetMinEpochDuration() time.Duration

GetMinEpochDuration returns the minimum duration of an epoch.

func (*Parameters) Validate 

func (p *Parameters) Validate() error

Validate validates epoch parameters.

type ParametersProvider added in v0.12.0 

type ParametersProvider interface {
	GetParameters(ctx context.Context) (*Parameters, error)
}

ParametersProvider provides epoch manager parameters.

type RangeMetadata 

type RangeMetadata struct {
	MinEpoch int             `json:"min"`
	MaxEpoch int             `json:"max"`
	Blobs    []blob.Metadata `json:"blobs"`
}

RangeMetadata represents a range of indexes for [min,max] epoch range. Both min and max are inclusive.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.