repair

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 19, 2021 License: Apache-2.0 Imports: 12 Imported by: 2

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type AggregatePeerMetadataComparisonResult added in v1.2.0 

type AggregatePeerMetadataComparisonResult struct {
	// ComparedDifferingPercent is the percent of blocks that mismatched from peers to local origin.
	ComparedDifferingPercent float64

	// ComparedBlocks returns the total number of blocks compared.
	ComparedBlocks int64

	// ComparedDifferingBlocks returns the number of differing blocks (mismatch + missing + extra).
	ComparedDifferingBlocks int64

	// ComparedMismatchBlocks returns the number of mismatching blocks (either size or checksum).
	ComparedMismatchBlocks int64

	// ComparedMissingBlocks returns the number of missing blocks.
	ComparedMissingBlocks int64

	// ComparedExtraBlocks returns the number of extra blocks.
	ComparedExtraBlocks int64
}

AggregatePeerMetadataComparisonResult captures an aggregate metadata comparison result of all peers relative to the local origin node.

type CopyFn 

type CopyFn func(ident.ID) ident.ID

CopyFn is the copy key function to execute when copying the key.

type EqualsFn 

type EqualsFn func(ident.ID, ident.ID) bool

EqualsFn is the equals key function to execute when detecting equality of a key.

type FinalizeFn 

type FinalizeFn func(ident.ID)

FinalizeFn is the finalize key function to execute when finished with a key.

type HashFn 

type HashFn func(ident.ID) MapHash

HashFn is the hash function to execute when hashing a key.

type Map 

type Map struct {
	// contains filtered or unexported fields
}

Map uses the genny package to provide a generic hash map that can be specialized by running the following command from this root of the repository: ``` make hashmap-gen pkg=outpkg key_type=Type value_type=Type out_dir=/tmp ``` Or if you would like to use bytes or ident.ID as keys you can use the partially specialized maps to generate your own maps as well: ``` make byteshashmap-gen pkg=outpkg value_type=Type out_dir=/tmp make idhashmap-gen pkg=outpkg value_type=Type out_dir=/tmp ``` This will output to stdout the generated source file to use for your map. It uses linear probing by incrementing the number of the hash created when hashing the identifier if there is a collision. Map is a value type and not an interface to allow for less painful upgrades when adding/removing methods, it is not likely to need mocking so an interface would not be super useful either.

func NewMap 

func NewMap(opts MapOptions) *Map

NewMap returns a new byte keyed map.

func (*Map) Contains 

func (m *Map) Contains(k ident.ID) bool

Contains returns true if value exists for key, false otherwise, it is shorthand for a call to Get that doesn't return the value.

func (*Map) Delete 

func (m *Map) Delete(k ident.ID)

Delete will remove a value set in the map for the specified key.

func (*Map) Get 

func (m *Map) Get(k ident.ID) (ReplicaSeriesBlocksMetadata, bool)

Get returns a value in the map for an identifier if found.

func (*Map) Iter 

func (m *Map) Iter() map[MapHash]MapEntry

Iter provides the underlying map to allow for using a native Go for loop to iterate the map, however callers should only ever read and not write the map.

func (*Map) Len 

func (m *Map) Len() int

Len returns the number of map entries in the map.

func (*Map) Reallocate 

func (m *Map) Reallocate()

Reallocate will avoid deleting all keys and reallocate a new map, this is useful if you believe you have a large map and will not need to grow back to a similar size.

func (*Map) Reset 

func (m *Map) Reset()

Reset will reset the map by simply deleting all keys to avoid allocating a new map.

func (*Map) Set 

func (m *Map) Set(k ident.ID, v ReplicaSeriesBlocksMetadata)

Set will set the value for an identifier.

func (*Map) SetUnsafe 

func (m *Map) SetUnsafe(k ident.ID, v ReplicaSeriesBlocksMetadata, opts SetUnsafeOptions)

SetUnsafe will set the value for an identifier with unsafe options for how the map treats the key.

type MapEntry 

type MapEntry struct {
	// contains filtered or unexported fields
}

MapEntry is an entry in the map, this is public to support iterating over the map using a native Go for loop.

func (MapEntry) Key 

func (e MapEntry) Key() ident.ID

Key returns the map entry key.

func (MapEntry) Value 

func (e MapEntry) Value() ReplicaSeriesBlocksMetadata

Value returns the map entry value.

type MapHash 

type MapHash uint64

MapHash is the hash for a given map entry, this is public to support iterating over the map using a native Go for loop.

type MapOptions 

type MapOptions struct {
	InitialSize int
	KeyCopyPool pool.BytesPool
}

MapOptions provides options used when created the map.

type MetadataComparisonResult 

type MetadataComparisonResult struct {
	// NumSeries returns the total number of series.
	NumSeries int64

	// NumBlocks returns the total number of blocks.
	NumBlocks int64

	// SizeResult returns the size differences.
	SizeDifferences ReplicaSeriesMetadata

	// ChecksumDifferences returns the checksum differences.
	ChecksumDifferences ReplicaSeriesMetadata

	// PeerMetadataComparisonResults the results comparative to each peer.
	PeerMetadataComparisonResults PeerMetadataComparisonResults
}

MetadataComparisonResult captures metadata comparison results.

type Options 

type Options interface {
	// SetType sets the type of repair to run.
	SetType(value Type) Options

	// Type returns the type of repair to run.
	Type() Type

	// SetStrategy sets the repair strategy.
	SetStrategy(value Strategy) Options

	// Strategy returns the repair strategy.
	Strategy() Strategy

	// SetForce sets whether to force repairs to run for all namespaces.
	SetForce(value bool) Options

	// Force returns whether to force repairs to run for all namespaces.
	Force() bool

	// SetAdminClient sets the admin client.
	SetAdminClients(value []client.AdminClient) Options

	// AdminClient returns the admin client.
	AdminClients() []client.AdminClient

	// SetRepairConsistencyLevel sets the repair read level consistency
	// for which to repair shards with.
	SetRepairConsistencyLevel(value topology.ReadConsistencyLevel) Options

	// RepairConsistencyLevel returns the repair read level consistency
	// for which to repair shards with.
	RepairConsistencyLevel() topology.ReadConsistencyLevel

	// SetRepairShardConcurrency sets the concurrency in which to repair shards with.
	SetRepairShardConcurrency(value int) Options

	// RepairShardConcurrency returns the concurrency in which to repair shards with.
	RepairShardConcurrency() int

	// SetRepairCheckInterval sets the repair check interval.
	SetRepairCheckInterval(value time.Duration) Options

	// RepairCheckInterval returns the repair check interval.
	RepairCheckInterval() time.Duration

	// SetRepairThrottle sets the repair throttle.
	SetRepairThrottle(value time.Duration) Options

	// RepairThrottle returns the repair throttle.
	RepairThrottle() time.Duration

	// SetReplicaMetadataSlicePool sets the replicaMetadataSlice pool.
	SetReplicaMetadataSlicePool(value ReplicaMetadataSlicePool) Options

	// ReplicaMetadataSlicePool returns the replicaMetadataSlice pool.
	ReplicaMetadataSlicePool() ReplicaMetadataSlicePool

	// SetResultOptions sets the result options.
	SetResultOptions(value result.Options) Options

	// ResultOptions returns the result options.
	ResultOptions() result.Options

	// SetDebugShadowComparisonsEnabled sets whether debug shadow comparisons are enabled.
	SetDebugShadowComparisonsEnabled(value bool) Options

	// DebugShadowComparisonsEnabled returns whether debug shadow comparisons are enabled.
	DebugShadowComparisonsEnabled() bool

	// SetDebugShadowComparisonsPercentage sets the debug shadow comparisons percentage.
	SetDebugShadowComparisonsPercentage(value float64) Options

	// DebugShadowComparisonsPercentage returns the debug shadow comparisons percentage.
	DebugShadowComparisonsPercentage() float64

	// Validate checks if the options are valid.
	Validate() error
}

Options are the repair options.

func NewOptions 

func NewOptions() Options

NewOptions creates new bootstrap options

type PeerMetadataComparisonResult added in v1.2.0 

type PeerMetadataComparisonResult struct {
	// ID is the peer ID.
	ID string

	// ComparedBlocks returns the total number of blocks.
	ComparedBlocks int64

	// ComparedDifferingBlocks returns the number of differing blocks (mismatch + missing + extra).
	ComparedDifferingBlocks int64

	// ComparedMismatchBlocks returns the number of mismatching blocks (either size or checksum).
	ComparedMismatchBlocks int64

	// ComparedMissingBlocks returns the number of missing blocks.
	ComparedMissingBlocks int64

	// ComparedExtraBlocks returns the number of extra blocks.
	ComparedExtraBlocks int64
}

PeerMetadataComparisonResult captures metadata comparison results relative to the local origin node.

type PeerMetadataComparisonResults added in v1.2.0 

type PeerMetadataComparisonResults []PeerMetadataComparisonResult

PeerMetadataComparisonResults is a slice of PeerMetadataComparisonResult.

func (PeerMetadataComparisonResults) Aggregate added in v1.2.0 

func (r PeerMetadataComparisonResults) Aggregate() AggregatePeerMetadataComparisonResult

Aggregate returns an aggregate result of the PeerMetadataComparisonResults.

type ReplicaBlockMetadata 

type ReplicaBlockMetadata interface {
	// Start is the start time of a block.
	Start() xtime.UnixNano

	// Metadata returns the metadata from all hosts.
	Metadata() []block.ReplicaMetadata

	// Add adds a metadata from a host.
	Add(metadata block.ReplicaMetadata)

	// Close performs cleanup.
	Close()
}

ReplicaBlockMetadata captures the block metadata from hosts in a shard replica set.

func NewReplicaBlockMetadata 

func NewReplicaBlockMetadata(start xtime.UnixNano, p ReplicaMetadataSlice) ReplicaBlockMetadata

NewReplicaBlockMetadata creates a new replica block metadata

type ReplicaBlocksMetadata 

type ReplicaBlocksMetadata interface {
	// NumBlocks returns the total number of blocks.
	NumBlocks() int64

	// Blocks returns the blocks metadata.
	Blocks() map[xtime.UnixNano]ReplicaBlockMetadata

	// Add adds a block metadata.
	Add(block ReplicaBlockMetadata)

	// GetOrAdd returns the blocks metadata for a start time, creating one if it doesn't exist.
	GetOrAdd(start xtime.UnixNano, p ReplicaMetadataSlicePool) ReplicaBlockMetadata

	// Close performs cleanup.
	Close()
}

ReplicaBlocksMetadata captures the blocks metadata from hosts in a shard replica set.

func NewReplicaBlocksMetadata 

func NewReplicaBlocksMetadata() ReplicaBlocksMetadata

NewReplicaBlocksMetadata creates a new replica blocks metadata

type ReplicaMetadataComparer 

type ReplicaMetadataComparer interface {
	// AddLocalMetadata adds metadata from local host.
	AddLocalMetadata(localIter block.FilteredBlocksMetadataIter) error

	// AddPeerMetadata adds metadata from peers.
	AddPeerMetadata(peerIter client.PeerBlockMetadataIter) error

	// Compare returns the metadata differences between local host and peers.
	Compare() MetadataComparisonResult

	// Finalize performs cleanup during close.
	Finalize()
}

ReplicaMetadataComparer compares metadata from hosts in a replica set.

func NewReplicaMetadataComparer 

func NewReplicaMetadataComparer(origin topology.Host, opts Options) ReplicaMetadataComparer

NewReplicaMetadataComparer creates a new replica metadata comparer

type ReplicaMetadataSlice added in v0.11.0 

type ReplicaMetadataSlice interface {
	// Add adds the metadata to the slice.
	Add(metadata block.ReplicaMetadata)

	// Metadata returns the metadata slice.
	Metadata() []block.ReplicaMetadata

	// Reset resets the metadata slice.
	Reset()

	// Close performs cleanup.
	Close()
}

ReplicaMetadataSlice captures a slice of block.ReplicaMetadata.

type ReplicaMetadataSlicePool added in v0.11.0 

type ReplicaMetadataSlicePool interface {
	// Get returns a ReplicaMetadata slice.
	Get() ReplicaMetadataSlice

	// Put puts a ReplicaMetadata slice back to pool.
	Put(m ReplicaMetadataSlice)
}

ReplicaMetadataSlicePool provides a pool for block.ReplicaMetadata slices.

func NewReplicaMetadataSlicePool added in v0.11.0 

func NewReplicaMetadataSlicePool(opts pool.ObjectPoolOptions, capacity int) ReplicaMetadataSlicePool

NewReplicaMetadataSlicePool creates a new replicaMetadataSlicePool pool

type ReplicaSeriesBlocksMetadata 

type ReplicaSeriesBlocksMetadata struct {
	ID       ident.ID
	Metadata ReplicaBlocksMetadata
}

ReplicaSeriesBlocksMetadata represents series metadata and an associated ID.

type ReplicaSeriesMetadata 

type ReplicaSeriesMetadata interface {
	// NumSeries returns the total number of series.
	NumSeries() int64

	// NumBlocks returns the total number of blocks.
	NumBlocks() int64

	// Series returns the series metadata.
	Series() *Map

	// GetOrAdd returns the series metadata for an id, creating one if it doesn't exist.
	GetOrAdd(id ident.ID) ReplicaBlocksMetadata

	// Close performs cleanup.
	Close()
}

ReplicaSeriesMetadata captures the metadata for a list of series from hosts in a shard replica set.

func NewReplicaSeriesMetadata 

func NewReplicaSeriesMetadata() ReplicaSeriesMetadata

NewReplicaSeriesMetadata creates a new replica series metadata

type SetUnsafeOptions 

type SetUnsafeOptions struct {
	NoCopyKey     bool
	NoFinalizeKey bool
}

SetUnsafeOptions is a set of options to use when setting a value with the SetUnsafe method.

type Strategy added in v1.2.0 

type Strategy uint

Strategy defines the repair strategy. 

const (
	// DefaultStrategy will compare iterating backwards then on repairing a
	// block needing repair it will restart from the latest block start and
	// work backwards again.
	// This strategy is best at keeping most recent data repaired as quickly
	// as possible but when turning on repair for the first time in a cluster
	// you may want to do run repairs in full sweep for a while first.
	DefaultStrategy Strategy = iota
	// FullSweepStrategy will compare iterating backwards and repairing
	// blocks needing repair until reaching the end of retention and then only
	// once reaching the end of retention to repair does the repair restart
	// evaluating blocks from the most recent block starts again.
	// This mode may be more ideal in clusters that have never had repair
	// enabled to ensure that historical data gets repaired at least once on
	// a full sweep before switching back to the default strategy.
	FullSweepStrategy
)

func (Strategy) MarshalYAML added in v1.4.0 

func (t Strategy) MarshalYAML() (interface{}, error)

MarshalYAML returns the YAML representation of the repair strategy.

func (Strategy) String added in v1.2.0 

func (t Strategy) String() string

String returns the bootstrap mode as a string

func (*Strategy) UnmarshalYAML added in v1.2.0 

func (t *Strategy) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML unmarshals an Type into a valid type from string.

type Type added in v1.2.0 

type Type uint

Type defines the type of repair to run. 

const (
	// DefaultRepair will compare node's integrity to other replicas and then repair blocks as required.
	DefaultRepair Type = iota
	// OnlyCompareRepair will compare node's integrity to other replicas without repairing blocks,
	// this is useful for looking at the metrics emitted by the comparison.
	OnlyCompareRepair
)

func (Type) MarshalYAML added in v1.4.0 

func (t Type) MarshalYAML() (interface{}, error)

MarshalYAML returns the YAML representation of the repair type.

func (Type) String added in v1.2.0 

func (t Type) String() string

String returns the bootstrap mode as a string

func (*Type) UnmarshalYAML added in v1.2.0 

func (t *Type) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML unmarshals an Type into a valid type from string.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.