durability

Documentation

Overview

Package durability helps us to keep segments in healthy state with reporting risks.

Index

• type Bucket
  • func (b *Bucket) Increment(segmentExemplar uuid.UUID, segmentExemplarPosition metabase.SegmentPosition, ...)
  • func (b *Bucket) Reset()
• type ClassGroupCounter
• type ClassGroupCounters
  • func (c *ClassGroupCounters) Increment(class ClassID)
• type ClassID
• type Config
  • func (c Config) CreateNodeClassifiers() (map[string]NodeClassifier, error)
• type Histogram
  • func (h *Histogram) AddPieceCount(pieceCount int, segmentExemplar uuid.UUID, ...)
  • func (h *Histogram) Merge(o Histogram)
  • func (h *Histogram) Reset()
• type HistogramByPlacement
  • func (p *HistogramByPlacement) Extend(size int)
  • func (p *HistogramByPlacement) Reset()
• type NodeClassifier
• type NodeGetter
• type ObserverFork
  • func (c *ObserverFork) Process(ctx context.Context, segments []rangedloop.Segment) (err error)
• type Report
  • func NewDurability(db overlay.DB, metabaseDB *metabase.DB, nodeGetter NodeGetter, class string, ...) *Report
  • func (c *Report) Finish(ctx context.Context) error
  • func (c *Report) Fork(ctx context.Context) (rangedloop.Partial, error)
  • func (c *Report) GetClass() string
  • func (c *Report) Join(ctx context.Context, partial rangedloop.Partial) (err error)
  • func (c *Report) Start(ctx context.Context, startTime time.Time) (err error)
  • func (c *Report) TestChangeReporter(r func(n time.Time, class string, missingProvider int, ix int, ...))
• type ReportConfig

Types

type Bucket

type Bucket struct {
	SegmentCount     int
	SegmentExemplars []string
	ClassExemplars   []ClassID
}

Bucket stores the number of segments (and some exemplars) for each piece count.

func (*Bucket) Increment

func (b *Bucket) Increment(segmentExemplar uuid.UUID, segmentExemplarPosition metabase.SegmentPosition, classExemplar ClassID)

Increment increments the bucket counters.

func (*Bucket) Reset

func (b *Bucket) Reset()

Reset resets the bucket counters.

type ClassGroupCounter

type ClassGroupCounter struct {
	Class   ClassID
	Counter int
}

ClassGroupCounter is a helper struct to count the number of pieces in a class.

type ClassGroupCounters

type ClassGroupCounters struct {
	Counters []ClassGroupCounter
}

ClassGroupCounters is a helper struct to count the number of pieces in each class.

func (*ClassGroupCounters) Increment

func (c *ClassGroupCounters) Increment(class ClassID)

Increment increments the counter for the given class.

type ClassID

type ClassID int32

ClassID is a fork level short identifier for each class.

type Config

type Config struct {
	Classes []string `help:"Node attributes used by the durability segment loop to classify risks" default:"last_net,last_ip,wallet,email"`
}

Config contains configuration for the durability ranged loop observer.

func (Config) CreateNodeClassifiers

func (c Config) CreateNodeClassifiers() (map[string]NodeClassifier, error)

CreateNodeClassifiers creates a list of node classifiers based on the configuration.

type Histogram

type Histogram struct {
	// pieceCount -> {number of segments, exemplars}
	Buckets []*Bucket
	// pieceCount * -1 -> {number of segments, exemplars}
	NegativeBuckets []*Bucket
}

Histogram stores the number of segments (and some exemplars) for each piece count.

func (*Histogram) AddPieceCount

func (h *Histogram) AddPieceCount(pieceCount int, segmentExemplar uuid.UUID, segmentExemplarPosition metabase.SegmentPosition, classExemplar ClassID)

AddPieceCount adds a piece count to the histogram.

func (*Histogram) Merge

func (h *Histogram) Merge(o Histogram)

Merge merges the histogram with another histogram.

func (*Histogram) Reset

func (h *Histogram) Reset()

Reset resets the histogram counters.

type HistogramByPlacement

type HistogramByPlacement []Histogram // placement -> Histogram

HistogramByPlacement contains multiple Histogram for each Placement.

func (*HistogramByPlacement) Extend

func (p *HistogramByPlacement) Extend(size int)

Extend will make sure the placement has an initialized Histogram.

func (*HistogramByPlacement) Reset

func (p *HistogramByPlacement) Reset()

Reset will reset all the included Histograms.

type NodeClassifier

type NodeClassifier func(node *nodeselection.SelectedNode) string

NodeClassifier identifies a risk class (for example an owner, or country) of the SelectedNode.

type NodeGetter

type NodeGetter interface {
	GetNodes(ctx context.Context, validUpTo time.Time, nodeIDs []storj.NodeID, selectedNodes []nodeselection.SelectedNode) ([]nodeselection.SelectedNode, error)
}

NodeGetter returns node information required by the durability loop.

type ObserverFork

type ObserverFork struct {
	// contains filtered or unexported fields
}

ObserverFork is the durability calculator for each segment range.

func (*ObserverFork) Process

func (c *ObserverFork) Process(ctx context.Context, segments []rangedloop.Segment) (err error)

Process implements rangedloop.Partial.

type Report

type Report struct {
	Class string
	// contains filtered or unexported fields
}

Report is a calculator (rangloop.Observer) which checks the availability of pieces without certain nodes. It can answer the following question:

1. loosing a given group of nodes (all nodes of one country or all nodes of one owner)...
2. what will be the lowest humber of healhty pieces, checking all the segments.

Example: we have one segment where 80 pieces are stored, but 42 of them are in Germany.

in this case this reporter will return 38 for the class "country:DE" (assuming all the other segments are more lucky).

func NewDurability

func NewDurability(db overlay.DB, metabaseDB *metabase.DB, nodeGetter NodeGetter, class string, classifier NodeClassifier, asOfSystemInterval time.Duration) *Report

NewDurability creates the new instance.

func (*Report) Finish

func (c *Report) Finish(ctx context.Context) error

Finish implements rangedloop.Observer.

func (*Report) Fork

func (c *Report) Fork(ctx context.Context) (rangedloop.Partial, error)

Fork implements rangedloop.Observer.

func (*Report) GetClass

func (c *Report) GetClass() string

GetClass return with the class instance name (like last_net or country).

func (*Report) Join

func (c *Report) Join(ctx context.Context, partial rangedloop.Partial) (err error)

Join implements rangedloop.Observer.

func (*Report) Start

func (c *Report) Start(ctx context.Context, startTime time.Time) (err error)

Start implements rangedloop.Observer.

func (*Report) TestChangeReporter

func (c *Report) TestChangeReporter(r func(n time.Time, class string, missingProvider int, ix int, placement storj.PlacementConstraint, stat Bucket, resolver func(id ClassID) string))

TestChangeReporter modifies the reporter for unit tests.

type ReportConfig

type ReportConfig struct {
	Enabled bool `help:"whether to enable durability report (rangedloop observer)" default:"true"`
}

ReportConfig configures durability report.