tally

package
v1.91.0-alpha Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 25, 2023 License: AGPL-3.0 Imports: 10 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	Error = errs.Class("tally")
)

Error is a standard error class for this package.

Functions

This section is empty.

Types

type BucketTallyCollector added in v1.35.1

type BucketTallyCollector struct {
	Now    time.Time
	Log    *zap.Logger
	Bucket map[metabase.BucketLocation]*accounting.BucketTally
	// contains filtered or unexported fields
}

BucketTallyCollector collects and adds up tallies for buckets.

func NewBucketTallyCollector added in v1.35.1

func NewBucketTallyCollector(log *zap.Logger, now time.Time, db *metabase.DB, bucketsDB buckets.DB, projectAccountingDB accounting.ProjectAccounting, config Config) *BucketTallyCollector

NewBucketTallyCollector returns a collector that adds up totals for buckets. The now argument controls when the collector considers objects to be expired.

func (*BucketTallyCollector) Run added in v1.35.1

func (observer *BucketTallyCollector) Run(ctx context.Context) (err error)

Run runs collecting bucket tallies.

type Config

type Config struct {
	Interval             time.Duration `help:"how frequently the tally service should run" releaseDefault:"1h" devDefault:"30s" testDefault:"$TESTINTERVAL"`
	SaveRollupBatchSize  int           `help:"how large of batches SaveRollup should process at a time" default:"1000"`
	ReadRollupBatchSize  int           `help:"how large of batches GetBandwidthSince should process at a time" default:"10000"`
	UseRangedLoop        bool          `help:"whether to enable node tally with ranged loop" default:"true"`
	SaveTalliesBatchSize int           `help:"how large should be insert into tallies" default:"10000"`

	ListLimit          int           `help:"how many buckets to query in a batch" default:"2500"`
	AsOfSystemInterval time.Duration `help:"as of system interval" releaseDefault:"-5m" devDefault:"-1us" testDefault:"-1us"`
}

Config contains configurable values for the tally service.

type Service

type Service struct {
	Loop *sync2.Cycle
	// contains filtered or unexported fields
}

Service is the tally service for data stored on each storage node.

architecture: Chore

func New

func New(log *zap.Logger, sdb accounting.StoragenodeAccounting, pdb accounting.ProjectAccounting, liveAccounting accounting.Cache, metabase *metabase.DB, bucketsDB buckets.DB, config Config) *Service

New creates a new tally Service.

func (*Service) Close added in v0.21.0

func (service *Service) Close() error

Close stops the service and releases any resources.

func (*Service) Run

func (service *Service) Run(ctx context.Context) (err error)

Run the tally service loop.

func (*Service) SetNow added in v1.3.2

func (service *Service) SetNow(now func() time.Time)

SetNow allows tests to have the Service act as if the current time is whatever they want. This avoids races and sleeping, making tests more reliable and efficient.

func (*Service) Tally

func (service *Service) Tally(ctx context.Context) (err error)

Tally calculates data-at-rest usage once.

How live accounting is calculated:

At the beginning of the tally iteration, we get a map containing the current project totals from the cache- initialLiveTotals (our current estimation of the project totals). At the end of the tally iteration, we have the totals from what we saw during the metainfo loop.

However, data which was uploaded during the loop may or may not have been seen in the metainfo loop. For this reason, we also read the live accounting totals again at the end of the tally iteration- latestLiveTotals.

The difference between latest and initial indicates how much data was uploaded during the metainfo loop and is assigned to delta. However, again, we aren't certain how much of the delta is accounted for in the metainfo totals. For the reason we make an assumption that 50% of the data is accounted for. So to calculate the new live accounting totals, we sum the metainfo totals and 50% of the deltas.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL