gc

Documentation

Overview

Package gc contains the functions needed to run garbage collection.

The gc.PieceTracker implements the segments loop Observer interface allowing us to subscribe to the loop to get information for every segment in the metabase db.

The gc.PieceTracker handling functions are used by the gc.Service to periodically account for all existing pieces on storage nodes and create "retain requests" which contain a bloom filter of all pieces that possibly exist on a storage node.

The gc.Service will send that request to the storagenode after a full segments loop iteration, and the storage node will use that request to delete the "garbage" pieces that are not in the bloom filter.

See storj/docs/design/garbage-collection.md for more info.

Index

• Variables
• type Config
• type PieceTracker
  • func NewPieceTracker(log *zap.Logger, config Config, pieceCounts map[storj.NodeID]int) *PieceTracker
  • func (pieceTracker *PieceTracker) InlineSegment(ctx context.Context, segment *segmentloop.Segment) (err error)
  • func (pieceTracker *PieceTracker) LoopStarted(ctx context.Context, info segmentloop.LoopInfo) (err error)
  • func (pieceTracker *PieceTracker) RemoteSegment(ctx context.Context, segment *segmentloop.Segment) error
• type RetainInfo
• type Service
  • func NewService(log *zap.Logger, config Config, dialer rpc.Dialer, overlay overlay.DB, ...) *Service
  • func (service *Service) Run(ctx context.Context) (err error)

Variables

var (
	// Error defines the gc service errors class.
	Error = errs.Class("gc")
)

Types

type Config

type Config struct {
	Interval time.Duration `` /* 147-byte string literal not displayed */
	Enabled  bool          `help:"set if garbage collection is enabled or not" releaseDefault:"true" devDefault:"true"`

	// value for InitialPieces currently based on average pieces per node
	InitialPieces     int           `` /* 139-byte string literal not displayed */
	FalsePositiveRate float64       `help:"the false positive rate used for creating a garbage collection bloom filter" releaseDefault:"0.1" devDefault:"0.1"`
	ConcurrentSends   int           `help:"the number of nodes to concurrently send garbage collection bloom filters to" releaseDefault:"1" devDefault:"1"`
	RetainSendTimeout time.Duration `help:"the amount of time to allow a node to handle a retain request" default:"1m"`
}

Config contains configurable values for garbage collection.

type PieceTracker

type PieceTracker struct {
	RetainInfos map[storj.NodeID]*RetainInfo
	// contains filtered or unexported fields
}

PieceTracker implements the metainfo loop observer interface for garbage collection.

architecture: Observer

func NewPieceTracker

func NewPieceTracker(log *zap.Logger, config Config, pieceCounts map[storj.NodeID]int) *PieceTracker

NewPieceTracker instantiates a new gc piece tracker to be subscribed to the metainfo loop.

func (*PieceTracker) InlineSegment

func (pieceTracker *PieceTracker) InlineSegment(ctx context.Context, segment *segmentloop.Segment) (err error)

InlineSegment returns nil because we're only doing gc for storage nodes for now.

func (*PieceTracker) LoopStarted

func (pieceTracker *PieceTracker) LoopStarted(ctx context.Context, info segmentloop.LoopInfo) (err error)

LoopStarted is called at each start of a loop.

func (*PieceTracker) RemoteSegment

func (pieceTracker *PieceTracker) RemoteSegment(ctx context.Context, segment *segmentloop.Segment) error

RemoteSegment takes a remote segment found in metabase and adds pieces to bloom filters.

type RetainInfo

type RetainInfo struct {
	Filter       *bloomfilter.Filter
	CreationDate time.Time
	Count        int
}

RetainInfo contains info needed for a storage node to retain important data and delete garbage data.

type Service

type Service struct {
	Loop *sync2.Cycle
	// contains filtered or unexported fields
}

Service implements the garbage collection service.

architecture: Chore

func NewService

func NewService(log *zap.Logger, config Config, dialer rpc.Dialer, overlay overlay.DB, loop *segmentloop.Service) *Service

NewService creates a new instance of the gc service.

func (*Service) Run

func (service *Service) Run(ctx context.Context) (err error)

Run starts the gc loop service.