piecedeletion

package
v1.31.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 31, 2021 License: AGPL-3.0 Imports: 13 Imported by: 0

Documentation

Overview

Package piecedeletion implements service for deleting pieces that combines concurrent requests.

Index

Constants

This section is empty.

Variables

View Source
var Error = errs.Class("piece deletion")

Error is the default error class for piece deletion.

Functions

func FailPending

func FailPending(jobs Queue)

FailPending fails all the jobs in the queue.

Types

type Combiner

type Combiner struct {
	// contains filtered or unexported fields
}

Combiner combines multiple concurrent deletion requests into batches.

func NewCombiner

func NewCombiner(parent context.Context, handler Handler, newQueue NewQueue) *Combiner

NewCombiner creates a new combiner.

func (*Combiner) Close

func (combiner *Combiner) Close()

Close shuts down all workers.

func (*Combiner) Enqueue

func (combiner *Combiner) Enqueue(node storj.NodeURL, job Job)

Enqueue adds a deletion job to the queue.

type Config added in v1.0.1

type Config struct {
	MaxConcurrency      int `help:"maximum number of concurrent requests to storage nodes" default:"100"`
	MaxConcurrentPieces int `help:"maximum number of concurrent pieces can be processed" default:"1000000"`

	MaxPiecesPerBatch   int `help:"maximum number of pieces per batch" default:"5000"`
	MaxPiecesPerRequest int `help:"maximum number pieces per single request" default:"1000"`

	DialTimeout    time.Duration `help:"timeout for dialing nodes (0 means satellite default)" default:"0"`
	FailThreshold  time.Duration `help:"threshold for retrying a failed node" releaseDefault:"5m" devDefault:"2s"`
	RequestTimeout time.Duration `help:"timeout for a single delete request" releaseDefault:"1m" devDefault:"2s"`
}

Config defines configuration options for Service.

func (*Config) Verify added in v1.0.1

func (config *Config) Verify() errs.Group

Verify verifies configuration sanity.

type Dialer added in v1.0.1

type Dialer struct {
	// contains filtered or unexported fields
}

Dialer implements dialing piecestores and sending delete requests with batching and redial threshold.

func NewDialer added in v1.0.1

func NewDialer(log *zap.Logger, dialer rpc.Dialer, requestTimeout, failThreshold time.Duration, piecesPerRequest int) *Dialer

NewDialer returns a new Dialer.

func (*Dialer) Handle added in v1.0.1

func (dialer *Dialer) Handle(ctx context.Context, node storj.NodeURL, queue Queue)

Handle tries to send the deletion requests to the specified node.

type Handler

type Handler interface {
	// Handle should call queue.PopAll until finished.
	Handle(ctx context.Context, node storj.NodeURL, queue Queue)
}

Handler handles piece deletion requests from a queue.

type Job

type Job struct {
	// Pieces are the pieces id-s that need to be deleted.
	Pieces []storj.PieceID
	// Resolve is for notifying the job issuer about the outcome.
	Resolve Promise
}

Job is a single of deletion.

type LimitedHandler

type LimitedHandler struct {
	Handler
	// contains filtered or unexported fields
}

LimitedHandler wraps handler with a concurrency limit.

func NewLimitedHandler

func NewLimitedHandler(handler Handler, limit int) *LimitedHandler

NewLimitedHandler wraps handler with a concurrency limit.

func (*LimitedHandler) Handle

func (handler *LimitedHandler) Handle(ctx context.Context, node storj.NodeURL, queue Queue)

Handle handles the job queue.

type LimitedJobs

type LimitedJobs struct {
	// contains filtered or unexported fields
}

LimitedJobs is a finalizable list of deletion jobs with a limit to how many jobs it can handle.

func NewLimitedJobs

func NewLimitedJobs(maxPiecesPerBatch int) *LimitedJobs

NewLimitedJobs returns a new limited job queue.

func (*LimitedJobs) PopAll

func (jobs *LimitedJobs) PopAll() (_ []Job, ok bool)

PopAll returns all the jobs in this list.

func (*LimitedJobs) PopAllWithoutClose added in v1.4.1

func (jobs *LimitedJobs) PopAllWithoutClose() []Job

PopAllWithoutClose returns all the jobs in this list without closing the queue.

func (*LimitedJobs) TryPush

func (jobs *LimitedJobs) TryPush(job Job) bool

TryPush tries to add a job to the queue.

maxPiecesPerBatch < 0, means no limit.

type NewQueue

type NewQueue func() Queue

NewQueue is a constructor func for queues.

type Nodes added in v1.11.1

type Nodes interface {
	KnownReliable(ctx context.Context, nodeIDs storj.NodeIDList) ([]*pb.Node, error)
}

Nodes stores reliable nodes information.

type Promise

type Promise interface {
	// Success is called when the job has been successfully handled.
	Success()
	// Failure is called when the job didn't complete successfully.
	Failure()
}

Promise is for signaling to the deletion requests about the result.

type Queue

type Queue interface {
	// TryPush tries to push a new job to the queue.
	TryPush(job Job) bool

	// PopAll fetches all jobs in the queue.
	//
	// When there are no more jobs, the queue must stop accepting new jobs.
	PopAll() ([]Job, bool)

	// PopAllWithoutClose fetches all jobs in the queue,
	// but without closing the queue for new requests.
	PopAllWithoutClose() []Job
}

Queue is a queue for jobs.

type Request added in v1.0.1

type Request struct {
	Node   storj.NodeURL
	Pieces []storj.PieceID
}

Request defines a deletion requests for a node.

func (*Request) IsValid added in v1.0.1

func (req *Request) IsValid() bool

IsValid returns whether the request is valid.

type Service added in v1.0.1

type Service struct {
	// contains filtered or unexported fields
}

Service handles combining piece deletion requests.

architecture: Service

func NewService added in v1.0.1

func NewService(log *zap.Logger, dialer rpc.Dialer, nodesDB Nodes, config Config) (*Service, error)

NewService creates a new service.

func (*Service) Close added in v1.0.1

func (service *Service) Close() error

Close shuts down the service.

func (*Service) Delete added in v1.0.1

func (service *Service) Delete(ctx context.Context, requests []Request, successThreshold float64) (err error)

Delete deletes the pieces specified in the requests waiting until success threshold is reached.

func (*Service) Run added in v1.0.1

func (service *Service) Run(ctx context.Context) error

Run initializes the service.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL