Documentation
¶
Overview ¶
Package piecedeletion implements service for deleting pieces that combines concurrent requests.
Index ¶
- Variables
- func FailPending(jobs Queue)
- type Combiner
- type Config
- type Dialer
- type Handler
- type Job
- type LimitedHandler
- type LimitedJobs
- type NewQueue
- type Nodes
- type Promise
- type Queue
- type Request
- type Service
- func (service *Service) Close() error
- func (service *Service) Delete(ctx context.Context, requests []Request) (err error)
- func (service *Service) DeleteWithCustomThreshold(ctx context.Context, requests []Request, successThreshold float64) (err error)
- func (service *Service) Run(ctx context.Context) error
Constants ¶
This section is empty.
Variables ¶
var Error = errs.Class("piece deletion")
Error is the default error class for piece deletion.
Functions ¶
Types ¶
type Combiner ¶
type Combiner struct {
// contains filtered or unexported fields
}
Combiner combines multiple concurrent deletion requests into batches.
func NewCombiner ¶
NewCombiner creates a new combiner.
type Config ¶ added in v1.0.1
type Config struct {
MaxConcurrency int `help:"maximum number of concurrent requests to storage nodes" default:"100"`
MaxConcurrentPieces int `help:"maximum number of concurrent pieces can be processed" default:"1000000" testDefault:"1000"`
MaxPiecesPerBatch int `help:"maximum number of pieces per batch" default:"5000" testDefault:"4000"`
MaxPiecesPerRequest int `help:"maximum number pieces per single request" default:"1000" testDefault:"2000"`
DialTimeout time.Duration `help:"timeout for dialing nodes (0 means satellite default)" default:"3s" testDefault:"2s"`
FailThreshold time.Duration `help:"threshold for retrying a failed node" releaseDefault:"10m" devDefault:"2s"`
RequestTimeout time.Duration `help:"timeout for a single delete request" releaseDefault:"15s" devDefault:"2s"`
DeleteSuccessThreshold float64 `` /* 139-byte string literal not displayed */
}
Config defines configuration options for Service.
type Dialer ¶ added in v1.0.1
type Dialer struct {
// contains filtered or unexported fields
}
Dialer implements dialing piecestores and sending delete requests with batching and redial threshold.
type Handler ¶
type Handler interface {
// Handle should call queue.PopAll until finished.
Handle(ctx context.Context, node storj.NodeURL, queue Queue)
}
Handler handles piece deletion requests from a queue.
type Job ¶
type Job struct {
// Pieces are the pieces id-s that need to be deleted.
Pieces []storj.PieceID
// Resolve is for notifying the job issuer about the outcome.
Resolve Promise
}
Job is a single of deletion.
type LimitedHandler ¶
type LimitedHandler struct {
Handler
// contains filtered or unexported fields
}
LimitedHandler wraps handler with a concurrency limit.
func NewLimitedHandler ¶
func NewLimitedHandler(handler Handler, limit int) *LimitedHandler
NewLimitedHandler wraps handler with a concurrency limit.
type LimitedJobs ¶
type LimitedJobs struct {
// contains filtered or unexported fields
}
LimitedJobs is a finalizable list of deletion jobs with a limit to how many jobs it can handle.
func NewLimitedJobs ¶
func NewLimitedJobs(maxPiecesPerBatch int) *LimitedJobs
NewLimitedJobs returns a new limited job queue.
func (*LimitedJobs) PopAll ¶
func (jobs *LimitedJobs) PopAll() (_ []Job, ok bool)
PopAll returns all the jobs in this list.
func (*LimitedJobs) PopAllWithoutClose ¶ added in v1.4.1
func (jobs *LimitedJobs) PopAllWithoutClose() []Job
PopAllWithoutClose returns all the jobs in this list without closing the queue.
func (*LimitedJobs) TryPush ¶
func (jobs *LimitedJobs) TryPush(job Job) bool
TryPush tries to add a job to the queue.
maxPiecesPerBatch < 0, means no limit.
type Nodes ¶ added in v1.11.1
type Nodes interface {
GetNodes(ctx context.Context, nodes []storj.NodeID) (_ map[storj.NodeID]*overlay.SelectedNode, err error)
}
Nodes stores reliable nodes information.
type Promise ¶
type Promise interface {
// Success is called when the job has been successfully handled.
Success()
// Failure is called when the job didn't complete successfully.
Failure()
}
Promise is for signaling to the deletion requests about the result.
type Queue ¶
type Queue interface {
// TryPush tries to push a new job to the queue.
TryPush(job Job) bool
// PopAll fetches all jobs in the queue.
//
// When there are no more jobs, the queue must stop accepting new jobs.
PopAll() ([]Job, bool)
// PopAllWithoutClose fetches all jobs in the queue,
// but without closing the queue for new requests.
PopAllWithoutClose() []Job
}
Queue is a queue for jobs.
type Service ¶ added in v1.0.1
type Service struct {
// contains filtered or unexported fields
}
Service handles combining piece deletion requests.
architecture: Service
func NewService ¶ added in v1.0.1
NewService creates a new service.
func (*Service) Delete ¶ added in v1.0.1
Delete deletes the pieces specified in the requests, returning when they have been deleted from the default fraction of storage nodes.
func (*Service) DeleteWithCustomThreshold ¶ added in v1.66.1
func (service *Service) DeleteWithCustomThreshold(ctx context.Context, requests []Request, successThreshold float64) (err error)
DeleteWithCustomThreshold deletes the pieces specified in the requests, returning when they have been deleted from the specified fraction of storage nodes.
Source Files