Documentation ¶
Index ¶
- Variables
- type AdminFetchInfo
- type Config
- type ECRepairer
- func (ec *ECRepairer) Get(ctx context.Context, limits []*pb.AddressedOrderLimit, ...) (_ io.ReadCloser, _ FetchResultReport, err error)
- func (ec *ECRepairer) Repair(ctx context.Context, limits []*pb.AddressedOrderLimit, ...) (successfulNodes []*pb.Node, successfulHashes []*pb.PieceHash, err error)
- func (ec *ECRepairer) TestingSetMinFailures(minFailures int)
- type FetchResultReport
- type PieceFetchResult
- type SegmentRepairer
- func (repairer *SegmentRepairer) AdminFetchPieces(ctx context.Context, seg *metabase.Segment, saveDir string) (pieceInfos []AdminFetchInfo, err error)
- func (repairer *SegmentRepairer) Repair(ctx context.Context, queueSegment *queue.InjuredSegment) (shouldDelete bool, err error)
- func (repairer *SegmentRepairer) SetNow(nowFn func() time.Time)
- type Service
Constants ¶
This section is empty.
Variables ¶
var ( // ErrPieceHashVerifyFailed is the errs class when a piece hash downloaded from storagenode fails to match the original hash. ErrPieceHashVerifyFailed = errs.Class("piece hashes don't match") // ErrDialFailed is the errs class when a failure happens during Dial. ErrDialFailed = errs.Class("dial failure") )
var (
Error = errs.Class("repairer")
)
Error is a standard error class for this package.
Functions ¶
This section is empty.
Types ¶
type AdminFetchInfo ¶ added in v1.49.3
type AdminFetchInfo struct { Reader io.ReadCloser Hash *pb.PieceHash GetLimit *pb.AddressedOrderLimit OriginalLimit *pb.OrderLimit FetchError error }
AdminFetchInfo groups together all the information about a piece that should be retrievable from storage nodes.
type Config ¶
type Config struct { MaxRepair int `help:"maximum segments that can be repaired concurrently" releaseDefault:"5" devDefault:"1" testDefault:"10"` Interval time.Duration `` /* 130-byte string literal not displayed */ Timeout time.Duration `help:"time limit for uploading repaired pieces to new storage nodes" default:"5m0s" testDefault:"1m"` DownloadTimeout time.Duration `help:"time limit for downloading pieces from a node for repair" default:"5m0s" testDefault:"1m"` TotalTimeout time.Duration `help:"time limit for an entire repair job, from queue pop to upload completion" default:"45m" testDefault:"10m"` MaxBufferMem memory.Size `help:"maximum buffer memory (in bytes) to be allocated for read buffers" default:"4.0 MiB"` MaxExcessRateOptimalThreshold float64 `` /* 135-byte string literal not displayed */ InMemoryRepair bool `help:"whether to download pieces for repair in memory (true) or download to disk (false)" default:"false"` ReputationUpdateEnabled bool `help:"whether the audit score of nodes should be updated as a part of repair" default:"false"` UseRangedLoop bool `help:"whether to use ranged loop instead of segment loop" default:"false"` }
Config contains configurable values for repairer.
type ECRepairer ¶ added in v0.21.0
type ECRepairer struct {
// contains filtered or unexported fields
}
ECRepairer allows the repairer to download, verify, and upload pieces from storagenodes.
func NewECRepairer ¶ added in v0.21.0
func NewECRepairer(log *zap.Logger, dialer rpc.Dialer, satelliteSignee signing.Signee, downloadTimeout time.Duration, inmemory bool) *ECRepairer
NewECRepairer creates a new repairer for interfacing with storagenodes.
func (*ECRepairer) Get ¶ added in v0.21.0
func (ec *ECRepairer) Get(ctx context.Context, limits []*pb.AddressedOrderLimit, cachedNodesInfo map[storj.NodeID]overlay.NodeReputation, privateKey storj.PiecePrivateKey, es eestream.ErasureScheme, dataSize int64) (_ io.ReadCloser, _ FetchResultReport, err error)
Get downloads pieces from storagenodes using the provided order limits, and decodes those pieces into a segment. It attempts to download from the minimum required number based on the redundancy scheme. It will further wait for additional error/failure results up to minFailures, for testing purposes. Under normal conditions, minFailures will be 0.
After downloading a piece, the ECRepairer will verify the hash and original order limit for that piece. If verification fails, another piece will be downloaded until we reach the minimum required or run out of order limits. If piece hash verification fails, it will return all failed node IDs.
func (*ECRepairer) Repair ¶ added in v0.21.0
func (ec *ECRepairer) Repair(ctx context.Context, limits []*pb.AddressedOrderLimit, privateKey storj.PiecePrivateKey, rs eestream.RedundancyStrategy, data io.Reader, timeout time.Duration, successfulNeeded int) (successfulNodes []*pb.Node, successfulHashes []*pb.PieceHash, err error)
Repair takes a provided segment, encodes it with the provided redundancy strategy, and uploads the pieces in need of repair to new nodes provided by order limits.
func (*ECRepairer) TestingSetMinFailures ¶ added in v1.77.2
func (ec *ECRepairer) TestingSetMinFailures(minFailures int)
TestingSetMinFailures sets the minFailures attribute, which tells the Repair machinery that we _expect_ there to be failures and that we should wait for them if necessary. This is only used in tests.
type FetchResultReport ¶ added in v1.65.1
type FetchResultReport struct { Successful []PieceFetchResult Failed []PieceFetchResult Offline []PieceFetchResult Contained []PieceFetchResult Unknown []PieceFetchResult }
FetchResultReport contains a categorization of a set of pieces based on the results of GET operations.
type PieceFetchResult ¶ added in v1.65.1
PieceFetchResult combines a piece pointer with the error we got when we tried to acquire that piece.
type SegmentRepairer ¶
type SegmentRepairer struct { OnTestingCheckSegmentAlteredHook func() OnTestingPiecesReportHook func(pieces FetchResultReport) // contains filtered or unexported fields }
SegmentRepairer for segments.
func NewSegmentRepairer ¶
func NewSegmentRepairer( log *zap.Logger, metabase *metabase.DB, orders *orders.Service, overlay *overlay.Service, reporter audit.Reporter, ecRepairer *ECRepairer, repairOverrides checker.RepairOverrides, config Config, ) *SegmentRepairer
NewSegmentRepairer creates a new instance of SegmentRepairer.
excessPercentageOptimalThreshold is the percentage to apply over the optimal threshould to determine the maximum limit of nodes to upload repaired pieces, when negative, 0 is applied.
func (*SegmentRepairer) AdminFetchPieces ¶ added in v1.49.3
func (repairer *SegmentRepairer) AdminFetchPieces(ctx context.Context, seg *metabase.Segment, saveDir string) (pieceInfos []AdminFetchInfo, err error)
AdminFetchPieces retrieves raw pieces and the associated hashes and original order limits from the storage nodes on which they are stored, and returns them intact to the caller rather than decoding or decrypting or verifying anything. This is to be used for debugging purposes.
func (*SegmentRepairer) Repair ¶
func (repairer *SegmentRepairer) Repair(ctx context.Context, queueSegment *queue.InjuredSegment) (shouldDelete bool, err error)
Repair retrieves an at-risk segment and repairs and stores lost pieces on new nodes note that shouldDelete is used even in the case where err is not null note that it will update audit status as failed for nodes that failed piece hash verification during repair downloading.
func (*SegmentRepairer) SetNow ¶ added in v1.26.2
func (repairer *SegmentRepairer) SetNow(nowFn func() time.Time)
SetNow allows tests to have the server act as if the current time is whatever they want.
type Service ¶
type Service struct { JobLimiter *semaphore.Weighted Loop *sync2.Cycle // contains filtered or unexported fields }
Service contains the information needed to run the repair service.
architecture: Worker
func NewService ¶
func NewService(log *zap.Logger, queue queue.RepairQueue, config *Config, repairer *SegmentRepairer) *Service
NewService creates repairing service.
func (*Service) SetNow ¶ added in v1.26.2
SetNow allows tests to have the server act as if the current time is whatever they want.
func (*Service) TestingSetMinFailures ¶ added in v1.77.2
TestingSetMinFailures sets the minFailures attribute, which tells the Repair machinery that we _expect_ there to be failures and that we should wait for them if necessary. This is only used in tests.
func (*Service) WaitForPendingRepairs ¶ added in v0.34.1
func (service *Service) WaitForPendingRepairs()
WaitForPendingRepairs waits for all ongoing repairs to complete.
NB: this assumes that service.config.MaxRepair will never be changed once this Service instance is initialized. If that is not a valid assumption, we should keep a copy of its initial value to use here instead.