Documentation
¶
Index ¶
- Variables
- func CalcPieceSize(dataSize int64, scheme ErasureScheme) int64
- func Decode(log *zap.Logger, rrs map[int]ranger.Ranger, es ErasureScheme, mbm int, ...) (ranger.Ranger, error)
- func DecodeReaders(ctx context.Context, cancel func(), log *zap.Logger, rs map[int]io.ReadCloser, ...) io.ReadCloser
- func EncodeReader(ctx context.Context, log *zap.Logger, r io.Reader, rs RedundancyStrategy) (_ []io.ReadCloser, err error)
- func Pad(data ranger.Ranger, blockSize int) (rr ranger.Ranger, padding int)
- func PadReader(data io.ReadCloser, blockSize int) io.ReadCloser
- func Unpad(data ranger.Ranger, padding int) (ranger.Ranger, error)
- func UnpadSlow(ctx context.Context, data ranger.Ranger) (_ ranger.Ranger, err error)
- type EncodedRanger
- type ErasureScheme
- type PieceBuffer
- func (b *PieceBuffer) Close() error
- func (b *PieceBuffer) HasShare(num int64) bool
- func (b *PieceBuffer) Read(p []byte) (n int, err error)
- func (b *PieceBuffer) ReadShare(num int64, p []byte) error
- func (b *PieceBuffer) SetError(err error)
- func (b *PieceBuffer) Skip(n int) error
- func (b *PieceBuffer) Write(p []byte) (n int, err error)
- type RedundancyStrategy
- type StripeReader
Constants ¶
This section is empty.
Variables ¶
var Error = errs.Class("eestream error")
Error is the default eestream errs class
Functions ¶
func CalcPieceSize ¶
func CalcPieceSize(dataSize int64, scheme ErasureScheme) int64
CalcPieceSize calculates what would be the piece size of the encoded data after erasure coding data with dataSize using the given ErasureScheme.
func Decode ¶
func Decode(log *zap.Logger, rrs map[int]ranger.Ranger, es ErasureScheme, mbm int, forceErrorDetection bool) (ranger.Ranger, error)
Decode takes a map of Rangers and an ErasureScheme and returns a combined Ranger.
rrs is a map of erasure piece numbers to erasure piece rangers. mbm is the maximum memory (in bytes) to be allocated for read buffers. If set to 0, the minimum possible memory will be used. if forceErrorDetection is set to true then k+1 pieces will be always required for decoding, so corrupted pieces can be detected.
func DecodeReaders ¶
func DecodeReaders(ctx context.Context, cancel func(), log *zap.Logger, rs map[int]io.ReadCloser, es ErasureScheme, expectedSize int64, mbm int, forceErrorDetection bool) io.ReadCloser
DecodeReaders takes a map of readers and an ErasureScheme returning a combined Reader.
rs is a map of erasure piece numbers to erasure piece streams. expectedSize is the number of bytes expected to be returned by the Reader. mbm is the maximum memory (in bytes) to be allocated for read buffers. If set to 0, the minimum possible memory will be used. if forceErrorDetection is set to true then k+1 pieces will be always required for decoding, so corrupted pieces can be detected.
func EncodeReader ¶
func EncodeReader(ctx context.Context, log *zap.Logger, r io.Reader, rs RedundancyStrategy) (_ []io.ReadCloser, err error)
EncodeReader takes a Reader and a RedundancyStrategy and returns a slice of io.ReadClosers.
func Pad ¶
Pad takes a Ranger and returns another Ranger that is a multiple of blockSize in length. The return value padding is a convenience to report how much padding was added.
func PadReader ¶
func PadReader(data io.ReadCloser, blockSize int) io.ReadCloser
PadReader is like Pad but works on a basic Reader instead of a Ranger.
Types ¶
type EncodedRanger ¶
type EncodedRanger struct {
// contains filtered or unexported fields
}
EncodedRanger will take an existing Ranger and provide a means to get multiple Ranged sub-Readers. EncodedRanger does not match the normal Ranger interface.
func NewEncodedRanger ¶
func NewEncodedRanger(log *zap.Logger, rr ranger.Ranger, rs RedundancyStrategy) (*EncodedRanger, error)
NewEncodedRanger from the given Ranger and RedundancyStrategy. See the comments for EncodeReader about the repair and success thresholds.
func (*EncodedRanger) OutputSize ¶
func (er *EncodedRanger) OutputSize() int64
OutputSize is like Ranger.Size but returns the Size of the erasure encoded pieces that come out.
func (*EncodedRanger) Range ¶
func (er *EncodedRanger) Range(ctx context.Context, offset, length int64) (_ []io.ReadCloser, err error)
Range is like Ranger.Range, but returns a slice of Readers
type ErasureScheme ¶
type ErasureScheme interface {
// Encode will take 'in' and call 'out' with erasure coded pieces.
Encode(in []byte, out func(num int, data []byte)) error
// EncodeSingle will take 'in' with the stripe and fill 'out' with the erasure share for piece 'num'.
EncodeSingle(in, out []byte, num int) error
// Decode will take a mapping of available erasure coded piece num -> data,
// 'in', and append the combined data to 'out', returning it.
Decode(out []byte, in map[int][]byte) ([]byte, error)
// and are passed to Decode.
ErasureShareSize() int
// StripeSize is the size the stripes that are passed to Encode and come
// from Decode.
StripeSize() int
// Encode will generate this many erasure shares and therefore this many pieces
TotalCount() int
// Decode requires at least this many pieces
RequiredCount() int
}
ErasureScheme represents the general format of any erasure scheme algorithm. If this interface can be implemented, the rest of this library will work with it.
func NewRSScheme ¶
func NewRSScheme(fc *infectious.FEC, erasureShareSize int) ErasureScheme
NewRSScheme returns a Reed-Solomon-based ErasureScheme.
type PieceBuffer ¶
type PieceBuffer struct {
// contains filtered or unexported fields
}
PieceBuffer is a synchronized buffer for storing erasure shares for a piece.
func NewPieceBuffer ¶
func NewPieceBuffer(log *zap.Logger, buf []byte, shareSize int, newDataCond *sync.Cond) *PieceBuffer
NewPieceBuffer creates and initializes a new PieceBuffer using buf as its internal content. If new data is written to the buffer, newDataCond will be notified.
func (*PieceBuffer) Close ¶
func (b *PieceBuffer) Close() error
Close sets io.ErrClosedPipe to the buffer to prevent further writes and blocking on read.
func (*PieceBuffer) HasShare ¶
func (b *PieceBuffer) HasShare(num int64) bool
HasShare checks if the num-th share can be read from the buffer without blocking. If there are older erasure shares in the buffer, they will be discarded to leave room for the newer erasure shares to be written.
func (*PieceBuffer) Read ¶
func (b *PieceBuffer) Read(p []byte) (n int, err error)
Read reads the next len(p) bytes from the buffer or until the buffer is drained. The return value n is the number of bytes read. If the buffer has no data to return and no error is set, the call will block until new data is written to the buffer. Otherwise the error will be returned.
func (*PieceBuffer) ReadShare ¶
func (b *PieceBuffer) ReadShare(num int64, p []byte) error
ReadShare reads the num-th erasure share from the buffer into p. Any shares before num will be discarded from the buffer.
func (*PieceBuffer) SetError ¶
func (b *PieceBuffer) SetError(err error)
SetError sets an error to be returned by Read and Write. Read will return the error after all data is read from the buffer.
func (*PieceBuffer) Skip ¶
func (b *PieceBuffer) Skip(n int) error
Skip advances the read pointer with n bytes. It the buffered number of bytes are less than n, the method will block until enough data is written to the buffer.
func (*PieceBuffer) Write ¶
func (b *PieceBuffer) Write(p []byte) (n int, err error)
Write writes the contents of p into the buffer. If the buffer is full it will block until some data is read from it, or an error is set. The return value n is the number of bytes written. If an error was set, it be returned.
type RedundancyStrategy ¶
type RedundancyStrategy struct {
ErasureScheme
// contains filtered or unexported fields
}
RedundancyStrategy is an ErasureScheme with a repair and optimal thresholds
func NewRedundancyStrategy ¶
func NewRedundancyStrategy(es ErasureScheme, repairThreshold, optimalThreshold int) (RedundancyStrategy, error)
NewRedundancyStrategy from the given ErasureScheme, repair and optimal thresholds.
repairThreshold is the minimum repair threshold. If set to 0, it will be reset to the TotalCount of the ErasureScheme. optimalThreshold is the optimal threshold. If set to 0, it will be reset to the TotalCount of the ErasureScheme.
func NewRedundancyStrategyFromProto ¶
func NewRedundancyStrategyFromProto(scheme *pb.RedundancyScheme) (RedundancyStrategy, error)
NewRedundancyStrategyFromProto creates new RedundancyStrategy from the given RedundancyScheme protobuf.
func NewRedundancyStrategyFromStorj ¶ added in v0.17.0
func NewRedundancyStrategyFromStorj(scheme storj.RedundancyScheme) (RedundancyStrategy, error)
NewRedundancyStrategyFromStorj creates new RedundancyStrategy from the given storj.RedundancyScheme.
func (*RedundancyStrategy) OptimalThreshold ¶
func (rs *RedundancyStrategy) OptimalThreshold() int
OptimalThreshold is the number of available erasure pieces above which there is no need for the data to be repaired
func (*RedundancyStrategy) RepairThreshold ¶
func (rs *RedundancyStrategy) RepairThreshold() int
RepairThreshold is the number of available erasure pieces below which the data must be repaired to avoid loss
type StripeReader ¶
type StripeReader struct {
// contains filtered or unexported fields
}
StripeReader can read and decodes stripes from a set of readers
func NewStripeReader ¶
func NewStripeReader(log *zap.Logger, rs map[int]io.ReadCloser, es ErasureScheme, mbm int, forceErrorDetection bool) *StripeReader
NewStripeReader creates a new StripeReader from the given readers, erasure scheme and max buffer memory.
func (*StripeReader) Close ¶
func (r *StripeReader) Close() error
Close closes the StripeReader and all PieceBuffers.
func (*StripeReader) ReadStripe ¶
ReadStripe reads and decodes the num-th stripe and concatenates it to p. The return value is the updated byte slice.
Source Files