eestream

package
v0.19.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 2, 2019 License: AGPL-3.0 Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var Error = errs.Class("eestream error")

Error is the default eestream errs class

Functions

func CalcPieceSize

func CalcPieceSize(dataSize int64, scheme ErasureScheme) int64

CalcPieceSize calculates what would be the piece size of the encoded data after erasure coding data with dataSize using the given ErasureScheme.

func Decode

func Decode(log *zap.Logger, rrs map[int]ranger.Ranger, es ErasureScheme, mbm int, forceErrorDetection bool) (ranger.Ranger, error)

Decode takes a map of Rangers and an ErasureScheme and returns a combined Ranger.

rrs is a map of erasure piece numbers to erasure piece rangers. mbm is the maximum memory (in bytes) to be allocated for read buffers. If set to 0, the minimum possible memory will be used. if forceErrorDetection is set to true then k+1 pieces will be always required for decoding, so corrupted pieces can be detected.

func DecodeReaders

func DecodeReaders(ctx context.Context, cancel func(), log *zap.Logger, rs map[int]io.ReadCloser, es ErasureScheme, expectedSize int64, mbm int, forceErrorDetection bool) io.ReadCloser

DecodeReaders takes a map of readers and an ErasureScheme returning a combined Reader.

rs is a map of erasure piece numbers to erasure piece streams. expectedSize is the number of bytes expected to be returned by the Reader. mbm is the maximum memory (in bytes) to be allocated for read buffers. If set to 0, the minimum possible memory will be used. if forceErrorDetection is set to true then k+1 pieces will be always required for decoding, so corrupted pieces can be detected.

func EncodeReader

func EncodeReader(ctx context.Context, log *zap.Logger, r io.Reader, rs RedundancyStrategy) (_ []io.ReadCloser, err error)

EncodeReader takes a Reader and a RedundancyStrategy and returns a slice of io.ReadClosers.

func Pad

func Pad(data ranger.Ranger, blockSize int) (
	rr ranger.Ranger, padding int)

Pad takes a Ranger and returns another Ranger that is a multiple of blockSize in length. The return value padding is a convenience to report how much padding was added.

func PadReader

func PadReader(data io.ReadCloser, blockSize int) io.ReadCloser

PadReader is like Pad but works on a basic Reader instead of a Ranger.

func Unpad

func Unpad(data ranger.Ranger, padding int) (ranger.Ranger, error)

Unpad takes a previously padded Ranger data source and returns an unpadded ranger, given the amount of padding. This is preferable to UnpadSlow if you can swing it.

func UnpadSlow

func UnpadSlow(ctx context.Context, data ranger.Ranger) (_ ranger.Ranger, err error)

UnpadSlow is like Unpad, but does not require the amount of padding. UnpadSlow will have to do extra work to make up for this missing information.

Types

type EncodedRanger

type EncodedRanger struct {
	// contains filtered or unexported fields
}

EncodedRanger will take an existing Ranger and provide a means to get multiple Ranged sub-Readers. EncodedRanger does not match the normal Ranger interface.

func NewEncodedRanger

func NewEncodedRanger(log *zap.Logger, rr ranger.Ranger, rs RedundancyStrategy) (*EncodedRanger, error)

NewEncodedRanger from the given Ranger and RedundancyStrategy. See the comments for EncodeReader about the repair and success thresholds.

func (*EncodedRanger) OutputSize

func (er *EncodedRanger) OutputSize() int64

OutputSize is like Ranger.Size but returns the Size of the erasure encoded pieces that come out.

func (*EncodedRanger) Range

func (er *EncodedRanger) Range(ctx context.Context, offset, length int64) (_ []io.ReadCloser, err error)

Range is like Ranger.Range, but returns a slice of Readers

type ErasureScheme

type ErasureScheme interface {
	// Encode will take 'in' and call 'out' with erasure coded pieces.
	Encode(in []byte, out func(num int, data []byte)) error

	// EncodeSingle will take 'in' with the stripe and fill 'out' with the erasure share for piece 'num'.
	EncodeSingle(in, out []byte, num int) error

	// Decode will take a mapping of available erasure coded piece num -> data,
	// 'in', and append the combined data to 'out', returning it.
	Decode(out []byte, in map[int][]byte) ([]byte, error)

	// ErasureShareSize is the size of the erasure shares that come from Encode
	// and are passed to Decode.
	ErasureShareSize() int

	// StripeSize is the size the stripes that are passed to Encode and come
	// from Decode.
	StripeSize() int

	// Encode will generate this many erasure shares and therefore this many pieces
	TotalCount() int

	// Decode requires at least this many pieces
	RequiredCount() int
}

ErasureScheme represents the general format of any erasure scheme algorithm. If this interface can be implemented, the rest of this library will work with it.

func NewRSScheme

func NewRSScheme(fc *infectious.FEC, erasureShareSize int) ErasureScheme

NewRSScheme returns a Reed-Solomon-based ErasureScheme.

type PieceBuffer

type PieceBuffer struct {
	// contains filtered or unexported fields
}

PieceBuffer is a synchronized buffer for storing erasure shares for a piece.

func NewPieceBuffer

func NewPieceBuffer(log *zap.Logger, buf []byte, shareSize int, newDataCond *sync.Cond) *PieceBuffer

NewPieceBuffer creates and initializes a new PieceBuffer using buf as its internal content. If new data is written to the buffer, newDataCond will be notified.

func (*PieceBuffer) Close

func (b *PieceBuffer) Close() error

Close sets io.ErrClosedPipe to the buffer to prevent further writes and blocking on read.

func (*PieceBuffer) HasShare

func (b *PieceBuffer) HasShare(num int64) bool

HasShare checks if the num-th share can be read from the buffer without blocking. If there are older erasure shares in the buffer, they will be discarded to leave room for the newer erasure shares to be written.

func (*PieceBuffer) Read

func (b *PieceBuffer) Read(p []byte) (n int, err error)

Read reads the next len(p) bytes from the buffer or until the buffer is drained. The return value n is the number of bytes read. If the buffer has no data to return and no error is set, the call will block until new data is written to the buffer. Otherwise the error will be returned.

func (*PieceBuffer) ReadShare

func (b *PieceBuffer) ReadShare(num int64, p []byte) error

ReadShare reads the num-th erasure share from the buffer into p. Any shares before num will be discarded from the buffer.

func (*PieceBuffer) SetError

func (b *PieceBuffer) SetError(err error)

SetError sets an error to be returned by Read and Write. Read will return the error after all data is read from the buffer.

func (*PieceBuffer) Skip

func (b *PieceBuffer) Skip(n int) error

Skip advances the read pointer with n bytes. It the buffered number of bytes are less than n, the method will block until enough data is written to the buffer.

func (*PieceBuffer) Write

func (b *PieceBuffer) Write(p []byte) (n int, err error)

Write writes the contents of p into the buffer. If the buffer is full it will block until some data is read from it, or an error is set. The return value n is the number of bytes written. If an error was set, it be returned.

type RedundancyStrategy

type RedundancyStrategy struct {
	ErasureScheme
	// contains filtered or unexported fields
}

RedundancyStrategy is an ErasureScheme with a repair and optimal thresholds

func NewRedundancyStrategy

func NewRedundancyStrategy(es ErasureScheme, repairThreshold, optimalThreshold int) (RedundancyStrategy, error)

NewRedundancyStrategy from the given ErasureScheme, repair and optimal thresholds.

repairThreshold is the minimum repair threshold. If set to 0, it will be reset to the TotalCount of the ErasureScheme. optimalThreshold is the optimal threshold. If set to 0, it will be reset to the TotalCount of the ErasureScheme.

func NewRedundancyStrategyFromProto

func NewRedundancyStrategyFromProto(scheme *pb.RedundancyScheme) (RedundancyStrategy, error)

NewRedundancyStrategyFromProto creates new RedundancyStrategy from the given RedundancyScheme protobuf.

func NewRedundancyStrategyFromStorj added in v0.17.0

func NewRedundancyStrategyFromStorj(scheme storj.RedundancyScheme) (RedundancyStrategy, error)

NewRedundancyStrategyFromStorj creates new RedundancyStrategy from the given storj.RedundancyScheme.

func (*RedundancyStrategy) OptimalThreshold

func (rs *RedundancyStrategy) OptimalThreshold() int

OptimalThreshold is the number of available erasure pieces above which there is no need for the data to be repaired

func (*RedundancyStrategy) RepairThreshold

func (rs *RedundancyStrategy) RepairThreshold() int

RepairThreshold is the number of available erasure pieces below which the data must be repaired to avoid loss

type StripeReader

type StripeReader struct {
	// contains filtered or unexported fields
}

StripeReader can read and decodes stripes from a set of readers

func NewStripeReader

func NewStripeReader(log *zap.Logger, rs map[int]io.ReadCloser, es ErasureScheme, mbm int, forceErrorDetection bool) *StripeReader

NewStripeReader creates a new StripeReader from the given readers, erasure scheme and max buffer memory.

func (*StripeReader) Close

func (r *StripeReader) Close() error

Close closes the StripeReader and all PieceBuffers.

func (*StripeReader) ReadStripe

func (r *StripeReader) ReadStripe(ctx context.Context, num int64, p []byte) (_ []byte, err error)

ReadStripe reads and decodes the num-th stripe and concatenates it to p. The return value is the updated byte slice.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL