shares

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 9, 2023 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package shares provides primitives for splitting block data into shares and parsing shares back into block data.

Compact vs. Sparse

There are two types of shares:

  1. Compact
  2. Sparse

Compact shares can contain data from one or more unit (transactions or intermediate state roots). Sparse shares can contain data from zero or one blob. Compact shares and sparse shares are encoded differently. The motivation behind the distinction is that transactions and intermediate state roots are expected to have small lengths so they are encoded in compact shares to minimize the number of shares needed to store them. On the other hand, blobs are expected to be larger and have the desideratum that clients should be able to create proofs of blob inclusion. This desiradum is infeasible if client A's blob is encoded into a share with another client B's blob that is unknown to A. It follows that client A's blob is encoded into a share such that the contents can be determined by client A without any additional information. See message layout rational or adr-006-non-interactive-defaults for more details.

Universal Prefix

Both types of shares have a universal prefix. The first 1 byte of a share contains the namespace version. The next 32 bytes contain the namespace ID. The next one byte contains an InfoByte that contains the share version and a sequence start indicator. If the sequence start indicator is `1` (i.e. this is the first share of a sequence) then the next 4 bytes contain a big endian uint32 of the sequence length.

For the first share of a sequence: 

| namespace_version | namespace_id | info_byte | sequence_length | sequence_data             |
| 1 byte            | 32 bytes     | 1 byte    | 4 bytes         | remaining bytes of share  |

For continuation share of a sequence: 

| namespace_version | namespace_id | info_byte | sequence_data             |
| 1 byte            | 32 bytes     | 1 byte    | remaining bytes of share  |

The remaining bytes depend on the share type.

Compact Share Schema

The four bytes after the universal prefix are reserved for the location in the share of the first unit of data that starts in this share.

For the first compact share: 

| namespace_version | namespace_id | info_byte | sequence_length | location_of_first_unit | transactions or intermediate state roots            |
| 1 byte            | 32 bytes     | 1 byte    | 4 bytes         | 4 bytes                | remaining bytes of share                            |

For continuation compact share: 

| namespace_version | namespace_id | info_byte | location_of_first_unit | transactions or intermediate state roots            |
| 1 byte            | 32 bytes     | 1 byte    | 4 bytes                | remaining bytes of share                            |

Notes

  • All shares in a reserved namespace belong to one sequence.
  • Each unit (transaction or intermediate state root) in data is prefixed with a varint of the length of the unit.

Sparse Share Schema

The remaining bytes contain blob data.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CompactSharesNeeded 

func CompactSharesNeeded(sequenceLen int) (sharesNeeded int)

CompactSharesNeeded returns the number of compact shares needed to store a sequence of length sequenceLen. The parameter sequenceLen is the number of bytes of transactions or intermediate state roots in a sequence.

func DelimLen 

func DelimLen(size uint64) int

DelimLen calculates the length of the delimiter for a given unit size

func MarshalDelimitedTx 

func MarshalDelimitedTx(tx coretypes.Tx) ([]byte, error)

MarshalDelimitedTx prefixes a transaction with the length of the transaction encoded as a varint.

func NewReservedBytes 

func NewReservedBytes(byteIndex uint32) ([]byte, error)

NewReservedBytes returns a byte slice of length appconsts.CompactShareReservedBytes that contains the byteIndex of the first unit that starts in a compact share.

func ParseCompactShares 

func ParseCompactShares(shares []Share) (data [][]byte, err error)

func ParseDelimiter 

func ParseDelimiter(input []byte) (inputWithoutLenDelimiter []byte, unitLen uint64, err error)

ParseDelimiter attempts to parse a varint length delimiter from the input provided. It returns the input without the len delimiter bytes, the length parsed from the varint optionally an error. Unit length delimiters are used in compact shares where units (i.e. a transaction) are prefixed with a length delimiter that is encoded as a varint. Input should not contain the namespace ID or info byte of a share.

func ParseReservedBytes 

func ParseReservedBytes(reservedBytes []byte) (uint32, error)

ParseReservedBytes parses a byte slice of length appconsts.CompactShareReservedBytes into a byteIndex.

func ParseTxs 

func ParseTxs(shares []Share) (coretypes.Txs, error)

ParseTxs collects all of the transactions from the shares provided

func SparseSharesNeeded 

func SparseSharesNeeded(sequenceLen uint32) (sharesNeeded int)

SparseSharesNeeded returns the number of shares needed to store a sequence of length sequenceLen.

func ToBytes 

func ToBytes(shares []Share) (bytes [][]byte)

func TxsFromBytes 

func TxsFromBytes(txs [][]byte) coretypes.Txs

func TxsToBytes 

func TxsToBytes(txs coretypes.Txs) [][]byte

Types

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

func NewBuilder 

func NewBuilder(ns appns.Namespace, shareVersion uint8, isFirstShare bool) *Builder

Init() needs to be called right after this method

func NewEmptyBuilder 

func NewEmptyBuilder() *Builder

func (*Builder) AddData 

func (b *Builder) AddData(rawData []byte) (rawDataLeftOver []byte)

func (*Builder) AvailableBytes 

func (b *Builder) AvailableBytes() int

func (*Builder) Build 

func (b *Builder) Build() (*Share, error)

func (*Builder) FlipSequenceStart 

func (b *Builder) FlipSequenceStart()

FlipSequenceStart flips the sequence start indicator of the share provided

func (*Builder) ImportRawShare 

func (b *Builder) ImportRawShare(rawBytes []byte) *Builder

func (*Builder) Init 

func (b *Builder) Init() (*Builder, error)

func (*Builder) IsEmptyShare 

func (b *Builder) IsEmptyShare() bool

IsEmptyShare returns true if no data has been written to the share

func (*Builder) MaybeWriteReservedBytes 

func (b *Builder) MaybeWriteReservedBytes() error

MaybeWriteReservedBytes will be a no-op for a compact share or if the reserved bytes have already been populated. If the reserved bytes are empty, it will write the location of the next unit of data to the reserved bytes.

func (*Builder) WriteSequenceLen 

func (b *Builder) WriteSequenceLen(sequenceLen uint32) error

writeSequenceLen writes the sequence length to the first share.

func (*Builder) ZeroPadIfNecessary 

func (b *Builder) ZeroPadIfNecessary() (bytesOfPadding int)

type CompactShareSplitter 

type CompactShareSplitter struct {
	// contains filtered or unexported fields
}

CompactShareSplitter will write raw data compactly across a progressively increasing set of shares. It is used to lazily split block data such as transactions or intermediate state roots into shares.

func NewCompactShareSplitter 

func NewCompactShareSplitter(ns appns.Namespace, shareVersion uint8) *CompactShareSplitter

NewCompactShareSplitter returns a CompactShareSplitter using the provided namespace and shareVersion.

func NewCompactShareSplitterWithIsCompactFalse 

func NewCompactShareSplitterWithIsCompactFalse(ns appns.Namespace, shareVersion uint8) *CompactShareSplitter

NewCompactShareSplitterWithIsCompactFalse returns a CompactShareSplitter using the provided namespace and shareVersion. Also, sets isCompact in the builder to false.

func (*CompactShareSplitter) Count 

func (css *CompactShareSplitter) Count() (shareCount int)

Count returns the number of shares that would be made if `Export` was invoked on this compact share splitter.

func (*CompactShareSplitter) Export 

func (css *CompactShareSplitter) Export(shareRangeOffset int) ([]Share, map[coretypes.TxKey]ShareRange, error)

Export finalizes and returns the underlying compact shares and a map of shareRanges. All share ranges in the map of shareRanges will be offset (i.e. incremented) by the shareRangeOffset provided. shareRangeOffset should be 0 for the first compact share sequence in the data square (transactions) but should be some non-zero number for subsequent compact share sequences (e.g. pfb txs).

func (*CompactShareSplitter) WriteTx 

func (css *CompactShareSplitter) WriteTx(tx coretypes.Tx) error

WriteTx adds the delimited data for the provided tx to the underlying compact share splitter.

func (*CompactShareSplitter) WriteWithNoReservedBytes 

func (css *CompactShareSplitter) WriteWithNoReservedBytes(rawData []byte) error

write adds the delimited data to the underlying compact shares.

type InfoByte 

type InfoByte byte

InfoByte is a byte with the following structure: the first 7 bits are reserved for version information in big endian form (initially `0000000`). The last bit is a "sequence start indicator", that is `1` if this is the first share of a sequence and `0` if this is a continuation share.

func NewInfoByte 

func NewInfoByte(version uint8, isSequenceStart bool) (InfoByte, error)

func ParseInfoByte 

func ParseInfoByte(i byte) (InfoByte, error)

func (InfoByte) IsSequenceStart 

func (i InfoByte) IsSequenceStart() bool

IsSequenceStart returns whether this share is the start of a sequence.

func (InfoByte) Version 

func (i InfoByte) Version() uint8

Version returns the version encoded in this InfoByte. Version is expected to be between 0 and appconsts.MaxShareVersion (inclusive).

type Range 

type Range struct {
	// Start is the index of the first share occupied by this range.
	Start int
	// End is the next index after the last share occupied by this range.
	End int
}

Range is an end exclusive set of share indexes.

func EmptyRange 

func EmptyRange() Range

func NewRange 

func NewRange(start, end int) Range

func (*Range) Add 

func (r *Range) Add(value int)

func (Range) IsEmpty 

func (r Range) IsEmpty() bool

type Share 

type Share struct {
	// contains filtered or unexported fields
}

Share contains the raw share data (including namespace ID).

func FromBytes 

func FromBytes(bytes [][]byte) (shares []Share, err error)

func NewShare 

func NewShare(data []byte) (*Share, error)

func SplitTxs 

func SplitTxs(txs coretypes.Txs) (txShares []Share, err error)

func (*Share) DoesSupportVersions 

func (s *Share) DoesSupportVersions(supportedShareVersions []uint8) error

func (*Share) InfoByte 

func (s *Share) InfoByte() (InfoByte, error)

func (Share) IsCompactShare 

func (s Share) IsCompactShare() (bool, error)

IsCompactShare returns true if this is a compact share.

func (*Share) IsPadding 

func (s *Share) IsPadding() (bool, error)

IsPadding returns whether this *share is padding or not.

func (*Share) IsSequenceStart 

func (s *Share) IsSequenceStart() (bool, error)

IsSequenceStart returns true if this is the first share in a sequence.

func (*Share) Len 

func (s *Share) Len() int

func (*Share) Namespace 

func (s *Share) Namespace() (appns.Namespace, error)

func (*Share) RawData 

func (s *Share) RawData() (rawData []byte, err error)

RawData returns the raw share data. The raw share data does not contain the namespace ID, info byte, sequence length, or reserved bytes.

func (*Share) RawDataUsingReserved 

func (s *Share) RawDataUsingReserved() (rawData []byte, err error)

RawDataWithReserved returns the raw share data while taking reserved bytes into account.

func (*Share) RawDataWithReserved 

func (s *Share) RawDataWithReserved() (rawData []byte, err error)

RawDataWithReserved returns the raw share data including the reserved bytes. The raw share data does not contain the namespace ID, info byte, or sequence length.

func (*Share) SequenceLen 

func (s *Share) SequenceLen() (sequenceLen uint32, err error)

SequenceLen returns the sequence length of this *share and optionally an error. It returns 0, nil if this is a continuation share (i.e. doesn't contain a sequence length).

func (*Share) ToBytes 

func (s *Share) ToBytes() []byte

func (*Share) Validate 

func (s *Share) Validate() error

func (*Share) Version 

func (s *Share) Version() (uint8, error)

type ShareRange 

type ShareRange struct {
	// Start is the index of the first share occupied by this range.
	Start int
	// End is the index of the last share occupied by this range.
	End int
}

type ShareSequence 

type ShareSequence struct {
	Namespace appns.Namespace
	Shares    []Share
}

ShareSequence represents a contiguous sequence of shares that are part of the same namespace and blob. For compact shares, one share sequence exists per reserved namespace. For sparse shares, one share sequence exists per blob.

func (ShareSequence) RawData 

func (s ShareSequence) RawData() (data []byte, err error)

RawData returns the raw share data of this share sequence. The raw data does not contain the namespace ID, info byte, sequence length, or reserved bytes.

func (ShareSequence) SequenceLen 

func (s ShareSequence) SequenceLen() (uint32, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.