Documentation
¶
Overview ¶
Package split implements reading and writing of hashsplit trees in a blob store. See github.com/bobg/hashsplit for more information.
Index ¶
- Variables
- func Protect(ctx context.Context, g bs.Getter, ref bs.Ref) ([]gc.ProtectPair, error)
- type Child
- type Node
- func (*Node) Descriptor() ([]byte, []int)deprecated
- func (x *Node) GetLeaves() []*Child
- func (x *Node) GetNodes() []*Child
- func (x *Node) GetOffset() uint64
- func (x *Node) GetSize() uint64
- func (*Node) ProtoMessage()
- func (x *Node) ProtoReflect() protoreflect.Message
- func (x *Node) Reset()
- func (x *Node) String() string
- type Option
- type Reader
- type Writer
Constants ¶
This section is empty.
Variables ¶
var File_split_proto protoreflect.FileDescriptor
Functions ¶
Types ¶
type Child ¶
type Child struct {
// Ref of child (node or blob).
Ref []byte `protobuf:"bytes,1,opt,name=ref,proto3" json:"ref,omitempty"`
// Offset of this child (duplicates Node.off).
Offset uint64 `protobuf:"varint,2,opt,name=offset,proto3" json:"offset,omitempty"`
// contains filtered or unexported fields
}
func (*Child) Descriptor
deprecated
func (*Child) ProtoMessage ¶
func (*Child) ProtoMessage()
func (*Child) ProtoReflect ¶
func (x *Child) ProtoReflect() protoreflect.Message
type Node ¶
type Node struct {
// Offsets and refs of other Nodes.
Nodes []*Child `protobuf:"bytes,1,rep,name=nodes,proto3" json:"nodes,omitempty"`
// Offsets and refs of leaf blobs.
Leaves []*Child `protobuf:"bytes,2,rep,name=leaves,proto3" json:"leaves,omitempty"`
// Offset in the input stream of the beginning of the bytes represented by this child.
Offset uint64 `protobuf:"varint,3,opt,name=offset,proto3" json:"offset,omitempty"`
// Number of bytes represented by this node.
Size uint64 `protobuf:"varint,4,opt,name=size,proto3" json:"size,omitempty"`
// contains filtered or unexported fields
}
Node is a node in a hashsplit tree created with split.Write.
func (*Node) Descriptor
deprecated
func (*Node) ProtoMessage ¶
func (*Node) ProtoMessage()
func (*Node) ProtoReflect ¶
func (x *Node) ProtoReflect() protoreflect.Message
type Option ¶
type Option = func(*Writer)
Option is the type of an option passed to NewWriter.
func Bits ¶
Bits is an option for NewWriter that changes the number of trailing zero bits in the rolling checksum used to identify chunk boundaries. A chunk boundary occurs on average once every 2^n bytes. (But the actual median chunk size is the logarithm, base (2^n-1)/(2^n), of 0.5.) The default value for n is 16, producing a chunk boundary every 65,536 bytes, and a median chunk size of 45,426 bytes.
func Fanout ¶
Fanout is an option for NewWriter that can change the fanout of the nodes in the tree produced. The value of n must be 1 or higher. The default is 8. In a nutshell, nodes in the hashsplit tree will tend to have around 2n children each, because each chunk's "level" is reduced by dividing it by n. For more information see https://pkg.go.dev/github.com/bobg/hashsplit#TreeBuilder.Add.
type Reader ¶
Reader traverses the nodes of a hashsplit tree to produce the original (unsplit) input. It implements io.ReadSeeker. The given context object is stored in the Reader and used in subsequent calls to Read. This is an antipattern but acceptable when an object must adhere to a context-free stdlib interface (https://github.com/golang/go/wiki/CodeReviewComments#contexts). Callers may replace the context object during the lifetime of the Reader as needed.
func NewReader ¶
NewReader produces a new Reader for the hashsplit tree stored in g and rooted at ref.
type Writer ¶
type Writer struct {
Ctx context.Context
Root bs.Ref // populated by Close
// contains filtered or unexported fields
}
Writer is an io.WriteCloser that splits its input with a hashsplit.Splitter, writing the chunks to a bs.Store as separate blobs. It additionally assembles those chunks into a tree with a hashsplit.TreeBuilder. The tree nodes are also written to the bs.Store as serialized Node objects. The bs.Ref of the tree root is available as Writer.Root after a call to Close. If no data was written before the Writer was closed, Root will be bs.Zero.
func NewWriter ¶
NewWriter produces a new Writer writing to the given blob store. The given context object is stored in the Writer and used in subsequent calls to Write and Close. This is an antipattern but acceptable when an object must adhere to a context-free stdlib interface (https://github.com/golang/go/wiki/CodeReviewComments#contexts). Callers may replace the context object during the lifetime of the Writer as needed.
Source Files