chunks

package
v0.0.0-...-e5fa29d Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 27, 2021 License: Apache-2.0 Imports: 8 Imported by: 71

Documentation

Overview

Package chunks provides facilities for representing, storing, and fetching content-addressed chunks of Noms data.

Index

Constants

This section is empty.

Variables

View Source
var EmptyChunk = NewChunk([]byte{})

Functions

func Deserialize

func Deserialize(reader io.Reader, chunkChan chan<- *Chunk) (err error)

Deserialize reads off of |reader| until EOF, sending chunks to chunkChan in the order they are read. Objects sent over chunkChan are *Chunk.

func Serialize

func Serialize(chunk Chunk, writer io.Writer)

Serialize a single Chunk to writer.

Types

type AbsentManyRequest

type AbsentManyRequest struct {
	// contains filtered or unexported fields
}

func NewAbsentManyRequest

func NewAbsentManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- hash.Hash) AbsentManyRequest

func (AbsentManyRequest) Hashes

func (h AbsentManyRequest) Hashes() hash.HashSet

func (AbsentManyRequest) Outstanding

func (h AbsentManyRequest) Outstanding() OutstandingRequest

type AbsentRequest

type AbsentRequest struct {
	// contains filtered or unexported fields
}

func NewAbsentRequest

func NewAbsentRequest(r hash.Hash, ch chan<- bool) AbsentRequest

func (AbsentRequest) Hashes

func (h AbsentRequest) Hashes() hash.HashSet

func (AbsentRequest) Outstanding

func (h AbsentRequest) Outstanding() OutstandingRequest

type Chunk

type Chunk struct {
	// contains filtered or unexported fields
}

Chunk is a unit of stored data in noms

func NewChunk

func NewChunk(data []byte) Chunk

NewChunk creates a new Chunk backed by data. This means that the returned Chunk has ownership of this slice of memory.

func NewChunkWithHash

func NewChunkWithHash(r hash.Hash, data []byte) Chunk

NewChunkWithHash creates a new chunk with a known hash. The hash is not re-calculated or verified. This should obviously only be used in cases where the caller already knows the specified hash is correct.

func (Chunk) Data

func (c Chunk) Data() []byte

func (Chunk) Hash

func (c Chunk) Hash() hash.Hash

func (Chunk) IsEmpty

func (c Chunk) IsEmpty() bool

type ChunkStore

type ChunkStore interface {
	// Get the Chunk for the value of the hash in the store. If the hash is
	// absent from the store EmptyChunk is returned.
	Get(h hash.Hash) Chunk

	// GetMany gets the Chunks with |hashes| from the store. On return,
	// |foundChunks| will have been fully sent all chunks which have been
	// found. Any non-present chunks will silently be ignored.
	GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

	// Returns true iff the value at the address |h| is contained in the
	// store
	Has(h hash.Hash) bool

	// Returns a new HashSet containing any members of |hashes| that are
	// absent from the store.
	HasMany(hashes hash.HashSet) (absent hash.HashSet)

	// Put caches c in the ChunkSource. Upon return, c must be visible to
	// subsequent Get and Has calls, but must not be persistent until a call
	// to Flush(). Put may be called concurrently with other calls to Put(),
	// Get(), GetMany(), Has() and HasMany().
	Put(c Chunk)

	// Returns the NomsVersion with which this ChunkSource is compatible.
	Version() string

	// Rebase brings this ChunkStore into sync with the persistent storage's
	// current root.
	Rebase()

	// Root returns the root of the database as of the time the ChunkStore
	// was opened or the most recent call to Rebase.
	Root() hash.Hash

	// Commit atomically attempts to persist all novel Chunks and update the
	// persisted root hash from last to current (or keeps it the same).
	// If last doesn't match the root in persistent storage, returns false.
	Commit(current, last hash.Hash) bool

	// Stats may return some kind of struct that reports statistics about the
	// ChunkStore instance. The type is implementation-dependent, and impls
	// may return nil
	Stats() interface{}

	// StatsSummary may return a string containing summarized statistics for
	// this ChunkStore. It must return "Unsupported" if this operation is not
	// supported.
	StatsSummary() string

	// Close tears down any resources in use by the implementation. After
	// Close(), the ChunkStore may not be used again. It is NOT SAFE to call
	// Close() concurrently with any other ChunkStore method; behavior is
	// undefined and probably crashy.
	io.Closer
}

ChunkStore is the core storage abstraction in noms. We can put data anyplace we have a ChunkStore implementation for.

type ChunkWriter

type ChunkWriter struct {
	// contains filtered or unexported fields
}

ChunkWriter wraps an io.WriteCloser, additionally providing the ability to grab the resulting Chunk for all data written through the interface. Calling Chunk() or Close() on an instance disallows further writing.

func NewChunkWriter

func NewChunkWriter() *ChunkWriter

func (*ChunkWriter) Chunk

func (w *ChunkWriter) Chunk() Chunk

Chunk() closes the writer and returns the resulting Chunk.

func (*ChunkWriter) Close

func (w *ChunkWriter) Close() error

Close() closes computes the hash and Puts it into the ChunkSink Note: The Write() method never returns an error. Instead, like other noms interfaces, errors are reported via panic.

func (*ChunkWriter) Write

func (w *ChunkWriter) Write(data []byte) (int, error)

type Factory

type Factory interface {
	CreateStore(ns string) ChunkStore

	// CreateStoreFromCache allows caller to signal to the factory that it's
	// willing to tolerate an out-of-date ChunkStore.
	CreateStoreFromCache(ns string) ChunkStore

	// Shutter shuts down the factory. Subsequent calls to CreateStore() will fail.
	Shutter()
}

Factory allows the creation of namespaced ChunkStore instances. The details of how namespaces are separated is left up to the particular implementation of Factory and ChunkStore.

func NewMemoryStoreFactory

func NewMemoryStoreFactory() Factory

type GetManyRequest

type GetManyRequest struct {
	// contains filtered or unexported fields
}

func NewGetManyRequest

func NewGetManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- *Chunk) GetManyRequest

func (GetManyRequest) Hashes

func (g GetManyRequest) Hashes() hash.HashSet

func (GetManyRequest) Outstanding

func (g GetManyRequest) Outstanding() OutstandingRequest

type GetRequest

type GetRequest struct {
	// contains filtered or unexported fields
}

func NewGetRequest

func NewGetRequest(r hash.Hash, ch chan<- *Chunk) GetRequest

func (GetRequest) Hashes

func (g GetRequest) Hashes() hash.HashSet

func (GetRequest) Outstanding

func (g GetRequest) Outstanding() OutstandingRequest

type MemoryStorage

type MemoryStorage struct {
	// contains filtered or unexported fields
}

MemoryStorage provides a "persistent" storage layer to back multiple MemoryStoreViews. A MemoryStorage instance holds the ground truth for the root and set of chunks that are visible to all MemoryStoreViews vended by NewView(), allowing them to implement the transaction-style semantics that ChunkStore requires.

func (*MemoryStorage) Get

func (ms *MemoryStorage) Get(h hash.Hash) Chunk

Get retrieves the Chunk with the Hash h, returning EmptyChunk if it's not present.

func (*MemoryStorage) Has

func (ms *MemoryStorage) Has(r hash.Hash) bool

Has returns true if the Chunk with the Hash h is present in ms.data, false if not.

func (*MemoryStorage) Len

func (ms *MemoryStorage) Len() int

Len returns the number of Chunks in ms.data.

func (*MemoryStorage) NewView

func (ms *MemoryStorage) NewView() ChunkStore

NewView vends a MemoryStoreView backed by this MemoryStorage. It's initialized with the currently "persisted" root.

func (*MemoryStorage) Root

func (ms *MemoryStorage) Root() hash.Hash

Root returns the currently "persisted" root hash of this in-memory store.

func (*MemoryStorage) Update

func (ms *MemoryStorage) Update(current, last hash.Hash, novel map[hash.Hash]Chunk) bool

Update checks the "persisted" root against last and, iff it matches, updates the root to current, adds all of novel to ms.data, and returns true. Otherwise returns false.

type MemoryStoreView

type MemoryStoreView struct {
	// contains filtered or unexported fields
}

MemoryStoreView is an in-memory implementation of store.ChunkStore. Useful mainly for tests. The proper way to get one: storage := &MemoryStorage{} ms := storage.NewView()

func (*MemoryStoreView) Close

func (ms *MemoryStoreView) Close() error

func (*MemoryStoreView) Commit

func (ms *MemoryStoreView) Commit(current, last hash.Hash) bool

func (*MemoryStoreView) Get

func (ms *MemoryStoreView) Get(h hash.Hash) Chunk

func (*MemoryStoreView) GetMany

func (ms *MemoryStoreView) GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

func (*MemoryStoreView) Has

func (ms *MemoryStoreView) Has(h hash.Hash) bool

func (*MemoryStoreView) HasMany

func (ms *MemoryStoreView) HasMany(hashes hash.HashSet) hash.HashSet

func (*MemoryStoreView) Len

func (ms *MemoryStoreView) Len() int

func (*MemoryStoreView) Put

func (ms *MemoryStoreView) Put(c Chunk)

func (*MemoryStoreView) Rebase

func (ms *MemoryStoreView) Rebase()

func (*MemoryStoreView) Root

func (ms *MemoryStoreView) Root() hash.Hash

func (*MemoryStoreView) Stats

func (ms *MemoryStoreView) Stats() interface{}

func (*MemoryStoreView) StatsSummary

func (ms *MemoryStoreView) StatsSummary() string

func (*MemoryStoreView) Version

func (ms *MemoryStoreView) Version() string

type OutstandingAbsent

type OutstandingAbsent chan<- bool

func (OutstandingAbsent) Fail

func (oh OutstandingAbsent) Fail()

func (OutstandingAbsent) Satisfy

func (oh OutstandingAbsent) Satisfy(h hash.Hash, c *Chunk)

type OutstandingAbsentMany

type OutstandingAbsentMany struct {
	// contains filtered or unexported fields
}

func (OutstandingAbsentMany) Fail

func (ohm OutstandingAbsentMany) Fail()

func (OutstandingAbsentMany) Satisfy

func (ohm OutstandingAbsentMany) Satisfy(h hash.Hash, c *Chunk)

type OutstandingGet

type OutstandingGet chan<- *Chunk

func (OutstandingGet) Fail

func (r OutstandingGet) Fail()

func (OutstandingGet) Satisfy

func (r OutstandingGet) Satisfy(h hash.Hash, c *Chunk)

type OutstandingGetMany

type OutstandingGetMany struct {
	// contains filtered or unexported fields
}

func (OutstandingGetMany) Fail

func (ogm OutstandingGetMany) Fail()

func (OutstandingGetMany) Satisfy

func (ogm OutstandingGetMany) Satisfy(h hash.Hash, c *Chunk)

type OutstandingRequest

type OutstandingRequest interface {
	Satisfy(h hash.Hash, c *Chunk)
	Fail()
}

type ReadBatch

type ReadBatch map[hash.Hash][]OutstandingRequest

ReadBatch represents a set of queued Get/Has requests, each of which are blocking on a receive channel for a response.

func (*ReadBatch) Close

func (rb *ReadBatch) Close() error

Close ensures that callers to Get() and Has() are failed correctly if the corresponding chunk wasn't in the response from the server (i.e. it wasn't found).

type ReadRequest

type ReadRequest interface {
	Hashes() hash.HashSet
	Outstanding() OutstandingRequest
}

type TestStorage

type TestStorage struct {
	MemoryStorage
}

func (*TestStorage) NewView

func (t *TestStorage) NewView() *TestStoreView

type TestStoreFactory

type TestStoreFactory struct {
	// contains filtered or unexported fields
}

func NewTestStoreFactory

func NewTestStoreFactory() *TestStoreFactory

func (*TestStoreFactory) CreateStore

func (f *TestStoreFactory) CreateStore(ns string) ChunkStore

func (*TestStoreFactory) Shutter

func (f *TestStoreFactory) Shutter()

type TestStoreView

type TestStoreView struct {
	ChunkStore
	Reads  int
	Hases  int
	Writes int
}

func (*TestStoreView) Get

func (s *TestStoreView) Get(h hash.Hash) Chunk

func (*TestStoreView) GetMany

func (s *TestStoreView) GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

func (*TestStoreView) Has

func (s *TestStoreView) Has(h hash.Hash) bool

func (*TestStoreView) HasMany

func (s *TestStoreView) HasMany(hashes hash.HashSet) hash.HashSet

func (*TestStoreView) Put

func (s *TestStoreView) Put(c Chunk)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL