chunks

package
v0.0.0-...-e5fa29d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2021 License: Apache-2.0 Imports: 8 Imported by: 71

Documentation

Overview

Package chunks provides facilities for representing, storing, and fetching content-addressed chunks of Noms data.

Index

Constants

This section is empty.

Variables

View Source 
var EmptyChunk = NewChunk([]byte{})

Functions

func Deserialize 

func Deserialize(reader io.Reader, chunkChan chan<- *Chunk) (err error)

Deserialize reads off of |reader| until EOF, sending chunks to chunkChan in the order they are read. Objects sent over chunkChan are *Chunk.

func Serialize 

func Serialize(chunk Chunk, writer io.Writer)

Serialize a single Chunk to writer.

Types

type AbsentManyRequest 

type AbsentManyRequest struct {
	// contains filtered or unexported fields
}

func NewAbsentManyRequest 

func NewAbsentManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- hash.Hash) AbsentManyRequest

func (AbsentManyRequest) Hashes 

func (h AbsentManyRequest) Hashes() hash.HashSet

func (AbsentManyRequest) Outstanding 

func (h AbsentManyRequest) Outstanding() OutstandingRequest

type AbsentRequest 

type AbsentRequest struct {
	// contains filtered or unexported fields
}

func NewAbsentRequest 

func NewAbsentRequest(r hash.Hash, ch chan<- bool) AbsentRequest

func (AbsentRequest) Hashes 

func (h AbsentRequest) Hashes() hash.HashSet

func (AbsentRequest) Outstanding 

func (h AbsentRequest) Outstanding() OutstandingRequest

type Chunk 

type Chunk struct {
	// contains filtered or unexported fields
}

Chunk is a unit of stored data in noms

func NewChunk 

func NewChunk(data []byte) Chunk

NewChunk creates a new Chunk backed by data. This means that the returned Chunk has ownership of this slice of memory.

func NewChunkWithHash 

func NewChunkWithHash(r hash.Hash, data []byte) Chunk

NewChunkWithHash creates a new chunk with a known hash. The hash is not re-calculated or verified. This should obviously only be used in cases where the caller already knows the specified hash is correct.

func (Chunk) Data 

func (c Chunk) Data() []byte

func (Chunk) Hash 

func (c Chunk) Hash() hash.Hash

func (Chunk) IsEmpty 

func (c Chunk) IsEmpty() bool

type ChunkStore 

type ChunkStore interface {
	// Get the Chunk for the value of the hash in the store. If the hash is
	// absent from the store EmptyChunk is returned.
	Get(h hash.Hash) Chunk

	// GetMany gets the Chunks with |hashes| from the store. On return,
	// |foundChunks| will have been fully sent all chunks which have been
	// found. Any non-present chunks will silently be ignored.
	GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

	// Returns true iff the value at the address |h| is contained in the
	// store
	Has(h hash.Hash) bool

	// Returns a new HashSet containing any members of |hashes| that are
	// absent from the store.
	HasMany(hashes hash.HashSet) (absent hash.HashSet)

	// Put caches c in the ChunkSource. Upon return, c must be visible to
	// subsequent Get and Has calls, but must not be persistent until a call
	// to Flush(). Put may be called concurrently with other calls to Put(),
	// Get(), GetMany(), Has() and HasMany().
	Put(c Chunk)

	// Returns the NomsVersion with which this ChunkSource is compatible.
	Version() string

	// Rebase brings this ChunkStore into sync with the persistent storage's
	// current root.
	Rebase()

	// Root returns the root of the database as of the time the ChunkStore
	// was opened or the most recent call to Rebase.
	Root() hash.Hash

	// Commit atomically attempts to persist all novel Chunks and update the
	// persisted root hash from last to current (or keeps it the same).
	// If last doesn't match the root in persistent storage, returns false.
	Commit(current, last hash.Hash) bool

	// Stats may return some kind of struct that reports statistics about the
	// ChunkStore instance. The type is implementation-dependent, and impls
	// may return nil
	Stats() interface{}

	// StatsSummary may return a string containing summarized statistics for
	// this ChunkStore. It must return "Unsupported" if this operation is not
	// supported.
	StatsSummary() string

	// Close tears down any resources in use by the implementation. After
	// Close(), the ChunkStore may not be used again. It is NOT SAFE to call
	// Close() concurrently with any other ChunkStore method; behavior is
	// undefined and probably crashy.
	io.Closer
}

ChunkStore is the core storage abstraction in noms. We can put data anyplace we have a ChunkStore implementation for.

type ChunkWriter 

type ChunkWriter struct {
	// contains filtered or unexported fields
}

ChunkWriter wraps an io.WriteCloser, additionally providing the ability to grab the resulting Chunk for all data written through the interface. Calling Chunk() or Close() on an instance disallows further writing.

func NewChunkWriter 

func NewChunkWriter() *ChunkWriter

func (*ChunkWriter) Chunk 

func (w *ChunkWriter) Chunk() Chunk

Chunk() closes the writer and returns the resulting Chunk.

func (*ChunkWriter) Close 

func (w *ChunkWriter) Close() error

Close() closes computes the hash and Puts it into the ChunkSink Note: The Write() method never returns an error. Instead, like other noms interfaces, errors are reported via panic.

func (*ChunkWriter) Write 

func (w *ChunkWriter) Write(data []byte) (int, error)

type Factory 

type Factory interface {
	CreateStore(ns string) ChunkStore

	// CreateStoreFromCache allows caller to signal to the factory that it's
	// willing to tolerate an out-of-date ChunkStore.
	CreateStoreFromCache(ns string) ChunkStore

	// Shutter shuts down the factory. Subsequent calls to CreateStore() will fail.
	Shutter()
}

Factory allows the creation of namespaced ChunkStore instances. The details of how namespaces are separated is left up to the particular implementation of Factory and ChunkStore.

func NewMemoryStoreFactory 

func NewMemoryStoreFactory() Factory

type GetManyRequest 

type GetManyRequest struct {
	// contains filtered or unexported fields
}

func NewGetManyRequest 

func NewGetManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- *Chunk) GetManyRequest

func (GetManyRequest) Hashes 

func (g GetManyRequest) Hashes() hash.HashSet

func (GetManyRequest) Outstanding 

func (g GetManyRequest) Outstanding() OutstandingRequest

type GetRequest 

type GetRequest struct {
	// contains filtered or unexported fields
}

func NewGetRequest 

func NewGetRequest(r hash.Hash, ch chan<- *Chunk) GetRequest

func (GetRequest) Hashes 

func (g GetRequest) Hashes() hash.HashSet

func (GetRequest) Outstanding 

func (g GetRequest) Outstanding() OutstandingRequest

type MemoryStorage 

type MemoryStorage struct {
	// contains filtered or unexported fields
}

MemoryStorage provides a "persistent" storage layer to back multiple MemoryStoreViews. A MemoryStorage instance holds the ground truth for the root and set of chunks that are visible to all MemoryStoreViews vended by NewView(), allowing them to implement the transaction-style semantics that ChunkStore requires.

func (*MemoryStorage) Get 

func (ms *MemoryStorage) Get(h hash.Hash) Chunk

Get retrieves the Chunk with the Hash h, returning EmptyChunk if it's not present.

func (*MemoryStorage) Has 

func (ms *MemoryStorage) Has(r hash.Hash) bool

Has returns true if the Chunk with the Hash h is present in ms.data, false if not.

func (*MemoryStorage) Len 

func (ms *MemoryStorage) Len() int

Len returns the number of Chunks in ms.data.

func (*MemoryStorage) NewView 

func (ms *MemoryStorage) NewView() ChunkStore

NewView vends a MemoryStoreView backed by this MemoryStorage. It's initialized with the currently "persisted" root.

func (*MemoryStorage) Root 

func (ms *MemoryStorage) Root() hash.Hash

Root returns the currently "persisted" root hash of this in-memory store.

func (*MemoryStorage) Update 

func (ms *MemoryStorage) Update(current, last hash.Hash, novel map[hash.Hash]Chunk) bool

Update checks the "persisted" root against last and, iff it matches, updates the root to current, adds all of novel to ms.data, and returns true. Otherwise returns false.

type MemoryStoreView 

type MemoryStoreView struct {
	// contains filtered or unexported fields
}

MemoryStoreView is an in-memory implementation of store.ChunkStore. Useful mainly for tests. The proper way to get one: storage := &MemoryStorage{} ms := storage.NewView()

func (*MemoryStoreView) Close 

func (ms *MemoryStoreView) Close() error

func (*MemoryStoreView) Commit 

func (ms *MemoryStoreView) Commit(current, last hash.Hash) bool

func (*MemoryStoreView) Get 

func (ms *MemoryStoreView) Get(h hash.Hash) Chunk

func (*MemoryStoreView) GetMany 

func (ms *MemoryStoreView) GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

func (*MemoryStoreView) Has 

func (ms *MemoryStoreView) Has(h hash.Hash) bool

func (*MemoryStoreView) HasMany 

func (ms *MemoryStoreView) HasMany(hashes hash.HashSet) hash.HashSet

func (*MemoryStoreView) Len 

func (ms *MemoryStoreView) Len() int

func (*MemoryStoreView) Put 

func (ms *MemoryStoreView) Put(c Chunk)

func (*MemoryStoreView) Rebase 

func (ms *MemoryStoreView) Rebase()

func (*MemoryStoreView) Root 

func (ms *MemoryStoreView) Root() hash.Hash

func (*MemoryStoreView) Stats 

func (ms *MemoryStoreView) Stats() interface{}

func (*MemoryStoreView) StatsSummary 

func (ms *MemoryStoreView) StatsSummary() string

func (*MemoryStoreView) Version 

func (ms *MemoryStoreView) Version() string

type OutstandingAbsent 

type OutstandingAbsent chan<- bool

func (OutstandingAbsent) Fail 

func (oh OutstandingAbsent) Fail()

func (OutstandingAbsent) Satisfy 

func (oh OutstandingAbsent) Satisfy(h hash.Hash, c *Chunk)

type OutstandingAbsentMany 

type OutstandingAbsentMany struct {
	// contains filtered or unexported fields
}

func (OutstandingAbsentMany) Fail 

func (ohm OutstandingAbsentMany) Fail()

func (OutstandingAbsentMany) Satisfy 

func (ohm OutstandingAbsentMany) Satisfy(h hash.Hash, c *Chunk)

type OutstandingGet 

type OutstandingGet chan<- *Chunk

func (OutstandingGet) Fail 

func (r OutstandingGet) Fail()

func (OutstandingGet) Satisfy 

func (r OutstandingGet) Satisfy(h hash.Hash, c *Chunk)

type OutstandingGetMany 

type OutstandingGetMany struct {
	// contains filtered or unexported fields
}

func (OutstandingGetMany) Fail 

func (ogm OutstandingGetMany) Fail()

func (OutstandingGetMany) Satisfy 

func (ogm OutstandingGetMany) Satisfy(h hash.Hash, c *Chunk)

type OutstandingRequest 

type OutstandingRequest interface {
	Satisfy(h hash.Hash, c *Chunk)
	Fail()
}

type ReadBatch 

type ReadBatch map[hash.Hash][]OutstandingRequest

ReadBatch represents a set of queued Get/Has requests, each of which are blocking on a receive channel for a response.

func (*ReadBatch) Close 

func (rb *ReadBatch) Close() error

Close ensures that callers to Get() and Has() are failed correctly if the corresponding chunk wasn't in the response from the server (i.e. it wasn't found).

type ReadRequest 

type ReadRequest interface {
	Hashes() hash.HashSet
	Outstanding() OutstandingRequest
}

type TestStorage 

type TestStorage struct {
	MemoryStorage
}

func (*TestStorage) NewView 

func (t *TestStorage) NewView() *TestStoreView

type TestStoreFactory 

type TestStoreFactory struct {
	// contains filtered or unexported fields
}

func NewTestStoreFactory 

func NewTestStoreFactory() *TestStoreFactory

func (*TestStoreFactory) CreateStore 

func (f *TestStoreFactory) CreateStore(ns string) ChunkStore

func (*TestStoreFactory) Shutter 

func (f *TestStoreFactory) Shutter()

type TestStoreView 

type TestStoreView struct {
	ChunkStore
	Reads  int
	Hases  int
	Writes int
}

func (*TestStoreView) Get 

func (s *TestStoreView) Get(h hash.Hash) Chunk

func (*TestStoreView) GetMany 

func (s *TestStoreView) GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

func (*TestStoreView) Has 

func (s *TestStoreView) Has(h hash.Hash) bool

func (*TestStoreView) HasMany 

func (s *TestStoreView) HasMany(hashes hash.HashSet) hash.HashSet

func (*TestStoreView) Put 

func (s *TestStoreView) Put(c Chunk)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.