chunks

package
v0.40.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 19, 2022 License: Apache-2.0 Imports: 12 Imported by: 1

Documentation

Overview

Package chunks provides facilities for representing, storing, and fetching content-addressed chunks of Noms data.

Index

Constants

This section is empty.

Variables

View Source 
var EmptyChunk = NewChunk([]byte{})
View Source 
var ErrGCGenerationExpired = errors.New("garbage collection generation expired")
View Source 
var ErrNothingToCollect = errors.New("no changes since last gc")
View Source 
var ErrUnsupportedOperation = errors.New("operation not supported")

Functions

func Deserialize 

func Deserialize(reader io.Reader, chunkChan chan<- *Chunk) (err error)

Deserialize reads off of |reader| until EOF, sending chunks to chunkChan in the order they are read. Objects sent over chunkChan are *Chunk.

func NewMemoryStoreFactory 

func NewMemoryStoreFactory() *memoryStoreFactory

func Serialize 

func Serialize(chunk Chunk, writer io.Writer)

Serialize a single Chunk to writer.

Types

type AbsentManyRequest 

type AbsentManyRequest struct {
	// contains filtered or unexported fields
}

func NewAbsentManyRequest 

func NewAbsentManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- hash.Hash) AbsentManyRequest

func (AbsentManyRequest) Hashes 

func (h AbsentManyRequest) Hashes() hash.HashSet

func (AbsentManyRequest) Outstanding 

func (h AbsentManyRequest) Outstanding() OutstandingRequest

type AbsentRequest 

type AbsentRequest struct {
	// contains filtered or unexported fields
}

func NewAbsentRequest 

func NewAbsentRequest(r hash.Hash, ch chan<- bool) AbsentRequest

func (AbsentRequest) Hashes 

func (h AbsentRequest) Hashes() hash.HashSet

func (AbsentRequest) Outstanding 

func (h AbsentRequest) Outstanding() OutstandingRequest

type CSMetricWrapper 

type CSMetricWrapper struct {
	TotalChunkGets      int32
	TotalChunkHasChecks int32
	TotalChunkPuts      int32
	// contains filtered or unexported fields
}

CSMetricWrapper is a ChunkStore implementation that wraps a ChunkStore, and collects metrics on the calls.

func NewCSMetricWrapper 

func NewCSMetricWrapper(cs ChunkStore) *CSMetricWrapper

NewCSMetricWrapper returns a new CSMetricWrapper

func (*CSMetricWrapper) Close 

func (csMW *CSMetricWrapper) Close() error

Close tears down any resources in use by the implementation. After Close(), the ChunkStore may not be used again. It is NOT SAFE to call Close() concurrently with any other ChunkStore method; behavior is undefined and probably crashy.

func (*CSMetricWrapper) Commit 

func (csMW *CSMetricWrapper) Commit(ctx context.Context, current, last hash.Hash) (bool, error)

Commit atomically attempts to persist all novel Chunks and update the persisted root hash from last to current (or keeps it the same). If last doesn't match the root in persistent storage, returns false.

func (*CSMetricWrapper) Get 

func (csMW *CSMetricWrapper) Get(ctx context.Context, h hash.Hash) (Chunk, error)

Get the Chunk for the value of the hash in the store. If the hash is absent from the store EmptyChunk is returned.

func (*CSMetricWrapper) GetMany 

func (csMW *CSMetricWrapper) GetMany(ctx context.Context, hashes hash.HashSet, found func(context.Context, *Chunk)) error

GetMany gets the Chunks with |hashes| from the store. On return, |foundChunks| will have been fully sent all chunks which have been found. Any non-present chunks will silently be ignored.

func (*CSMetricWrapper) Has 

func (csMW *CSMetricWrapper) Has(ctx context.Context, h hash.Hash) (bool, error)

Returns true iff the value at the address |h| is contained in the store

func (*CSMetricWrapper) HasMany 

func (csMW *CSMetricWrapper) HasMany(ctx context.Context, hashes hash.HashSet) (absent hash.HashSet, err error)

Returns a new HashSet containing any members of |hashes| that are absent from the store.

func (*CSMetricWrapper) Put 

func (csMW *CSMetricWrapper) Put(ctx context.Context, c Chunk) error

Put caches c in the ChunkSource. Upon return, c must be visible to subsequent Get and Has calls, but must not be persistent until a call to Flush(). Put may be called concurrently with other calls to Put(), Get(), GetMany(), Has() and HasMany().

func (*CSMetricWrapper) Rebase 

func (csMW *CSMetricWrapper) Rebase(ctx context.Context) error

Rebase brings this ChunkStore into sync with the persistent storage's current root.

func (*CSMetricWrapper) Root 

func (csMW *CSMetricWrapper) Root(ctx context.Context) (hash.Hash, error)

Root returns the root of the database as of the time the ChunkStore was opened or the most recent call to Rebase.

func (*CSMetricWrapper) Stats 

func (csMW *CSMetricWrapper) Stats() interface{}

Stats may return some kind of struct that reports statistics about the ChunkStore instance. The type is implementation-dependent, and impls may return nil

func (*CSMetricWrapper) StatsSummary 

func (csMW *CSMetricWrapper) StatsSummary() string

StatsSummary may return a string containing summarized statistics for this ChunkStore. It must return "Unsupported" if this operation is not supported.

func (*CSMetricWrapper) Version 

func (csMW *CSMetricWrapper) Version() string

Returns the NomsVersion with which this ChunkSource is compatible.

type CSMetrics 

type CSMetrics struct {
	TotalChunkGets      int32
	TotalChunkHasChecks int32
	TotalChunkPuts      int32
	Delegate            interface{}
	DelegateSummary     string
}

CSMetrics contains the metrics aggregated by a CSMetricsWrapper

func NewCSMetrics 

func NewCSMetrics(csMW *CSMetricWrapper) CSMetrics

NewCSMetrics creates a CSMetrics instance

func (CSMetrics) String 

func (csm CSMetrics) String() string

String prints CSMetrics as JSON with indenting

type Chunk 

type Chunk struct {
	// contains filtered or unexported fields
}

Chunk is a unit of stored data in noms

func DeserializeData 

func DeserializeData(data []byte) (Chunk, error)

DeserializeData deserializes a chunk from a byte array

func NewChunk 

func NewChunk(data []byte) Chunk

NewChunk creates a new Chunk backed by data. This means that the returned Chunk has ownership of this slice of memory.

func NewChunkWithHash 

func NewChunkWithHash(r hash.Hash, data []byte) Chunk

NewChunkWithHash creates a new chunk with a known hash. The hash is not re-calculated or verified. This should obviously only be used in cases where the caller already knows the specified hash is correct.

func (Chunk) Data 

func (c Chunk) Data() []byte

func (Chunk) Hash 

func (c Chunk) Hash() hash.Hash

func (Chunk) IsEmpty 

func (c Chunk) IsEmpty() bool

func (Chunk) Size 

func (c Chunk) Size() int

func (Chunk) ToChunk 

func (c Chunk) ToChunk() (Chunk, error)

type ChunkStore 

type ChunkStore interface {
	// Get the Chunk for the value of the hash in the store. If the hash is
	// absent from the store EmptyChunk is returned.
	Get(ctx context.Context, h hash.Hash) (Chunk, error)

	// GetMany gets the Chunks with |hashes| from the store. On return,
	// |foundChunks| will have been fully sent all chunks which have been
	// found. Any non-present chunks will silently be ignored.
	GetMany(ctx context.Context, hashes hash.HashSet, found func(context.Context, *Chunk)) error

	// Returns true iff the value at the address |h| is contained in the
	// store
	Has(ctx context.Context, h hash.Hash) (bool, error)

	// Returns a new HashSet containing any members of |hashes| that are
	// absent from the store.
	HasMany(ctx context.Context, hashes hash.HashSet) (absent hash.HashSet, err error)

	// Put caches c in the ChunkSource. Upon return, c must be visible to
	// subsequent Get and Has calls, but must not be persistent until a call
	// to Flush(). Put may be called concurrently with other calls to Put(),
	// Get(), GetMany(), Has() and HasMany().
	Put(ctx context.Context, c Chunk) error

	// Returns the NomsVersion with which this ChunkSource is compatible.
	Version() string

	// Rebase brings this ChunkStore into sync with the persistent storage's
	// current root.
	Rebase(ctx context.Context) error

	// Root returns the root of the database as of the time the ChunkStore
	// was opened or the most recent call to Rebase.
	Root(ctx context.Context) (hash.Hash, error)

	// Commit atomically attempts to persist all novel Chunks and update the
	// persisted root hash from last to current (or keeps it the same).
	// If last doesn't match the root in persistent storage, returns false.
	Commit(ctx context.Context, current, last hash.Hash) (bool, error)

	// Stats may return some kind of struct that reports statistics about the
	// ChunkStore instance. The type is implementation-dependent, and impls
	// may return nil
	Stats() interface{}

	// StatsSummary may return a string containing summarized statistics for
	// this ChunkStore. It must return "Unsupported" if this operation is not
	// supported.
	StatsSummary() string

	// Close tears down any resources in use by the implementation. After
	// Close(), the ChunkStore may not be used again. It is NOT SAFE to call
	// Close() concurrently with any other ChunkStore method; behavior is
	// undefined and probably crashy.
	io.Closer
}

ChunkStore is the core storage abstraction in noms. We can put data anyplace we have a ChunkStore implementation for.

type ChunkStoreGarbageCollector 

type ChunkStoreGarbageCollector interface {
	ChunkStore

	// MarkAndSweepChunks expects |keepChunks| to receive the chunk hashes
	// that should be kept in the chunk store. Once |keepChunks| is closed
	// and MarkAndSweepChunks returns, the chunk store will only have the
	// chunks sent on |keepChunks| and will have removed all other content
	// from the ChunkStore.
	MarkAndSweepChunks(ctx context.Context, last hash.Hash, keepChunks <-chan []hash.Hash, dest ChunkStore) error
}

ChunkStoreGarbageCollector is a ChunkStore that supports garbage collection.

type ChunkWriter 

type ChunkWriter struct {
	// contains filtered or unexported fields
}

ChunkWriter wraps an io.WriteCloser, additionally providing the ability to grab the resulting Chunk for all data written through the interface. Calling Chunk() or Close() on an instance disallows further writing.

func NewChunkWriter 

func NewChunkWriter() *ChunkWriter

func (*ChunkWriter) Chunk 

func (w *ChunkWriter) Chunk() Chunk

Chunk() closes the writer and returns the resulting Chunk.

func (*ChunkWriter) Close 

func (w *ChunkWriter) Close() error

Close() closes computes the hash and Puts it into the ChunkSink Note: The Write() method never returns an error. Instead, like other noms interfaces, errors are reported via panic.

func (*ChunkWriter) Write 

func (w *ChunkWriter) Write(data []byte) (int, error)

type DebugLogger 

type DebugLogger interface {
	Logf(fmt string, args ...interface{})
}

type GenerationalCS 

type GenerationalCS interface {
	NewGen() ChunkStoreGarbageCollector
	OldGen() ChunkStoreGarbageCollector
}

GenerationalCS is an interface supporting the getting old gen and new gen chunk stores

type GetManyRequest 

type GetManyRequest struct {
	// contains filtered or unexported fields
}

func NewGetManyRequest 

func NewGetManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- *Chunk) GetManyRequest

func (GetManyRequest) Hashes 

func (g GetManyRequest) Hashes() hash.HashSet

func (GetManyRequest) Outstanding 

func (g GetManyRequest) Outstanding() OutstandingRequest

type GetRequest 

type GetRequest struct {
	// contains filtered or unexported fields
}

func NewGetRequest 

func NewGetRequest(r hash.Hash, ch chan<- *Chunk) GetRequest

func (GetRequest) Hashes 

func (g GetRequest) Hashes() hash.HashSet

func (GetRequest) Outstanding 

func (g GetRequest) Outstanding() OutstandingRequest

type LoggingChunkStore 

type LoggingChunkStore interface {
	ChunkStore
	SetLogger(logger DebugLogger)
}

type MemoryStorage 

type MemoryStorage struct {
	// contains filtered or unexported fields
}

MemoryStorage provides a "persistent" storage layer to back multiple MemoryStoreViews. A MemoryStorage instance holds the ground truth for the root and set of chunks that are visible to all MemoryStoreViews vended by NewView(), allowing them to implement the transaction-style semantics that ChunkStore requires.

func (*MemoryStorage) Get 

func (ms *MemoryStorage) Get(ctx context.Context, h hash.Hash) (Chunk, error)

Get retrieves the Chunk with the Hash h, returning EmptyChunk if it's not present.

func (*MemoryStorage) Has 

func (ms *MemoryStorage) Has(ctx context.Context, r hash.Hash) (bool, error)

Has returns true if the Chunk with the Hash h is present in ms.data, false if not.

func (*MemoryStorage) Len 

func (ms *MemoryStorage) Len() int

Len returns the number of Chunks in ms.data.

func (*MemoryStorage) NewView 

func (ms *MemoryStorage) NewView() ChunkStore

NewView vends a MemoryStoreView backed by this MemoryStorage. It's initialized with the currently "persisted" root.

func (*MemoryStorage) NewViewWithDefaultFormat 

func (ms *MemoryStorage) NewViewWithDefaultFormat() ChunkStore

NewViewWithVersion vends a MemoryStoreView backed by this MemoryStorage. It's initialized with the currently "persisted" root. Uses the default format.

func (*MemoryStorage) Root 

func (ms *MemoryStorage) Root(ctx context.Context) hash.Hash

Root returns the currently "persisted" root hash of this in-memory store.

func (*MemoryStorage) Update 

func (ms *MemoryStorage) Update(current, last hash.Hash, novel map[hash.Hash]Chunk) bool

Update checks the "persisted" root against last and, iff it matches, updates the root to current, adds all of novel to ms.data, and returns true. Otherwise returns false.

type MemoryStoreView 

type MemoryStoreView struct {
	// contains filtered or unexported fields
}

MemoryStoreView is an in-memory implementation of store.ChunkStore. Useful mainly for tests. The proper way to get one: storage := &MemoryStorage{} ms := storage.NewView()

func (*MemoryStoreView) Close 

func (ms *MemoryStoreView) Close() error

func (*MemoryStoreView) Commit 

func (ms *MemoryStoreView) Commit(ctx context.Context, current, last hash.Hash) (bool, error)

func (*MemoryStoreView) Get 

func (ms *MemoryStoreView) Get(ctx context.Context, h hash.Hash) (Chunk, error)

func (*MemoryStoreView) GetMany 

func (ms *MemoryStoreView) GetMany(ctx context.Context, hashes hash.HashSet, found func(context.Context, *Chunk)) error

func (*MemoryStoreView) Has 

func (ms *MemoryStoreView) Has(ctx context.Context, h hash.Hash) (bool, error)

func (*MemoryStoreView) HasMany 

func (ms *MemoryStoreView) HasMany(ctx context.Context, hashes hash.HashSet) (hash.HashSet, error)

func (*MemoryStoreView) Len 

func (ms *MemoryStoreView) Len() int

func (*MemoryStoreView) MarkAndSweepChunks 

func (ms *MemoryStoreView) MarkAndSweepChunks(ctx context.Context, last hash.Hash, keepChunks <-chan []hash.Hash, dest ChunkStore) error

func (*MemoryStoreView) Put 

func (ms *MemoryStoreView) Put(ctx context.Context, c Chunk) error

func (*MemoryStoreView) Rebase 

func (ms *MemoryStoreView) Rebase(ctx context.Context) error

func (*MemoryStoreView) Root 

func (ms *MemoryStoreView) Root(ctx context.Context) (hash.Hash, error)

func (*MemoryStoreView) Stats 

func (ms *MemoryStoreView) Stats() interface{}

func (*MemoryStoreView) StatsSummary 

func (ms *MemoryStoreView) StatsSummary() string

func (*MemoryStoreView) Version 

func (ms *MemoryStoreView) Version() string

type OutstandingAbsent 

type OutstandingAbsent chan<- bool

func (OutstandingAbsent) Fail 

func (oh OutstandingAbsent) Fail()

func (OutstandingAbsent) Satisfy 

func (oh OutstandingAbsent) Satisfy(h hash.Hash, c *Chunk)

type OutstandingAbsentMany 

type OutstandingAbsentMany struct {
	// contains filtered or unexported fields
}

func (OutstandingAbsentMany) Fail 

func (ohm OutstandingAbsentMany) Fail()

func (OutstandingAbsentMany) Satisfy 

func (ohm OutstandingAbsentMany) Satisfy(h hash.Hash, c *Chunk)

type OutstandingGet 

type OutstandingGet chan<- *Chunk

func (OutstandingGet) Fail 

func (r OutstandingGet) Fail()

func (OutstandingGet) Satisfy 

func (r OutstandingGet) Satisfy(h hash.Hash, c *Chunk)

type OutstandingGetMany 

type OutstandingGetMany struct {
	// contains filtered or unexported fields
}

func (OutstandingGetMany) Fail 

func (ogm OutstandingGetMany) Fail()

func (OutstandingGetMany) Satisfy 

func (ogm OutstandingGetMany) Satisfy(h hash.Hash, c *Chunk)

type OutstandingRequest 

type OutstandingRequest interface {
	Satisfy(h hash.Hash, c *Chunk)
	Fail()
}

type PrefixChunkStore 

type PrefixChunkStore interface {
	ChunkStore

	ResolveShortHash(ctx context.Context, short []byte) (hash.Hash, error)
}

type ReadBatch 

type ReadBatch map[hash.Hash][]OutstandingRequest

ReadBatch represents a set of queued Get/Has requests, each of which are blocking on a receive channel for a response.

func (*ReadBatch) Close 

func (rb *ReadBatch) Close() error

Close ensures that callers to Get() and Has() are failed correctly if the corresponding chunk wasn't in the response from the server (i.e. it wasn't found).

type ReadRequest 

type ReadRequest interface {
	Hashes() hash.HashSet
	Outstanding() OutstandingRequest
}

type TestStorage 

type TestStorage struct {
	MemoryStorage
}

func (*TestStorage) NewView 

func (t *TestStorage) NewView() *TestStoreView

type TestStoreFactory 

type TestStoreFactory struct {
	// contains filtered or unexported fields
}

func NewTestStoreFactory 

func NewTestStoreFactory() *TestStoreFactory

func (*TestStoreFactory) CreateStore 

func (f *TestStoreFactory) CreateStore(ns string) ChunkStore

func (*TestStoreFactory) Shutter 

func (f *TestStoreFactory) Shutter()

type TestStoreView 

type TestStoreView struct {
	ChunkStore
	// contains filtered or unexported fields
}

func (*TestStoreView) Get 

func (s *TestStoreView) Get(ctx context.Context, h hash.Hash) (Chunk, error)

func (*TestStoreView) GetMany 

func (s *TestStoreView) GetMany(ctx context.Context, hashes hash.HashSet, found func(context.Context, *Chunk)) error

func (*TestStoreView) Has 

func (s *TestStoreView) Has(ctx context.Context, h hash.Hash) (bool, error)

func (*TestStoreView) HasMany 

func (s *TestStoreView) HasMany(ctx context.Context, hashes hash.HashSet) (hash.HashSet, error)

func (*TestStoreView) Hases 

func (s *TestStoreView) Hases() int

func (*TestStoreView) MarkAndSweepChunks 

func (s *TestStoreView) MarkAndSweepChunks(ctx context.Context, last hash.Hash, keepChunks <-chan []hash.Hash, dest ChunkStore) error

func (*TestStoreView) Put 

func (s *TestStoreView) Put(ctx context.Context, c Chunk) error

func (*TestStoreView) Reads 

func (s *TestStoreView) Reads() int

func (*TestStoreView) Writes 

func (s *TestStoreView) Writes() int

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.