chunks

package
v0.0.0-...-22f70f5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 27, 2017 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package chunks provides facilities for representing, storing, and fetching content-addressed chunks of Noms data.

Index

Constants

This section is empty.

Variables

View Source 
var EmptyChunk = NewChunk([]byte{})

Functions

func Deserialize 

func Deserialize(reader io.Reader, chunkChan chan<- *Chunk) (err error)

Deserialize reads off of |reader| until EOF, sending chunks to chunkChan in the order they are read. Objects sent over chunkChan are *Chunk.

func Serialize 

func Serialize(chunk Chunk, writer io.Writer)

Serialize a single Chunk to writer.

Types

type Chunk 

type Chunk struct {
	// contains filtered or unexported fields
}

Chunk is a unit of stored data in noms

func NewChunk 

func NewChunk(data []byte) Chunk

NewChunk creates a new Chunk backed by data. This means that the returned Chunk has ownership of this slice of memory.

func NewChunkWithHash 

func NewChunkWithHash(r hash.Hash, data []byte) Chunk

NewChunkWithHash creates a new chunk with a known hash. The hash is not re-calculated or verified. This should obviously only be used in cases where the caller already knows the specified hash is correct.

func (Chunk) Data 

func (c Chunk) Data() []byte

func (Chunk) Hash 

func (c Chunk) Hash() hash.Hash

func (Chunk) IsEmpty 

func (c Chunk) IsEmpty() bool

type ChunkSink 

type ChunkSink interface {
	// Put writes c into the ChunkSink, blocking until the operation is complete.
	Put(c Chunk)

	// PutMany writes chunks into the sink, blocking until the operation is complete.
	PutMany(chunks []Chunk)

	// On return, any previously Put chunks should be durable
	Flush()

	io.Closer
}

ChunkSink is a place to put chunks.

type ChunkSource 

type ChunkSource interface {
	// Get the Chunk for the value of the hash in the store. If the hash is
	// absent from the store nil is returned.
	Get(h hash.Hash) Chunk

	// GetMany gets the Chunks with |hashes| from the store. On return,
	// |foundChunks| will have been fully sent all chunks which have been
	// found. Any non-present chunks will silently be ignored.
	GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

	// Returns true iff the value at the address |h| is contained in the source
	Has(h hash.Hash) bool

	// Returns the NomsVersion with which this ChunkSource is compatible.
	Version() string
}

ChunkSource is a place to get chunks from.

type ChunkStore 

type ChunkStore interface {
	ChunkSource
	ChunkSink
	RootTracker
}

ChunkStore is the core storage abstraction in noms. We can put data anyplace we have a ChunkStore implementation for.

type ChunkWriter 

type ChunkWriter struct {
	// contains filtered or unexported fields
}

ChunkWriter wraps an io.WriteCloser, additionally providing the ability to grab the resulting Chunk for all data written through the interface. Calling Chunk() or Close() on an instance disallows further writing.

func NewChunkWriter 

func NewChunkWriter() *ChunkWriter

func (*ChunkWriter) Chunk 

func (w *ChunkWriter) Chunk() Chunk

Chunk() closes the writer and returns the resulting Chunk.

func (*ChunkWriter) Close 

func (w *ChunkWriter) Close() error

Close() closes computes the hash and Puts it into the ChunkSink Note: The Write() method never returns an error. Instead, like other noms interfaces, errors are reported via panic.

func (*ChunkWriter) Write 

func (w *ChunkWriter) Write(data []byte) (int, error)

type DynamoStore 

type DynamoStore struct {
	// contains filtered or unexported fields
}

DynamoStore implements ChunkStore by storing data to DynamoDB and, if needed, S3. It assumes the existence of a DynamoDB table whose primary partition key is in Binary format and named `ref`.

func NewDynamoStore 

func NewDynamoStore(table, namespace string, ddb ddbsvc, showStats bool) *DynamoStore

NewDynamoStore returns a new DynamoStore instance pointed at a DynamoDB table in the given region. All keys used to access items are prefixed with the given namespace. Uses the given ddbsvc object to access DynamoDB.

func (*DynamoStore) Close 

func (s *DynamoStore) Close() error

func (*DynamoStore) Flush 

func (s *DynamoStore) Flush()

func (*DynamoStore) Get 

func (s *DynamoStore) Get(h hash.Hash) Chunk

func (*DynamoStore) GetMany 

func (s *DynamoStore) GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

func (*DynamoStore) Has 

func (s *DynamoStore) Has(h hash.Hash) bool

func (*DynamoStore) Put 

func (s *DynamoStore) Put(c Chunk)

func (*DynamoStore) PutMany 

func (s *DynamoStore) PutMany(chunks []Chunk)

func (*DynamoStore) Root 

func (s *DynamoStore) Root() hash.Hash

func (*DynamoStore) UpdateRoot 

func (s *DynamoStore) UpdateRoot(current, last hash.Hash) bool

func (*DynamoStore) Version 

func (s *DynamoStore) Version() string

type DynamoStoreFlags 

type DynamoStoreFlags struct {
	// contains filtered or unexported fields
}

func DynamoFlags 

func DynamoFlags(prefix string) DynamoStoreFlags

func (DynamoStoreFlags) CreateFactory 

func (f DynamoStoreFlags) CreateFactory() (factree Factory)

func (DynamoStoreFlags) CreateStore 

func (f DynamoStoreFlags) CreateStore(ns string) ChunkStore

func (DynamoStoreFlags) Shutter 

func (f DynamoStoreFlags) Shutter()

type Factory 

type Factory interface {
	CreateStore(ns string) ChunkStore

	// Shutter shuts down the factory. Subsequent calls to CreateStore() will fail.
	Shutter()
}

Factory allows the creation of namespaced ChunkStore instances. The details of how namespaces are separated is left up to the particular implementation of Factory and ChunkStore.

func NewMemoryStoreFactory 

func NewMemoryStoreFactory() Factory

type GetManyRequest 

type GetManyRequest struct {
	// contains filtered or unexported fields
}

func NewGetManyRequest 

func NewGetManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- *Chunk) GetManyRequest

func (GetManyRequest) Hashes 

func (g GetManyRequest) Hashes() hash.HashSet

func (GetManyRequest) Outstanding 

func (g GetManyRequest) Outstanding() OutstandingRequest

type GetRequest 

type GetRequest struct {
	// contains filtered or unexported fields
}

func NewGetRequest 

func NewGetRequest(r hash.Hash, ch chan<- *Chunk) GetRequest

func (GetRequest) Hashes 

func (g GetRequest) Hashes() hash.HashSet

func (GetRequest) Outstanding 

func (g GetRequest) Outstanding() OutstandingRequest

type HasRequest 

type HasRequest struct {
	// contains filtered or unexported fields
}

func NewHasRequest 

func NewHasRequest(r hash.Hash, ch chan<- bool) HasRequest

func (HasRequest) Hashes 

func (h HasRequest) Hashes() hash.HashSet

func (HasRequest) Outstanding 

func (h HasRequest) Outstanding() OutstandingRequest

type MemoryStore 

type MemoryStore struct {
	// contains filtered or unexported fields
}

An in-memory implementation of store.ChunkStore. Useful mainly for tests.

func NewMemoryStore 

func NewMemoryStore() *MemoryStore

func (*MemoryStore) Close 

func (ms *MemoryStore) Close() error

func (*MemoryStore) Flush 

func (ms *MemoryStore) Flush()

func (*MemoryStore) Get 

func (ms *MemoryStore) Get(h hash.Hash) Chunk

func (*MemoryStore) GetMany 

func (ms *MemoryStore) GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

func (*MemoryStore) Has 

func (ms *MemoryStore) Has(r hash.Hash) bool

func (*MemoryStore) Len 

func (ms *MemoryStore) Len() int

func (*MemoryStore) Put 

func (ms *MemoryStore) Put(c Chunk)

func (*MemoryStore) PutMany 

func (ms *MemoryStore) PutMany(chunks []Chunk)

func (*MemoryStore) Root 

func (ms *MemoryStore) Root() hash.Hash

func (*MemoryStore) UpdateRoot 

func (ms *MemoryStore) UpdateRoot(current, last hash.Hash) bool

func (*MemoryStore) Version 

func (ms *MemoryStore) Version() string

type MemoryStoreFactory 

type MemoryStoreFactory struct {
	// contains filtered or unexported fields
}

func (*MemoryStoreFactory) CreateStore 

func (f *MemoryStoreFactory) CreateStore(ns string) ChunkStore

func (*MemoryStoreFactory) Shutter 

func (f *MemoryStoreFactory) Shutter()

type OutstandingGet 

type OutstandingGet chan<- *Chunk

func (OutstandingGet) Fail 

func (r OutstandingGet) Fail()

func (OutstandingGet) Satisfy 

func (r OutstandingGet) Satisfy(c *Chunk)

type OutstandingGetMany 

type OutstandingGetMany struct {
	// contains filtered or unexported fields
}

func (OutstandingGetMany) Fail 

func (ogm OutstandingGetMany) Fail()

func (OutstandingGetMany) Satisfy 

func (ogm OutstandingGetMany) Satisfy(c *Chunk)

type OutstandingHas 

type OutstandingHas chan<- bool

func (OutstandingHas) Fail 

func (h OutstandingHas) Fail()

func (OutstandingHas) Satisfy 

func (h OutstandingHas) Satisfy(c *Chunk)

type OutstandingRequest 

type OutstandingRequest interface {
	Satisfy(c *Chunk)
	Fail()
}

type ReadBatch 

type ReadBatch map[hash.Hash][]OutstandingRequest

ReadBatch represents a set of queued Get/Has requests, each of which are blocking on a receive channel for a response.

func (*ReadBatch) Close 

func (rb *ReadBatch) Close() error

Close ensures that callers to Get() and Has() are failed correctly if the corresponding chunk wasn't in the response from the server (i.e. it wasn't found).

type ReadRequest 

type ReadRequest interface {
	Hashes() hash.HashSet
	Outstanding() OutstandingRequest
}

type RootTracker 

type RootTracker interface {
	Root() hash.Hash
	UpdateRoot(current, last hash.Hash) bool
}

RootTracker allows querying and management of the root of an entire tree of references. The "root" is the single mutable variable in a ChunkStore. It can store any hash, but it is typically used by higher layers (such as Database) to store a hash to a value that represents the current state and entire history of a database.

type TestStore 

type TestStore struct {
	MemoryStore
	Reads  int
	Hases  int
	Writes int
}

func NewTestStore 

func NewTestStore() *TestStore

func (*TestStore) Get 

func (s *TestStore) Get(h hash.Hash) Chunk

func (*TestStore) GetMany 

func (s *TestStore) GetMany(hashes hash.HashSet, foundChunks chan *Chunk)

func (*TestStore) Has 

func (s *TestStore) Has(h hash.Hash) bool

func (*TestStore) Put 

func (s *TestStore) Put(c Chunk)

func (*TestStore) PutMany 

func (s *TestStore) PutMany(chunks []Chunk)

func (*TestStore) Root 

func (ms *TestStore) Root() hash.Hash

func (*TestStore) UpdateRoot 

func (ms *TestStore) UpdateRoot(current, last hash.Hash) bool

type TestStoreFactory 

type TestStoreFactory struct {
	Stores map[string]*TestStore
}

TestStoreFactory is public, and exposes Stores to ensure that test code can directly query instances vended by this factory.

func NewTestStoreFactory 

func NewTestStoreFactory() *TestStoreFactory

func (*TestStoreFactory) CreateStore 

func (f *TestStoreFactory) CreateStore(ns string) ChunkStore

func (*TestStoreFactory) Shutter 

func (f *TestStoreFactory) Shutter()

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.