chunks

package
v0.0.0-...-3afbc18 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 21, 2016 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package chunks provides facilities for representing, storing, and fetching content-addressed chunks of Noms data.

Index

Constants

This section is empty.

Variables

View Source 
var EmptyChunk = NewChunk([]byte{})

Functions

func Deserialize 

func Deserialize(reader io.Reader, cs ChunkSink, rateLimit chan struct{})

Deserialize reads off of |reader| until EOF, sending chunks to |cs|. If |rateLimit| is non-nil, concurrency will be limited to the available capacity of the channel.

func DeserializeToChan 

func DeserializeToChan(reader io.Reader, chunkChan chan<- interface{})

DeserializeToChan reads off of |reader| until EOF, sending chunks to chunkChan in the order they are read. Objects sent over chunkChan are *Chunk. The type is `chan<- interface{}` so that this is compatible with orderedparallel.New().

func RegisterLevelDBFlags 

func RegisterLevelDBFlags(flags *flag.FlagSet)

func Serialize 

func Serialize(chunk Chunk, writer io.Writer)

Serialize a single Chunk to writer.

Types

type BackpressureError 

type BackpressureError hash.HashSlice

BackpressureError is a slice of hash.Hash that indicates some chunks could not be Put(). Caller is free to try to Put them again later.

func (BackpressureError) AsHashes 

func (b BackpressureError) AsHashes() hash.HashSlice

func (BackpressureError) Error 

func (b BackpressureError) Error() string

type Chunk 

type Chunk struct {
	// contains filtered or unexported fields
}

Chunk is a unit of stored data in noms

func NewChunk 

func NewChunk(data []byte) Chunk

NewChunk creates a new Chunk backed by data. This means that the returned Chunk has ownership of this slice of memory.

func NewChunkWithHash 

func NewChunkWithHash(r hash.Hash, data []byte) Chunk

NewChunkWithHash creates a new chunk with a known hash. The hash is not re-calculated or verified. This should obviously only be used in cases where the caller already knows the specified hash is correct.

func (Chunk) Data 

func (c Chunk) Data() []byte

func (Chunk) Hash 

func (c Chunk) Hash() hash.Hash

func (Chunk) IsEmpty 

func (c Chunk) IsEmpty() bool

type ChunkSink 

type ChunkSink interface {
	// Put writes c into the ChunkSink, blocking until the operation is complete.
	Put(c Chunk)

	// PutMany tries to write chunks into the sink. It will block as it handles as many as possible, then return a BackpressureError containing the rest (if any).
	PutMany(chunks []Chunk) BackpressureError

	io.Closer
}

ChunkSink is a place to put chunks.

type ChunkSource 

type ChunkSource interface {
	// Get the Chunk for the value of the hash in the store. If the hash is absent from the store nil is returned.
	Get(h hash.Hash) Chunk

	// Returns true iff the value at the address |h| is contained in the source
	Has(h hash.Hash) bool

	// Returns the NomsVersion with which this ChunkSource is compatible.
	Version() string
}

ChunkSource is a place to get chunks from.

type ChunkStore 

type ChunkStore interface {
	ChunkSource
	ChunkSink
	RootTracker
}

ChunkStore is the core storage abstraction in noms. We can put data anyplace we have a ChunkStore implementation for.

type ChunkWriter 

type ChunkWriter struct {
	// contains filtered or unexported fields
}

ChunkWriter wraps an io.WriteCloser, additionally providing the ability to grab the resulting Chunk for all data written through the interface. Calling Chunk() or Close() on an instance disallows further writing.

func NewChunkWriter 

func NewChunkWriter() *ChunkWriter

func (*ChunkWriter) Chunk 

func (w *ChunkWriter) Chunk() Chunk

Chunk() closes the writer and returns the resulting Chunk.

func (*ChunkWriter) Close 

func (w *ChunkWriter) Close() error

Close() closes computes the hash and Puts it into the ChunkSink Note: The Write() method never returns an error. Instead, like other noms interfaces, errors are reported via panic.

func (*ChunkWriter) Write 

func (w *ChunkWriter) Write(data []byte) (int, error)

type DynamoStore 

type DynamoStore struct {
	// contains filtered or unexported fields
}

DynamoStore implements ChunkStore by storing data to DynamoDB and, if needed, S3. It assumes the existence of a DynamoDB table whose primary partition key is in Binary format and named `ref`.

func NewDynamoStore 

func NewDynamoStore(table, namespace string, config *aws.Config, showStats bool) *DynamoStore

NewDynamoStore returns a new DynamoStore instance pointed at a DynamoDB table in the given region. All keys used to access items are prefixed with the given namespace. Uses credential from the AWS config parameter.

func (*DynamoStore) Close 

func (s *DynamoStore) Close() error

func (*DynamoStore) Get 

func (s *DynamoStore) Get(h hash.Hash) Chunk

func (*DynamoStore) Has 

func (s *DynamoStore) Has(h hash.Hash) bool

func (*DynamoStore) Put 

func (s *DynamoStore) Put(c Chunk)

func (*DynamoStore) PutMany 

func (s *DynamoStore) PutMany(chunks []Chunk) (e BackpressureError)

func (*DynamoStore) Root 

func (s *DynamoStore) Root() hash.Hash

func (*DynamoStore) UpdateRoot 

func (s *DynamoStore) UpdateRoot(current, last hash.Hash) bool

func (*DynamoStore) Version 

func (s *DynamoStore) Version() string

type DynamoStoreFlags 

type DynamoStoreFlags struct {
	// contains filtered or unexported fields
}

func DynamoFlags 

func DynamoFlags(prefix string) DynamoStoreFlags

func (DynamoStoreFlags) CreateFactory 

func (f DynamoStoreFlags) CreateFactory() (factree Factory)

func (DynamoStoreFlags) CreateStore 

func (f DynamoStoreFlags) CreateStore(ns string) ChunkStore

func (DynamoStoreFlags) Shutter 

func (f DynamoStoreFlags) Shutter()

type Factory 

type Factory interface {
	CreateStore(ns string) ChunkStore

	// Shutter shuts down the factory. Subsequent calls to CreateStore() will fail.
	Shutter()
}

Factory allows the creation of namespaced ChunkStore instances. The details of how namespaces are separated is left up to the particular implementation of Factory and ChunkStore.

func NewLevelDBStoreFactory 

func NewLevelDBStoreFactory(dir string, maxHandles int, dumpStats bool) Factory

func NewLevelDBStoreFactoryUseFlags 

func NewLevelDBStoreFactoryUseFlags(dir string) Factory

func NewMemoryStoreFactory 

func NewMemoryStoreFactory() Factory

type GetRequest 

type GetRequest struct {
	// contains filtered or unexported fields
}

func NewGetRequest 

func NewGetRequest(r hash.Hash, ch chan Chunk) GetRequest

func (GetRequest) Hash 

func (g GetRequest) Hash() hash.Hash

func (GetRequest) Outstanding 

func (g GetRequest) Outstanding() OutstandingRequest

type HasRequest 

type HasRequest struct {
	// contains filtered or unexported fields
}

func NewHasRequest 

func NewHasRequest(r hash.Hash, ch chan bool) HasRequest

func (HasRequest) Hash 

func (h HasRequest) Hash() hash.Hash

func (HasRequest) Outstanding 

func (h HasRequest) Outstanding() OutstandingRequest

type LevelDBStore 

type LevelDBStore struct {
	// contains filtered or unexported fields
}

func NewLevelDBStore 

func NewLevelDBStore(dir, ns string, maxFileHandles int, dumpStats bool) *LevelDBStore

func NewLevelDBStoreUseFlags 

func NewLevelDBStoreUseFlags(dir, ns string) *LevelDBStore

func (*LevelDBStore) Close 

func (l *LevelDBStore) Close() error

func (*LevelDBStore) Get 

func (l *LevelDBStore) Get(ref hash.Hash) Chunk

func (*LevelDBStore) Has 

func (l *LevelDBStore) Has(ref hash.Hash) bool

func (*LevelDBStore) Put 

func (l *LevelDBStore) Put(c Chunk)

func (*LevelDBStore) PutMany 

func (l *LevelDBStore) PutMany(chunks []Chunk) (e BackpressureError)

func (*LevelDBStore) Root 

func (l *LevelDBStore) Root() hash.Hash

func (*LevelDBStore) UpdateRoot 

func (l *LevelDBStore) UpdateRoot(current, last hash.Hash) bool

func (*LevelDBStore) Version 

func (l *LevelDBStore) Version() string

type LevelDBStoreFactory 

type LevelDBStoreFactory struct {
	// contains filtered or unexported fields
}

func (*LevelDBStoreFactory) CreateStore 

func (f *LevelDBStoreFactory) CreateStore(ns string) ChunkStore

func (*LevelDBStoreFactory) Shutter 

func (f *LevelDBStoreFactory) Shutter()

type LevelDBStoreFlags 

type LevelDBStoreFlags struct {
	// contains filtered or unexported fields
}

type MemoryStore 

type MemoryStore struct {
	// contains filtered or unexported fields
}

An in-memory implementation of store.ChunkStore. Useful mainly for tests.

func NewMemoryStore 

func NewMemoryStore() *MemoryStore

func (*MemoryStore) Close 

func (ms *MemoryStore) Close() error

func (*MemoryStore) Get 

func (ms *MemoryStore) Get(h hash.Hash) Chunk

func (*MemoryStore) Has 

func (ms *MemoryStore) Has(r hash.Hash) bool

func (*MemoryStore) Len 

func (ms *MemoryStore) Len() int

func (*MemoryStore) Put 

func (ms *MemoryStore) Put(c Chunk)

func (*MemoryStore) PutMany 

func (ms *MemoryStore) PutMany(chunks []Chunk) (e BackpressureError)

func (*MemoryStore) Root 

func (ms *MemoryStore) Root() hash.Hash

func (*MemoryStore) UpdateRoot 

func (ms *MemoryStore) UpdateRoot(current, last hash.Hash) bool

func (*MemoryStore) Version 

func (ms *MemoryStore) Version() string

type MemoryStoreFactory 

type MemoryStoreFactory struct {
	// contains filtered or unexported fields
}

func (*MemoryStoreFactory) CreateStore 

func (f *MemoryStoreFactory) CreateStore(ns string) ChunkStore

func (*MemoryStoreFactory) Shutter 

func (f *MemoryStoreFactory) Shutter()

type OutstandingGet 

type OutstandingGet chan Chunk

func (OutstandingGet) Fail 

func (r OutstandingGet) Fail()

func (OutstandingGet) Satisfy 

func (r OutstandingGet) Satisfy(c Chunk)

type OutstandingHas 

type OutstandingHas chan bool

func (OutstandingHas) Fail 

func (h OutstandingHas) Fail()

func (OutstandingHas) Satisfy 

func (h OutstandingHas) Satisfy(c Chunk)

type OutstandingRequest 

type OutstandingRequest interface {
	Satisfy(c Chunk)
	Fail()
}

type ReadBatch 

type ReadBatch map[hash.Hash][]OutstandingRequest

ReadBatch represents a set of queued Get/Has requests, each of which are blocking on a receive channel for a response.

func (*ReadBatch) Close 

func (rb *ReadBatch) Close() error

Close ensures that callers to Get() and Has() are failed correctly if the corresponding chunk wasn't in the response from the server (i.e. it wasn't found).

type ReadRequest 

type ReadRequest interface {
	Hash() hash.Hash
	Outstanding() OutstandingRequest
}

type ReadThroughStore 

type ReadThroughStore struct {
	io.Closer
	// contains filtered or unexported fields
}

ReadThroughStore is a store that consists of two other stores. A caching and a backing store. All reads check the caching store first and if the h is present there the caching store is used. If not present the backing store is used and the value gets cached in the caching store. All writes go directly to the backing store.

func NewReadThroughStore 

func NewReadThroughStore(cachingStore ChunkStore, backingStore ChunkStore) ReadThroughStore

func (ReadThroughStore) Get 

func (rts ReadThroughStore) Get(h hash.Hash) Chunk

func (ReadThroughStore) Has 

func (rts ReadThroughStore) Has(h hash.Hash) bool

func (ReadThroughStore) Put 

func (rts ReadThroughStore) Put(c Chunk)

func (ReadThroughStore) PutMany 

func (rts ReadThroughStore) PutMany(chunks []Chunk) BackpressureError

func (ReadThroughStore) Root 

func (rts ReadThroughStore) Root() hash.Hash

func (ReadThroughStore) UpdateRoot 

func (rts ReadThroughStore) UpdateRoot(current, last hash.Hash) bool

func (ReadThroughStore) Version 

func (rts ReadThroughStore) Version() string

type RootTracker 

type RootTracker interface {
	Root() hash.Hash
	UpdateRoot(current, last hash.Hash) bool
}

RootTracker allows querying and management of the root of an entire tree of references. The "root" is the single mutable variable in a ChunkStore. It can store any hash, but it is typically used by higher layers (such as Database) to store a hash to a value that represents the current state and entire history of a database.

type TestStore 

type TestStore struct {
	MemoryStore
	Reads  int
	Hases  int
	Writes int
}

func NewTestStore 

func NewTestStore() *TestStore

func (*TestStore) Get 

func (s *TestStore) Get(h hash.Hash) Chunk

func (*TestStore) Has 

func (s *TestStore) Has(h hash.Hash) bool

func (*TestStore) Put 

func (s *TestStore) Put(c Chunk)

func (*TestStore) PutMany 

func (s *TestStore) PutMany(chunks []Chunk) (e BackpressureError)

func (*TestStore) Root 

func (ms *TestStore) Root() hash.Hash

func (*TestStore) UpdateRoot 

func (ms *TestStore) UpdateRoot(current, last hash.Hash) bool

type TestStoreFactory 

type TestStoreFactory struct {
	Stores map[string]*TestStore
}

TestStoreFactory is public, and exposes Stores to ensure that test code can directly query instances vended by this factory.

func NewTestStoreFactory 

func NewTestStoreFactory() *TestStoreFactory

func (*TestStoreFactory) CreateStore 

func (f *TestStoreFactory) CreateStore(ns string) ChunkStore

func (*TestStoreFactory) Shutter 

func (f *TestStoreFactory) Shutter()

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.