blobmap

package
v0.0.0-...-ba1c585 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2017 License: BSD-3-Clause Imports: 10 Imported by: 0

Documentation

Overview

Package blobmap implements a persistent map from blob identifiers to blob meta-data and vice versa. Meta-data includes locations of chunks within blobs, used for incremental blob transfer over the network, hints for which device might have blobs (called signposts), and metadata about blob ownership and usage. blobmap is implemented using a store.Store (currently, one implemented with leveldb).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BlobMap 

type BlobMap struct {
	// contains filtered or unexported fields
}

A BlobMap maps chunk checksums to Locations, and vice versa.

func New 

func New(ctx *context.T, stEngine, dir string) (bm *BlobMap, err error)

New() returns a pointer to a BlobMap, backed by storage in directory dir.

func Open 

func Open(ctx *context.T, stEngine, dir string) (bm *BlobMap, err error)

Open() returns a pointer to a BlobMap, backed by storage in directory dir, which must already exist.

func (*BlobMap) AssociateChunkWithLocation 

func (bm *BlobMap) AssociateChunkWithLocation(ctx *context.T, chunk []byte, loc Location) (err error)

AssociateChunkWithLocation() remembers that the specified chunk hash is associated with the specified Location.

func (*BlobMap) Close 

func (bm *BlobMap) Close() error

Close() closes any files or other resources associated with *bm. No other methods on bm may be called after Close().

func (*BlobMap) DeleteBlob 

func (bm *BlobMap) DeleteBlob(ctx *context.T, blob []byte) (err error)

DeleteBlob() deletes any of the chunk associations previously added with AssociateChunkWithLocation(..., chunk, ...).

func (*BlobMap) DeleteBlobMetadata 

func (bm *BlobMap) DeleteBlobMetadata(ctx *context.T, blobId wire.BlobRef) error

DeleteBlobMetadata() deletes the BlobMetadata for the specified blob.

func (*BlobMap) DeletePerSyncgroup 

func (bm *BlobMap) DeletePerSyncgroup(ctx *context.T, sgId interfaces.GroupId) error

DeletePerSyncgroup() deletes the PerSyncgroup for the specified syncgroup.

func (*BlobMap) DeleteSignpost 

func (bm *BlobMap) DeleteSignpost(ctx *context.T, blobId wire.BlobRef) error

DeleteSignpost() deletes the Signpost for the specified blob.

func (*BlobMap) GetBlobMetadata 

func (bm *BlobMap) GetBlobMetadata(ctx *context.T, blobId wire.BlobRef, bmd *localblobstore.BlobMetadata) (err error)

GetBlobMetadata() yields in *bmd the BlobMetadata associated with a blob. If there is an error, *bmd is set to a canonical empty BlobMetadata. On return, it is guaranteed that any maps in *bmd are non-nil.

func (*BlobMap) GetPerSyncgroup 

func (bm *BlobMap) GetPerSyncgroup(ctx *context.T, sgId interfaces.GroupId, psg *localblobstore.PerSyncgroup) (err error)

GetPerSyncgroup() yields in *psg the PerSyncgroup associated with a syncgroup. If there is an error, *psg is set to a canonical empty PerSyncgroup. On return, it is guaranteed that any maps in *psg are non-nil.

func (*BlobMap) GetSignpost 

func (bm *BlobMap) GetSignpost(ctx *context.T, blobId wire.BlobRef, sp *interfaces.Signpost) (err error)

GetSignpost() yields in *sp the Signpost associated with a blob. If there is an error, *sp is set to a canonical empty Signpost. On return, it is guaranteed that any maps in *sp are non-nil.

func (*BlobMap) LookupChunk 

func (bm *BlobMap) LookupChunk(ctx *context.T, chunkHash []byte) (loc Location, err error)

LookupChunk() returns a Location for the specified chunk. Only one Location is returned, even if several are available in the database. If the client finds that the Location is not available, perhaps because its blob has been deleted, the client should remove the blob from the BlobMap using DeleteBlob(loc.Blob), and try again. (The client may also wish to arrange at some point to call GC() on the blob store.)

func (*BlobMap) NewBlobMetadataStream 

func (bm *BlobMap) NewBlobMetadataStream(ctx *context.T) localblobstore.BlobMetadataStream

NewBlobMetadataStream() returns, as a localblobstore.BlobMetadataStream interface, a pointer to a new blobmap.BlobMetadataStream that allows the client to enumerate the BlobMetadatas associated with blob IDs, in order.

func (*BlobMap) NewBlobStream 

func (bm *BlobMap) NewBlobStream(ctx *context.T) *BlobStream

NewBlobStream() returns a pointer to a new BlobStream that allows the client to enumerate the blobs BlobMap, in lexicographic order.

func (*BlobMap) NewChunkStream 

func (bm *BlobMap) NewChunkStream(ctx *context.T, blob []byte) *ChunkStream

NewChunkStream() returns a pointer to a new ChunkStream that allows the client to enumerate the chunk hashes in a blob, in order.

func (*BlobMap) NewPerSyncgroupStream 

func (bm *BlobMap) NewPerSyncgroupStream(ctx *context.T) localblobstore.PerSyncgroupStream

NewPerSyncgroupStream() returns, as a localblobstore.BlobMetadataStream interface, a pointer to a new blobmap.PerSyncgroupStream that allows the client to enumerate the PerSyncgroups asscoaiedted with blob IDs, in order.

func (*BlobMap) NewSignpostStream 

func (bm *BlobMap) NewSignpostStream(ctx *context.T) localblobstore.SignpostStream

NewSignpostStream() returns, as a localblobstore.SignpostStream interface, a pointer to a new blobmap.SignpostStream that allows the client to enumerate the Signposts associated with blob IDs, in order.

func (*BlobMap) SetBlobMetadata 

func (bm *BlobMap) SetBlobMetadata(ctx *context.T, blobId wire.BlobRef, bmd *localblobstore.BlobMetadata) (err error)

SetBlobMetadata() sets the BlobMetadata associated with a blob to *bmd.

func (*BlobMap) SetPerSyncgroup 

func (bm *BlobMap) SetPerSyncgroup(ctx *context.T, sgId interfaces.GroupId, psg *localblobstore.PerSyncgroup) (err error)

SetPerSyncgroup() sets the PerSyncgroup associated with a syncgroup to *psg.

func (*BlobMap) SetSignpost 

func (bm *BlobMap) SetSignpost(ctx *context.T, blobId wire.BlobRef, sp *interfaces.Signpost) (err error)

SetSignpost() sets the Signpost associated with a blob to *sp.

type BlobMetadataStream 

type BlobMetadataStream struct {
	// contains filtered or unexported fields
}

A BlobMetadataStream allows the client to iterate over the BlobMetadatas associated with blob IDs. 

bms := bm.NewBlobMetadataStream(ctx, blob)
for bms.Advance() {
        blob := bms.BlobId()
        metadata := bms.BlobMetadata()
        ...process blob, metadata...
}
if bms.Err() != nil {
        ...there was an error...
}

func (*BlobMetadataStream) Advance 

func (bms *BlobMetadataStream) Advance() (ok bool)

Advance() stages an element so the client can retrieve the blob ID hash with BlobId(), or its BlobMetadata with BlobMetadata(). Advance() returns true iff there is an element to retrieve. The client must call Advance() before calling BlobId() or BlobMetadata() The client must call Cancel if it does not iterate through all elements (i.e. until Advance() returns false). Advance() may block if an element is not immediately available.

func (*BlobMetadataStream) BlobId 

func (bms *BlobMetadataStream) BlobId() (result wire.BlobRef)

BlobId() returns the blob ID of the blob Id and BlobMetadata staged by Advance(). BlobId() may panic if Advance() returned false or was not called at all. BlobId() does not block.

func (*BlobMetadataStream) BlobMetadata 

func (bms *BlobMetadataStream) BlobMetadata() localblobstore.BlobMetadata

BlobMetadata() returns the BlobMetadata associated with the blob ID staged by Advance(). BlobMetadata() may panic if Advance() returned false or was not called at all. BlobMetadata() does not block.

func (*BlobMetadataStream) Cancel 

func (bms *BlobMetadataStream) Cancel()

Cancel() notifies the stream provider that it can stop producing elements. The client must call Cancel() if it does not iterate through all elements (i.e. until Advance() returns false). Cancel() is idempotent and can be called concurrently with a goroutine that is iterating via Advance() and BlobId(). Cancel() causes Advance() to subsequently return false. Cancel() does not block.

func (*BlobMetadataStream) Err 

func (bms *BlobMetadataStream) Err() error

Err() returns a non-nil error iff the stream encountered any errors. Err() does not block.

type BlobStream 

type BlobStream struct {
	// contains filtered or unexported fields
}

A BlobStream allows the client to iterate over the blobs in BlobMap: 

bs := bm.NewBlobStream(ctx)
for bs.Advance() {
	blobID := bs.Value()
	...process blobID...
}
if bs.Err() != nil {
	...there was an error...
}

func (*BlobStream) Advance 

func (bs *BlobStream) Advance() (ok bool)

Advance() stages an element so the client can retrieve the next blob ID with Value(). Advance() returns true iff there is an element to retrieve. The client must call Advance() before calling Value(). The client must call Cancel if it does not iterate through all elements (i.e. until Advance() returns false). Advance() may block if an element is not immediately available.

func (*BlobStream) Cancel 

func (bs *BlobStream) Cancel()

Cancel() notifies the stream provider that it can stop producing elements. The client must call Cancel() if it does not iterate through all elements (i.e. until Advance() returns false). Cancel() is idempotent and can be called concurrently with a goroutine that is iterating via Advance() and Value(). Cancel() causes Advance() to subsequently return false. Cancel() does not block.

func (*BlobStream) Err 

func (bs *BlobStream) Err() error

Err() returns a non-nil error iff the stream encountered any errors. Err() does not block.

func (*BlobStream) Value 

func (bs *BlobStream) Value(buf []byte) (result []byte)

Value() returns the blob ID staged by Advance(). The returned slice may be a sub-slice of buf if buf is large enough to hold the entire value. Otherwise, a newly allocated slice will be returned. It is valid to pass a nil buf. Value() may panic if Advance() returned false or was not called at all. Value() does not block.

type ChunkStream 

type ChunkStream struct {
	// contains filtered or unexported fields
}

A ChunkStream allows the client to iterate over the chunks in a blob: 

cs := bm.NewChunkStream(ctx, blob)
for cs.Advance() {
	chunkHash := cs.Value()
	...process chunkHash...
}
if cs.Err() != nil {
	...there was an error...
}

func (*ChunkStream) Advance 

func (cs *ChunkStream) Advance() (ok bool)

Advance() stages an element so the client can retrieve the chunk hash with Value(), or its Location with Location(). Advance() returns true iff there is an element to retrieve. The client must call Advance() before calling Value() or Location() The client must call Cancel if it does not iterate through all elements (i.e. until Advance() returns false). Advance() may block if an element is not immediately available.

func (*ChunkStream) Cancel 

func (cs *ChunkStream) Cancel()

Cancel() notifies the stream provider that it can stop producing elements. The client must call Cancel() if it does not iterate through all elements (i.e. until Advance() returns false). Cancel() is idempotent and can be called concurrently with a goroutine that is iterating via Advance() and Value(). Cancel() causes Advance() to subsequently return false. Cancel() does not block.

func (*ChunkStream) Err 

func (cs *ChunkStream) Err() error

Err() returns a non-nil error iff the stream encountered any errors. Err() does not block.

func (*ChunkStream) Location 

func (cs *ChunkStream) Location() Location

Location() returns the Location associated with the chunk staged by Advance(). Location() may panic if Advance() returned false or was not called at all. Location() does not block.

func (*ChunkStream) Value 

func (cs *ChunkStream) Value(buf []byte) (result []byte)

Value() returns the content hash of the chunk staged by Advance(). The returned slice may be a sub-slice of buf if buf is large enough to hold the entire value. Otherwise, a newly allocated slice will be returned. It is valid to pass a nil buf. Value() may panic if Advance() returned false or was not called at all. Value() does not block.

type Location 

type Location struct {
	BlobID []byte // ID of blob
	Offset int64  // byte offset of chunk within blob
	Size   int64  // size of chunk
}

A Location describes chunk's location within a blob.

type PerSyncgroupStream 

type PerSyncgroupStream struct {
	// contains filtered or unexported fields
}

A PerSyncgroupStream allows the client to iterate over the PerSyncgroups associated with syncgroups. 

psgs := bm.NewPerSyncgroupStream(ctx, sgId)
for psgs.Advance() {
        syncgroup := psgs.SyncgroupName()
        perSyncgroup := psgs.PerSyncgroup()
        ...process blob, perSyncgroup...
}
if psgs.Err() != nil {
        ...there was an error...
}

func (*PerSyncgroupStream) Advance 

func (psgs *PerSyncgroupStream) Advance() (ok bool)

Advance() stages an element so the client can retrieve the blob ID hash with BlobID(), or its PerSyncgroup with PerSyncgroup(). Advance() returns true iff there is an element to retrieve. The client must call Advance() before calling BlobID() or PerSyncgroup() The client must call Cancel if it does not iterate through all elements (i.e. until Advance() returns false). Advance() may block if an element is not immediately available.

func (*PerSyncgroupStream) Cancel 

func (psgs *PerSyncgroupStream) Cancel()

Cancel() notifies the stream provider that it can stop producing elements. The client must call Cancel() if it does not iterate through all elements (i.e. until Advance() returns false). Cancel() is idempotent and can be called concurrently with a goroutine that is iterating via Advance() and BlobID(). Cancel() causes Advance() to subsequently return false. Cancel() does not block.

func (*PerSyncgroupStream) Err 

func (psgs *PerSyncgroupStream) Err() error

Err() returns a non-nil error iff the stream encountered any errors. Err() does not block.

func (*PerSyncgroupStream) PerSyncgroup 

func (psgs *PerSyncgroupStream) PerSyncgroup() localblobstore.PerSyncgroup

PerSyncgroup() returns the PerSyncgroup associated with the blob ID staged by Advance(). PerSyncgroup() may panic if Advance() returned false or was not called at all. PerSyncgroup() does not block.

func (*PerSyncgroupStream) SyncgroupId 

func (psgs *PerSyncgroupStream) SyncgroupId() interfaces.GroupId

SyncgroupID() returns the syncgroup Id staged by Advance(). SyncgroupID() may panic if Advance() returned false or was not called at all. SyncgroupID() does not block.

type SignpostStream 

type SignpostStream struct {
	// contains filtered or unexported fields
}

A SignpostStream allows the client to iterate over the Signposts associated with blob IDs. 

ss := bm.NewSignpostStream(ctx, blob)
for ss.Advance() {
        blob := ss.BlobId()
        signpost := ss.Signpost()
        ...process blob, signpost...
}
if ss.Err() != nil {
        ...there was an error...
}

func (*SignpostStream) Advance 

func (ss *SignpostStream) Advance() (ok bool)

Advance() stages an element so the client can retrieve the blob ID hash with BlobId(), or its Signpost with Signpost(). Advance() returns true iff there is an element to retrieve. The client must call Advance() before calling BlobId() or Signpost() The client must call Cancel if it does not iterate through all elements (i.e. until Advance() returns false). Advance() may block if an element is not immediately available.

func (*SignpostStream) BlobId 

func (ss *SignpostStream) BlobId() (result wire.BlobRef)

BlobId() returns the blob Id of the blob Id and Signpost staged by Advance(). BlobId() may panic if Advance() returned false or was not called at all. BlobId() does not block.

func (*SignpostStream) Cancel 

func (ss *SignpostStream) Cancel()

Cancel() notifies the stream provider that it can stop producing elements. The client must call Cancel() if it does not iterate through all elements (i.e. until Advance() returns false). Cancel() is idempotent and can be called concurrently with a goroutine that is iterating via Advance() and BlobId(). Cancel() causes Advance() to subsequently return false. Cancel() does not block.

func (*SignpostStream) Err 

func (ss *SignpostStream) Err() error

Err() returns a non-nil error iff the stream encountered any errors. Err() does not block.

func (*SignpostStream) Signpost 

func (ss *SignpostStream) Signpost() interfaces.Signpost

Signpost() returns the Signpost associated with the blob ID staged by Advance(). Signpost() may panic if Advance() returned false or was not called at all. Signpost() does not block.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.