Documentation
¶
Overview ¶
Package storage implements simple storage of immutable, unstructured binary large objects (BLOBs).
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrBlockNotFound = errors.New("block not found")
ErrBlockNotFound is returned when a block cannot be found in storage.
Functions ¶
func AddSupportedStorage ¶
func AddSupportedStorage( urlScheme string, defaultConfigFunc func() interface{}, createStorageFunc func(context.Context, interface{}) (Storage, error))
AddSupportedStorage registers factory function to create storage with a given type name.
func WithUploadProgressCallback ¶
func WithUploadProgressCallback(ctx context.Context, callback ProgressFunc) context.Context
WithUploadProgressCallback returns a context that passes callback function to be used storage upload progress.
Types ¶
type BlockMetadata ¶
BlockMetadata represents metadata about a single block in a storage.
func ListAllBlocks ¶
ListAllBlocks returns BlockMetadata for all blocks in a given storage that have the provided name prefix.
func ListAllBlocksConsistent ¶
func ListAllBlocksConsistent(ctx context.Context, st Storage, prefix string, maxAttempts int) ([]BlockMetadata, error)
ListAllBlocksConsistent lists all blocks with given name prefix in the provided storage until the results are consistent. The results are consistent if the list result fetched twice is identical. This guarantees that while the first scan was in progress, no new block was added or removed. maxAttempts specifies maximum number of list attempts (must be >= 2)
type ConnectionInfo ¶
type ConnectionInfo struct {
Type string
Config interface{}
}
ConnectionInfo represents JSON-serializable configuration of a blob storage.
func (ConnectionInfo) MarshalJSON ¶
func (c ConnectionInfo) MarshalJSON() ([]byte, error)
MarshalJSON returns JSON-encoded storage configuration.
func (*ConnectionInfo) UnmarshalJSON ¶
func (c *ConnectionInfo) UnmarshalJSON(b []byte) error
UnmarshalJSON parses the JSON-encoded data into ConnectionInfo.
type ProgressFunc ¶
ProgressFunc is used to report progress of a long-running storage operation.
func ProgressCallback ¶
func ProgressCallback(ctx context.Context) ProgressFunc
ProgressCallback gets the progress callback function from the context.
type Storage ¶
type Storage interface {
// PutBlock uploads the block with given data to the repository or replaces existing block with the provided
// id with given contents.
PutBlock(ctx context.Context, id string, data []byte) error
// DeleteBlock removes the block from storage. Future GetBlock() operations will fail with ErrBlockNotFound.
DeleteBlock(ctx context.Context, id string) error
// GetBlock returns full or partial contents of a block with given ID.
// If length>0, the the function retrieves a range of bytes [offset,offset+length)
// If length<0, the entire block must be fetched.
GetBlock(ctx context.Context, id string, offset, length int64) ([]byte, error)
// ListBlocks returns a channel of BlockMetadata that describes storage blocks with existing name prefixes.
// Iteration continues until all blocks have been listed or until client code invokes the returned cancellation function.
ListBlocks(ctx context.Context, prefix string, cb func(bm BlockMetadata) error) error
// ConnectionInfo returns JSON-serializable data structure containing information required to
// connect to storage.
ConnectionInfo() ConnectionInfo
// Close releases all resources associated with storage.
Close(ctx context.Context) error
}
Storage encapsulates API for connecting to blob storage.
The underlying storage system must provide:
* high durability, availability and bit-rot protection * read-after-write - block written using PutBlock() must be immediately readable using GetBlock() and ListBlocks() * atomicity - it mustn't be possible to observe partial results of PutBlock() via either GetBlock() or ListBlocks() * timestamps that don't go back in time (small clock skew up to minutes is allowed) * reasonably low latency for retrievals
The required semantics are provided by existing commercial cloud storage products (Google Cloud, AWS, Azure).
func NewStorage ¶
func NewStorage(ctx context.Context, cfg ConnectionInfo) (Storage, error)
NewStorage creates new storage based on ConnectionInfo. The storage type must be previously registered using AddSupportedStorage.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package filesystem implements filesystem-based Storage.
|
Package filesystem implements filesystem-based Storage. |
|
Package gcs implements Storage based on Google Cloud Storage bucket.
|
Package gcs implements Storage based on Google Cloud Storage bucket. |
|
Package logging implements wrapper around Storage that logs all activity.
|
Package logging implements wrapper around Storage that logs all activity. |
|
Package providers registers all storage providers that are included as part of Kopia.
|
Package providers registers all storage providers that are included as part of Kopia. |
|
Package s3 implements Storage based on an S3 bucket.
|
Package s3 implements Storage based on an S3 bucket. |
|
Package webdav implements WebDAV-based Storage.
|
Package webdav implements WebDAV-based Storage. |