Documentation
¶
Overview ¶
Package blob implements a set of generic interfaces used to implement blob storage implementations such as S3, GCS, and local file system implementations. Package blob also contains a blob storage multiplexer that can be used to retrieve buckets from several underlying implementations.
Index ¶
- func Assertions(f reflow.File) *reflow.Assertions
- type Bucket
- type Mux
- func (m Mux) Bucket(ctx context.Context, rawurl string) (Bucket, string, error)
- func (m Mux) CanTransfer(ctx context.Context, dsturl, srcurl string) (bool, error)
- func (m Mux) Download(ctx context.Context, url, etag string, size int64, w io.WriterAt) (int64, error)
- func (m Mux) File(ctx context.Context, url string) (reflow.File, error)
- func (m Mux) Generate(ctx context.Context, key reflow.AssertionKey) (*reflow.Assertions, error)
- func (m Mux) Get(ctx context.Context, url, etag string) (io.ReadCloser, reflow.File, error)
- func (m Mux) NeedTransfer(ctx context.Context, dst, src reflow.File) (bool, error)
- func (m Mux) Put(ctx context.Context, url string, size int64, body io.Reader, ...) error
- func (m Mux) Scan(ctx context.Context, url string, withMetadata bool) (Scanner, error)
- func (m Mux) Snapshot(ctx context.Context, url string) (reflow.Fileset, error)
- func (m Mux) Transfer(ctx context.Context, dsturl, srcurl string) error
- type Scanner
- type Store
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Assertions ¶
func Assertions(f reflow.File) *reflow.Assertions
Assertions returns assertions for a blob file.
Types ¶
type Bucket ¶
type Bucket interface {
// File retrieves file metadata for the provided key.
File(ctx context.Context, key string) (reflow.File, error)
// Scan returns a scanner for the provided prefix, which is
// then used to retrieve all keys (in order) with this prefix.
// If withMetadata is true, the scanner is configured to make
// a best-effort attempt to fetch each object's metadata.
Scan(prefix string, withMetadata bool) Scanner
// Download downloads the provided key into the provided writer. The
// underlying implementation may perform concurrent downloading,
// writing to arbitrary offsets in the writer, potentially leaving
// holes on error on incomplete downloads. If the provided ETag is
// nonempty, then it is taken as a precondition for fetching.
// If the provided size is non-zero, it is used as a hint to manage
// concurrency.
Download(ctx context.Context, key, etag string, size int64, w io.WriterAt) (int64, error)
// Get returns a (streaming) reader for the contents at the provided
// key. The returned reflow.File represents the known metadata of
// the file object. If the provided ETag is nonempty, then it is taken
// as a precondition for fetching.
Get(ctx context.Context, key, etag string) (io.ReadCloser, reflow.File, error)
// Put streams the provided body to the provided key. Put overwrites
// any existing object at the same key.
// If the provided size is non-zero, it is used as a hint to manage
// concurrency.
// If a non-empty contentHash is provided, it is stored in the object's metadata.
// TODO(swami): Remove ContentHash and pass Headers/Metadata instead.
Put(ctx context.Context, key string, size int64, body io.Reader, contentHash string) error
// Snapshot returns an un-loaded Reflow fileset representing the
// contents of the provided prefix. This may then later be used to
// load and verify the contents of the returned fileset.
Snapshot(ctx context.Context, prefix string) (reflow.Fileset, error)
// Copy copies key src to key dst in this bucket.
// A non-empty contentHash will be added to dst's metadata only if not already set in src.
// TODO(swami): Remove ContentHash and pass Headers/Metadata instead.
Copy(ctx context.Context, src, dst, contentHash string) error
// CopyFrom copies from bucket src and key srcKey into this bucket.
CopyFrom(ctx context.Context, srcBucket Bucket, src, dst string) error
// Delete removes the provided keys.
Delete(ctx context.Context, keys ...string) error
// Location returns a URL indicating the location of the bucket.
// A key location can generally be derived by appending the
// key to the bucket's location.
Location() string
}
A Bucket is a single Namespace of keys from which files can be retrieved. Buckets support efficient prefix scans as well as both streaming and "direct" (concurrent) downloads.
type Mux ¶
Mux multiplexes a number of blob store implementations. Mux implements bucket operations based on blob store URLs. URLs that are passed into Mux are intepreted as:
store://bucket/key
func (Mux) Bucket ¶
Bucket parses the provided URL, looks up its implementation, and returns the store's Bucket and the prefix implied by the URL. A errors.NotSupported is returned if there is no implementation for the requested scheme.
func (Mux) CanTransfer ¶
CanTransfer returns whether contents of object in srcurl can be transferred to dsturl. If not supported, then the error corresponds to the reason why.
func (Mux) Download ¶
func (m Mux) Download(ctx context.Context, url, etag string, size int64, w io.WriterAt) (int64, error)
Download downloads the object named by the provided URL to the provided io.WriterAt. If the provided etag is nonempty, then it is checked as a precondition on downloading the object. Download may download multiple chunks concurrently.
func (Mux) Generate ¶
func (m Mux) Generate(ctx context.Context, key reflow.AssertionKey) (*reflow.Assertions, error)
Generate implements the AssertionGenerator interface for the blob namespace.
func (Mux) Get ¶
Get returns a (streaming) reader of the object named by the provided URL. If the provided etag is nonempty, then it is checked as a precondition on streaming the object.
func (Mux) NeedTransfer ¶
NeedTransfer returns whether src needs to be transferred to the location of dst. It expects both src and dst to be reference files, and it only determines that a transfer is unnecessary if the objects have the same ETag or ContentHash.
func (Mux) Put ¶
func (m Mux) Put(ctx context.Context, url string, size int64, body io.Reader, contentHash string) error
Put stores the contents of the provided io.Reader at the provided URL and attaches the given contentHash.
func (Mux) Scan ¶
Scan returns a scanner for the provided URL (which represents a prefix). If withMetadata is true, the scanner is configured to make a best-effort attempt to fetch each object's metadata.
type Scanner ¶
type Scanner interface {
// Scan forwards the scanner to the next entry, returning a boolean
// indicating whether the operation was successful. When Scan
// returns false, the caller should check Err() to distinguish
// between errors and complete scans.
Scan(ctx context.Context) bool
// Err returns the error, if any, that occurred while scanning.
Err() error
// File returns the currently scanned file metadata.
File() reflow.File
// Key returns the current key.
Key() string
}
A Scanner scans keys in a bucket. Scanners are provided by Bucket implementations. Scanning commences after the first call to Scan.
Source Files
Directories