blobstore

README

juju/blobstore

This package provides a Mongo GridFS-backed blob storage engine.

Documentation

Index

• Variables
• func NewPutResponse(requestId int64, sha384hash string) putResponse
• type ManagedResource
• type ManagedStorage
  • func NewManagedStorage(db *mgo.Database, rs ResourceStorage) ManagedStorage
• type PutRequest
• type RequestResponse
• type Resource
• type ResourceCatalog
• type ResourceStorage
  • func NewGridFS(dbName, namespace string, session *mgo.Session) ResourceStorage

Variables

var ErrRequestExpired = fmt.Errorf("request expired")

ErrRequestExpired is used to indicate that a put request has already expired when an attempt is made to supply a response.

var ErrResourceDeleted = fmt.Errorf("resource was deleted")

ErrResourceDeleted is used to indicate that a resource was deleted before the put response could be acted on.

var ErrResponseMismatch = fmt.Errorf("response checksums do not match")

ErrResponseMismatch is used to indicate that a put response did not contain the expected checksums.

var (
	// ErrUploadPending is used to indicate that the underlying resource for a catalog entry
	// is not yet fully uploaded.
	ErrUploadPending = errors.New("Resource not available because upload is not yet complete")
)

Functions

func NewPutResponse

func NewPutResponse(requestId int64, sha384hash string) putResponse

NewPutResponse creates a new putResponse for the given requestId and hashes.

Types

type ManagedResource

type ManagedResource struct {
	BucketUUID string
	User       string
	Path       string
}

ManagedResource is a catalog entry for stored data. The data may be associated with a specified bucket and/or user. The data is logically considered to be stored at the specified path.

type ManagedStorage

type ManagedStorage interface {
	// GetForBucket returns a reader for data at path, namespaced to the bucket.
	// If the data is still being uploaded and is not fully written yet,
	// an ErrUploadPending error is returned. This means the path is valid but the caller
	// should try again to retrieve the data.
	GetForBucket(bucketUUID, path string) (r io.ReadCloser, length int64, err error)

	// PutForBucket stores data from reader at path, namespaced to the bucket.
	//
	// PutForBucket is equivalent to PutForBucketAndCheckHash with an empty
	// hash string.
	PutForBucket(bucketUUID, path string, r io.Reader, length int64) error

	// PutForBucketAndCheckHash is the same as PutForBucket
	// except that it also checks that the content matches the provided
	// hash. The hash must be hex-encoded SHA-384.
	//
	// If checkHash is empty, then the hash check is elided.
	//
	// If length is < 0, then the reader will be consumed until EOF.
	PutForBucketAndCheckHash(bucketUUID, path string, r io.Reader, length int64, checkHash string) error

	// RemoveForBucket deletes data at path, namespaced to the bucket.
	RemoveForBucket(bucketUUID, path string) error

	// PutForBucketRequest requests that data, which may already exist in storage,
	// be saved at path, namespaced to the bucket. It allows callers who can
	// demonstrate proof of ownership of the data to store a reference to it without
	// having to upload it all. If no such data exists, a NotFound error is returned
	// and a call to bucket is required. If matching data is found, the caller
	// is returned a response indicating the random byte range to for which they must
	// provide a checksum to complete the process.
	PutForBucketRequest(bucketUUID, path string, hash string) (*RequestResponse, error)

	// ProofOfAccessResponse is called to respond to a Put..Request call in order to
	// prove ownership of data for which a storage reference is created.
	ProofOfAccessResponse(putResponse) error
}

ManagedStorage instances persist data for a bucket, for a user, or globally. (Only bucket storage is currently implemented).

func NewManagedStorage

func NewManagedStorage(db *mgo.Database, rs ResourceStorage) ManagedStorage

NewManagedStorage creates a new ManagedStorage using the transaction runner, storing resource entries in the specified database, and resource data in the specified resource storage.

type PutRequest

type PutRequest struct {
	// contains filtered or unexported fields
}

PutRequest is to record a request to put a file pending proof of access.

type RequestResponse

type RequestResponse struct {
	RequestId   int64
	RangeStart  int64
	RangeLength int64
}

RequestResponse is returned by a put request to inform the caller the data range over which to calculate the hashes for the response.

type Resource

type Resource struct {
	SHA384Hash string
	Path       string
	Length     int64
}

Resource is a catalog entry for stored data. It contains the path where the data is stored as well as a hash of the data which are used for de-duping.

type ResourceCatalog

type ResourceCatalog interface {
	// Get fetches a Resource with the given id.
	Get(id string) (*Resource, error)

	// Find returns the resource id for the Resource with the given hash.
	Find(hash string) (id string, err error)

	// Put ensures a Resource entry exists for the given hash,
	// returning the id and path recorded by UploadComplete.
	// If UploadComplete has not been called, path will be empty.
	//
	// If the Resource entry exists, its reference count is incremented,
	// otherwise a new entry is created with a reference count of 1.
	Put(hash string, length int64) (id, path string, err error)

	// UploadComplete records that the underlying resource described by
	// the Resource entry with id is now fully uploaded to the specified
	// storage path, and the resource is available for use. If another
	// uploader already recorded a path, then UploadComplete will return
	// an error satisfiying juju/errors.IsAlreadyExists.
	UploadComplete(id, path string) error

	// Remove decrements the reference count for a Resource with the given id, deleting it
	// if the reference count reaches zero. The path of the Resource is returned.
	// If the Resource is deleted, wasDeleted is returned as true.
	Remove(id string) (wasDeleted bool, path string, err error)
}

ResourceCatalog instances persist Resources. Resources with the same hash values are not duplicated; instead a reference count is incremented. Similarly, when a Resource is removed, the reference count is decremented. When the reference count reaches zero, the Resource is deleted.

type ResourceStorage

type ResourceStorage interface {
	// Get returns a reader for the resource located at path.
	Get(path string) (io.ReadCloser, error)

	// Put writes data from the specified reader to path and returns a checksum of the data written.
	Put(path string, r io.Reader, length int64) (checksum string, err error)

	// Remove deletes the data at the specified path.
	Remove(path string) error
}

ResourceStorage instances save and retrieve data from an underlying storage implementation.

func NewGridFS

func NewGridFS(dbName, namespace string, session *mgo.Session) ResourceStorage

NewGridFS returns a ResourceStorage instance backed by a mongo GridFS. namespace is used to segregate different sets of data.