blobstore

Documentation

Index

• Variables
• func NewHash() hash.Hash
• type Backend
  • func NewMongoBackend(db *mgo.Database, prefix string) Backend
  • func NewSwiftBackend(cred *identity.Credentials, authmode identity.AuthMode, ...) Backend
• type Part
• type PartInfo
• type ReadSeekCloser
• type Refs
  • func NewRefs(n int) *Refs
  • func (r *Refs) Add(hash string)
• type Store
  • func New(db *mgo.Database, prefix string, backend Backend) *Store
  • func (s *Store) FinishUpload(uploadId string, parts []Part) (idx *mongodoc.MultipartIndex, hash string, err error)
  • func (s *Store) GC(refs *Refs, before time.Time) (monitoring.BlobStats, error)
  • func (s *Store) NewUpload(expires time.Time) (uploadId string, err error)
  • func (s *Store) Open(hash string, index *mongodoc.MultipartIndex) (ReadSeekCloser, int64, error)
  • func (s *Store) Put(r io.Reader, hash string, size int64) error
  • func (s *Store) PutAtTime(r io.Reader, hash string, size int64, now time.Time) error
  • func (s *Store) PutPart(uploadId string, part int, r io.Reader, size, offset int64, hash string) error
  • func (s *Store) RemoveExpiredUploads() error
  • func (s *Store) RemoveUpload(uploadId string) error
  • func (s *Store) SetOwner(uploadId, owner string, expires time.Time) error
  • func (s *Store) UploadInfo(uploadId string) (UploadInfo, error)
• type UploadInfo
  • func (info *UploadInfo) Index() (*mongodoc.MultipartIndex, bool)

Variables

var ErrBadParams = errgo.New("bad parameters")

var ErrNotFound = errgo.New("blob not found")

Functions

func NewHash

func NewHash() hash.Hash

NewHash is used to calculate checksums for the blob store.

Types

type Backend

type Backend interface {
	// Get gets a reader for the object with the given name
	// and its size. The returned reader should be closed after use.
	// If the object doesn't exist, an error with an ErrNotFound
	// cause should be returned.
	// If the object is removed while reading, the read
	// error's cause should be ErrNotFound.
	Get(name string) (r ReadSeekCloser, size int64, err error)

	// Put puts an object by reading its data from the given reader.
	// The data read from the reader should have the given size and
	// hex-encoded SHA384 hash. If it is not possible to read size
	// bytes from r then an error with a cause of io.ErrUnexpectedEOF
	// will be returned.
	Put(name string, r io.Reader, size int64, hash string) error

	// Remove removes the object with the given name.
	Remove(name string) error
}

Backend represents the underlying data store used by blobstore.Store to store blob data.

func NewMongoBackend

func NewMongoBackend(db *mgo.Database, prefix string) Backend

NewMongoBackend returns a backend implementation which stores data in the given MongoDB database, using prefix as a prefix for the collections created.

func NewSwiftBackend

func NewSwiftBackend(cred *identity.Credentials, authmode identity.AuthMode, container, tmpdir string) Backend

NewSwiftBackend returns a backend which uses OpenStack's Swift for its operations with the given credentials and auth mode. It stores all the data objects in the container with the given name.

type Part

type Part struct {
	Hash string
}

Part represents one part of a multipart blob.

type PartInfo

type PartInfo struct {
	// Hash holds the SHA384 hash of the part.
	Hash string
	// Size holds the size of the part.
	Size int64
	// Complete holds whether the part has been
	// successfully uploaded.
	Complete bool
	// Offset holds the offset relative to the start of the upload.
	Offset int64
}

PartInfo holds information about one part of a multipart upload.

type ReadSeekCloser

type ReadSeekCloser interface {
	io.Reader
	io.Seeker
	io.Closer
}

type Refs

type Refs struct {
	// contains filtered or unexported fields
}

Refs holds information about the existence of a set of blob hashes.

func NewRefs

func NewRefs(n int) *Refs

NewRefs returns a new Refs instance, using n as a hint for the number of references that will be added (the final number does not need to match this).

func (*Refs) Add

func (r *Refs) Add(hash string)

Add records that the given hash is referenced. It ignores the hash if it's invalid.

type Store

type Store struct {

	// MinPartSize holds the minimum size of a multipart upload part.
	MinPartSize int64

	// MaxPartSize holds the maximum size of a multipart upload part.
	MaxPartSize int64

	// MaxParts holds the maximum number of parts that there
	// can be in a multipart upload.
	MaxParts int
	// contains filtered or unexported fields
}

Store stores data blobs in mongodb, de-duplicating by blob hash.

func New

func New(db *mgo.Database, prefix string, backend Backend) *Store

New returns a new blob store that writes to the given database, prefixing its collections with the given prefix.

func (*Store) FinishUpload

func (s *Store) FinishUpload(uploadId string, parts []Part) (idx *mongodoc.MultipartIndex, hash string, err error)

FinishUpload completes a multipart upload by joining all the given parts into one blob. The resulting blob can be opened by passing uploadId and the returned multipart index to Open.

The part numbers used will be from 0 to len(parts)-1.

This does not delete the multipart metadata, which should still be deleted explicitly by calling RemoveUpload after the index data is stored.

func (*Store) GC

func (s *Store) GC(refs *Refs, before time.Time) (monitoring.BlobStats, error)

GC runs the garbage collector, deleting all blobs not present in refs that have not been Put since the given time. Note that it also adds any internal blobs held by in-progress uploads to refs.

func (*Store) NewUpload

func (s *Store) NewUpload(expires time.Time) (uploadId string, err error)

NewUpload created a new multipart entry to track a multipart upload. It returns an uploadId that can be used to refer to it. After creating the upload, each part must be uploaded individually, and then the whole completed by calling FinishUpload and RemoveUpload.

func (*Store) Open

func (s *Store) Open(hash string, index *mongodoc.MultipartIndex) (ReadSeekCloser, int64, error)

Open opens the entry with the given hash. It returns an error with an ErrNotFound cause if the entry does not exist.

func (*Store) Put

func (s *Store) Put(r io.Reader, hash string, size int64) error

Put streams the content with the given hex-encoded SHA384 hash, and size from the given reader into blob storage.

func (*Store) PutAtTime

func (s *Store) PutAtTime(r io.Reader, hash string, size int64, now time.Time) error

PutAtTime is like Put but puts the content as if the current time is now. This should be used for testing purposes only.

func (*Store) PutPart

func (s *Store) PutPart(uploadId string, part int, r io.Reader, size, offset int64, hash string) error

PutPart uploads a part to the given upload id. The part number is specified with the part parameter; its content will be read from r and is expected to have the given size and hex-encoded SHA384 hash. A given part may not be uploaded more than once with a different hash.

If the upload id was not found (for example, because it's expired), PutPart returns an error with an ErrNotFound cause. If one of the parameters is badly formed, it returns an error with an ErrBadParams cause.

func (*Store) RemoveExpiredUploads

func (s *Store) RemoveExpiredUploads() error

RemoveExpiredUploads deletes any multipart entries that have passed their expiry date.

func (*Store) RemoveUpload

func (s *Store) RemoveUpload(uploadId string) error

RemoveUpload deletes all the parts associated with the given upload id. It is a no-op if called twice on the same upload id. If the upload has an owner and isOwnedBy is not nil, isOwnedBy will be called with uploadId and the upload's current owner to determine be sure that it is not actually used. If isOwnedBy is nil, it is assumed to return true - i.e. the document is owned.

func (*Store) SetOwner

func (s *Store) SetOwner(uploadId, owner string, expires time.Time) error

SetOwner sets the "owner" of a given upload. This should be set just before the owner document is updated to refer to the upload. SetOwner will fail if the upload is already owned. The upload expiry deadline is also set to expires, so the caller has that much leeway to associate the upload id with some owner document.

func (*Store) UploadInfo

func (s *Store) UploadInfo(uploadId string) (UploadInfo, error)

UploadInfo returns information on a given upload. It returns ErrNotFound if the upload has been deleted.

type UploadInfo

type UploadInfo struct {
	// Parts holds all the known parts of the upload.
	// Parts that haven't been uploaded yet will have nil
	// elements. Parts that are in progress or have been
	// aborted will have false Complete fields.
	Parts []*PartInfo

	// Expires holds when the upload will expire.
	Expires time.Time

	// Hash holds the hash of the entire upload.
	// This will be empty until the upload has
	// been completed with FinishUpload.
	Hash string `bson:"hash,omitempty"`
}

UploadInfo holds information on a given upload.

func (*UploadInfo) Index

func (info *UploadInfo) Index() (*mongodoc.MultipartIndex, bool)

Index returns a multipart index suitable for opening the multipart blob with the given info. It returns false if the info does not represent a completed upload.