blobserver

package
v0.0.0-...-4b1b879 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2013 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package blobserver defines how raw blobs are stored and accessed.

Index

Constants

View Source 
const MaxBlobSize = 16 << 20

MaxBlobSize is the size of a single blob in Camlistore.

TODO: formalize this in the specs. This value of 16 MB is less than App Engine's 32 MB request limit, much more than Venti's limit, and much more than the ~64 KB & 256 KB chunks that the FileWriter make

Variables

View Source 
var ErrCorruptBlob = errors.New("corrupt blob; digest doesn't match")
View Source 
var ErrHandlerTypeNotFound = errors.New("requested handler type not loaded")

Functions

func CreateHandler 

func CreateHandler(typ string, loader Loader, config jsonconfig.Obj) (http.Handler, error)

CreateHandler instantiates an http Handler of type 'typ' from the provided JSON configuration, and finding peer handlers and configuration from the environment in 'loader'.

The handler 'typ' must have bee previously registered with RegisterHandlerConstructor.

func EnumerateAll 

func EnumerateAll(src BlobEnumerator, fn func(blobref.SizedBlobRef) error) error

EnumerateAll runs fn for each blob in src. If fn returns an error, iteration stops and fn isn't called again. EnumerateAll will not return concurrently with fn.

func MergedEnumerate 

func MergedEnumerate(dest chan<- blobref.SizedBlobRef, sources []Storage, after string, limit int, wait time.Duration) error

TODO: it'd be nice to make sources be []BlobEnumerator, but that makes callers more complex since assignable interfaces' slice forms aren't assignable.

func RegisterHandlerConstructor 

func RegisterHandlerConstructor(typ string, ctor HandlerConstructor)

RegisterHandlerConstructor registers an http Handler constructor function for a given handler type.

It is an error to register the same handler type twice.

func RegisterStorageConstructor 

func RegisterStorageConstructor(typ string, ctor StorageConstructor)

func StatBlob 

func StatBlob(bs BlobStatter, br *blobref.BlobRef) (sb blobref.SizedBlobRef, err error)

func Unwrap 

func Unwrap(sto interface{}) interface{}

Unwrap returns the wrapped Storage interface, if wrapped, else returns sto.

Types

type BlobEnumerator 

type BlobEnumerator interface {
	// EnumerateBobs sends at most limit SizedBlobRef into dest,
	// sorted, as long as they are lexigraphically greater than
	// after (if provided).
	// limit will be supplied and sanity checked by caller.
	// wait is the max time to wait for any blobs to exist,
	// or 0 for no delay.
	// EnumerateBlobs must close the channel.  (even if limit
	// was hit and more blobs remain)
	//
	// after and waitSeconds can't be used together. One must be
	// its zero value.
	EnumerateBlobs(dest chan<- blobref.SizedBlobRef,
		after string,
		limit int,
		wait time.Duration) error
}

type BlobHub 

type BlobHub interface {
	// For new blobs to notify
	NotifyBlobReceived(blob *blobref.BlobRef)

	RegisterListener(ch chan *blobref.BlobRef)
	UnregisterListener(ch chan *blobref.BlobRef)

	RegisterBlobListener(blob *blobref.BlobRef, ch chan *blobref.BlobRef)
	UnregisterBlobListener(blob *blobref.BlobRef, ch chan *blobref.BlobRef)
}

type BlobReceiveConfiger 

type BlobReceiveConfiger interface {
	BlobReceiver
	Configer
}

type BlobReceiver 

type BlobReceiver interface {
	// ReceiveBlob accepts a newly uploaded blob and writes it to
	// disk.
	ReceiveBlob(blob *blobref.BlobRef, source io.Reader) (blobref.SizedBlobRef, error)
}

type BlobRemover 

type BlobRemover interface {
	// RemoveBlobs removes 0 or more blobs.  Removal of
	// non-existent items isn't an error.  Returns failure if any
	// items existed but failed to be deleted.
	RemoveBlobs(blobs []*blobref.BlobRef) error
}

type BlobStatter 

type BlobStatter interface {
	// Stat checks for the existence of blobs, writing their sizes
	// (if found back to the dest channel), and returning an error
	// or nil.  Stat() should NOT close the channel.
	// wait is the max time to wait for the blobs to exist,
	// or 0 for no delay.
	StatBlobs(dest chan<- blobref.SizedBlobRef,
		blobs []*blobref.BlobRef,
		wait time.Duration) error
}

type Cache 

type Cache interface {
	blobref.SeekFetcher
	BlobReceiver
	BlobStatter
}

Cache is the minimal interface expected of a blob cache.

type Config 

type Config struct {
	Writable, Readable bool
	IsQueue            bool // supports deletes
	CanLongPoll        bool

	// the "http://host:port" and optional path (but without trailing slash) to have "/camli/*" appended
	URLBase       string
	HandlerFinder FindHandlerByTyper
}

type Configer 

type Configer interface {
	Config() *Config
}

type ContextWrapper 

type ContextWrapper interface {
	WrapContext(*http.Request) Storage
}

ContextWrapper is an optional interface for App Engine.

While Camlistore's internals are separated out into a part which maps HTTP requests to the interfaces in this file (pkg/blobserver/handlers) and parts which map these interfaces to implementations (localdisk, s3, etc), the App Engine implementation requires access to the original HTTP request. (because a security token is stored on the incoming HTTP request in a magic header). All the handlers will do an interface check on this type and use the resulting Storage instead.

type FindHandlerByTyper 

type FindHandlerByTyper interface {
	// FindHandlerByType finds a handler by its handlerType and
	// returns its prefix and handler if it's loaded.  If it's not
	// loaded, the error will be ErrHandlerTypeNotFound.
	//
	// This is used by handlers to find siblings (such as the "ui" type handler)
	// which might have more knowledge about the configuration for discovery, etc.
	//
	// Note that if this is called during handler construction
	// time, only the prefix may be returned with a nil handler
	// and nil err.  Unlike GetHandler and GetStorage, this does
	// not cause the prefix to load immediately. At runtime (after
	// construction of all handlers), then prefix and handler will
	// both be non-nil when err is nil.
	FindHandlerByType(handlerType string) (prefix string, handler interface{}, err error)
}

type GenerationNotSupportedError 

type GenerationNotSupportedError string

A GenerationNotSupportedError explains why a Storage value implemented the Generationer interface but failed due to a wrapped Storage value not implementing the interface.

func (GenerationNotSupportedError) Error 

func (s GenerationNotSupportedError) Error() string

type Generationer 

type Generationer interface {
	// Generation returns a Storage's initialization time and
	// and unique random string (or UUID).  Implementations
	// should call ResetStorageGeneration on demand if no
	// information is known.
	// The error will be of type GenerationNotSupportedError if an underlying
	// storage target doesn't support the Generationer interface.
	StorageGeneration() (initTime time.Time, random string, err error)

	// ResetGeneration deletes the information returned by Generation
	// and re-generates it.
	ResetStorageGeneration() error
}

The optional Generationer interface is an optimization and paranoia facility for clients which can be implemented by Storage implementations.

If the client sees the same random string in multiple upload sessions, it assumes that the blobserver still has all the same blobs, and also it's the same server. This mechanism is not fundamental to Camlistore's operation: the client could also check each blob before uploading, or enumerate all blobs from the server too. This is purely an optimization so clients can mix this value into their "is this file uploaded?" local cache keys.

type HandlerConstructor 

type HandlerConstructor func(Loader, jsonconfig.Obj) (http.Handler, error)

A HandlerConstructor returns an http.Handler from a Loader environment and a configuration.

type Loader 

type Loader interface {
	FindHandlerByTyper

	// MyPrefix returns the prefix of the handler currently being constructed.
	MyPrefix() string

	// GetHandlerType returns the handler's configured type, but does
	// not force it to start being loaded yet.
	GetHandlerType(prefix string) string // returns "" if unknown

	// GetHandler returns either a Storage or an http.Handler.
	// It forces the handler to be loaded and returns an error if
	// a cycle is created.
	GetHandler(prefix string) (interface{}, error)

	// GetStorage is like GetHandler but requires that the Handler be
	// a storage Handler.
	GetStorage(prefix string) (Storage, error)

	// If we're loading configuration in response to a web request
	// (as we do with App Engine), then this returns a request and
	// true.
	GetRequestContext() (ctx *http.Request, ok bool)
}

type MaxEnumerateConfig 

type MaxEnumerateConfig interface {
	// Returns the max that this storage interface is capable
	// of enumerating at once.
	MaxEnumerate() int
}

type NoImplStorage 

type NoImplStorage struct{}

NoImplStorage is an implementation of Storage that return a not implemented error for all operations.

func (*NoImplStorage) EnumerateBlobs 

func (nis *NoImplStorage) EnumerateBlobs(dest chan<- blobref.SizedBlobRef,
	after string,
	limit int,
	wait time.Duration) error

func (*NoImplStorage) Fetch 

func (nis *NoImplStorage) Fetch(*blobref.BlobRef) (file types.ReadSeekCloser, size int64, err error)

func (*NoImplStorage) FetchStreaming 

func (nis *NoImplStorage) FetchStreaming(*blobref.BlobRef) (file io.ReadCloser, size int64, err error)

func (*NoImplStorage) GetBlobHub 

func (nis *NoImplStorage) GetBlobHub() BlobHub

func (*NoImplStorage) ReceiveBlob 

func (nis *NoImplStorage) ReceiveBlob(blob *blobref.BlobRef, source io.Reader) (sb blobref.SizedBlobRef, err error)

func (*NoImplStorage) RemoveBlobs 

func (nis *NoImplStorage) RemoveBlobs(blobs []*blobref.BlobRef) error

func (*NoImplStorage) StatBlobs 

func (nis *NoImplStorage) StatBlobs(dest chan<- blobref.SizedBlobRef,
	blobs []*blobref.BlobRef,
	wait time.Duration) error

type QueueCreator 

type QueueCreator interface {
	CreateQueue(name string) (Storage, error)
}

QueueCreator is implemented by Storage interfaces which support creating queues in which all new uploads go to both the root storage as well as the named queue, which is then returned. This is used by replication.

type SimpleBlobHub 

type SimpleBlobHub struct {
	// contains filtered or unexported fields
}

func (*SimpleBlobHub) NotifyBlobReceived 

func (h *SimpleBlobHub) NotifyBlobReceived(blob *blobref.BlobRef)

func (*SimpleBlobHub) RegisterBlobListener 

func (h *SimpleBlobHub) RegisterBlobListener(blob *blobref.BlobRef, ch chan *blobref.BlobRef)

func (*SimpleBlobHub) RegisterListener 

func (h *SimpleBlobHub) RegisterListener(ch chan *blobref.BlobRef)

func (*SimpleBlobHub) UnregisterBlobListener 

func (h *SimpleBlobHub) UnregisterBlobListener(blob *blobref.BlobRef, ch chan *blobref.BlobRef)

func (*SimpleBlobHub) UnregisterListener 

func (h *SimpleBlobHub) UnregisterListener(ch chan *blobref.BlobRef)

type SimpleBlobHubPartitionMap 

type SimpleBlobHubPartitionMap struct {
	// contains filtered or unexported fields
}

func (*SimpleBlobHubPartitionMap) GetBlobHub 

func (spm *SimpleBlobHubPartitionMap) GetBlobHub() BlobHub

type StatReceiver 

type StatReceiver interface {
	BlobReceiver
	BlobStatter
}

type Storage 

type Storage interface {
	blobref.StreamingFetcher
	BlobReceiver
	BlobStatter
	BlobEnumerator
	BlobRemover

	// GetBlobHub returns the blob notification bus.
	GetBlobHub() BlobHub
}

Storage is the interface that must be implemented by a blobserver storage type. (e.g. localdisk, s3, encrypt, shard, replica, remote)

func CreateStorage 

func CreateStorage(typ string, loader Loader, config jsonconfig.Obj) (Storage, error)

func MaybeWrapContext 

func MaybeWrapContext(sto Storage, req *http.Request) Storage

type StorageConfiger 

type StorageConfiger interface {
	Storage
	Configer
}

type StorageConstructor 

type StorageConstructor func(Loader, jsonconfig.Obj) (Storage, error)

A StorageConstructor returns a Storage implementation from a Loader environment and a configuration.

type StorageQueueCreator 

type StorageQueueCreator interface {
	Storage
	QueueCreator
}

Source Files

View all Source files

Directories

Path Synopsis
Package cond registers the "cond" conditional blobserver storage type to select routing of get/put operations on blobs to other storage targets as a function of their content.
 Package cond registers the "cond" conditional blobserver storage type to select routing of get/put operations on blobs to other storage targets as a function of their content.
Package encrypt registers the "encrypt" blobserver storage type which stores all blobs and metadata with AES encryption into other wrapped storage targets (e.g.
 Package encrypt registers the "encrypt" blobserver storage type which stores all blobs and metadata with AES encryption into other wrapped storage targets (e.g.
Package gethandler implements the HTTP handler for fetching blobs.
 Package gethandler implements the HTTP handler for fetching blobs.
Package google registers the "google" blob storage type, storing blobs on Google Cloud Storage (not Google Drive).
 Package google registers the "google" blob storage type, storing blobs on Google Cloud Storage (not Google Drive).
Package handlers implements the HTTP interface to the Camlistore blob server.
 Package handlers implements the HTTP interface to the Camlistore blob server.
Package localdisk registers the "filesystem" blobserver storage type, storing blobs in a forest of sharded directories at the specified root.
 Package localdisk registers the "filesystem" blobserver storage type, storing blobs in a forest of sharded directories at the specified root.
Package remote registers the "remote" blobserver storage type, storing and fetching blobs from a remote Camlistore server, speaking the HTTP protocol.
 Package remote registers the "remote" blobserver storage type, storing and fetching blobs from a remote Camlistore server, speaking the HTTP protocol.
Package replica registers the "replica" blobserver storage type, providing synchronous replication to one more backends.
 Package replica registers the "replica" blobserver storage type, providing synchronous replication to one more backends.
Package s3 registers the "s3" blobserver storage type, storing blobs in an Amazon Web Services' S3 storage bucket.
 Package s3 registers the "s3" blobserver storage type, storing blobs in an Amazon Web Services' S3 storage bucket.
Package shard registers the "shard" blobserver storage type, predictably spraying out blobs out over the provided backends based on their blobref.
 Package shard registers the "shard" blobserver storage type, predictably spraying out blobs out over the provided backends based on their blobref.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.