README
¶
Introduction
Cloudstorage is an library for working with Cloud Storage (Google, AWS, Azure) and SFTP, Local Files. It provides a unified api for local files, sftp and Cloud files that aids testing and operating on multiple cloud storage.
Features
- Provide single unified api for multiple cloud (google, azure, aws) & local files.
- Cloud Upload/Download is unified in api so you don't have to download file to local, work with it, then upload.
- Buffer/Cache files from cloud local so speed of usage is very high.
Similar/Related works
- https://github.com/graymeta/stow similar to this pkg, library for interacting with cloud services. Less of the buffer/local cache. Different clouds.
- https://github.com/ncw/rclone great cli sync tool, many connections (30+), well tested. Designed as cli tool, config is less suited for use as library.
Example usage:
Note: For these examples all errors are ignored, using the _ for them.
// This is an example of a local storage object:
// See(https://github.com/lytics/cloudstorage/blob/master/google/google_test.go) for a GCS example:
config := &cloudstorage.Config{
Type: localfs.StoreType,
AuthMethod: localfs.AuthFileSystem,
LocalFS: "/tmp/mockcloud",
TmpDir: "/tmp/localcache",
}
store, _ := cloudstorage.NewStore(config)
See go Iterator pattern doc for api-design: https://github.com/GoogleCloudPlatform/google-cloud-go/wiki/Iterator-Guidelines
// From a store that has been created
// Create a query
q := cloudstorage.NewQuery("list-test/")
// Create an Iterator
iter, err := store.Objects(context.Background(), q)
if err != nil {
// handle
}
for {
o, err := iter.Next()
if err == iterator.Done {
break
}
log.Println("found object ", o.Name())
}
obj, _ := store.NewObject("prefix/test.csv")
// open for read and writing. f is a filehandle to the local filesystem.
f, _ := obj.Open(cloudstorage.ReadWrite)
w := bufio.NewWriter(f)
_, _ := w.WriteString("Year,Make,Model\n")
_, _ := w.WriteString("1997,Ford,E350\n")
w.Flush()
// Close sync's the local file to the remote store and removes the local tmp file.
obj.Close()
// Calling Get on an existing object will return a cloudstorage object or the cloudstorage.ErrObjectNotFound error.
obj2, _ := store.Get(context.Background(), "prefix/test.csv")
f2, _ := obj2.Open(cloudstorage.ReadOnly)
bytes, _ := ioutil.ReadAll(f2)
fmt.Println(string(bytes)) // should print the CSV file from the block above...
var config = &storeutils.TransferConfig{
Type: google.StoreType,
AuthMethod: google.AuthGCEDefaultOAuthToken
ProjectID: "my-project",
DestBucket: "my-destination-bucket",
Src: storeutils.NewGcsSource("my-source-bucket"),
IncludePrefxies: []string{"these", "prefixes"},
}
transferer, _ := storeutils.NewTransferer(client)
resp, _ := transferer.NewTransfer(config)
See testsuite.go for more examples
Testing
Due to the way integration tests act against a cloud bucket and objects; run tests without parallelization.
cd $GOPATH/src/github.com/lytics/cloudstorage
go test -p 1 ./...
Documentation
¶
Overview ¶
Package cloudstorage is an interface to make Local, Google, s3 file storage share a common interface to aid testing local as well as running in the cloud.
The primary goal is to create a Store which is a common interface over each of the (google, s3, local-file-system, azure) etc file storage systems. Then the methods (Query, filter, get, put) are common, as are the Files (Objects) themselves. Writing code that supports multiple backends is now simple.
Creating and iterating files ¶
In this example we are going to create a local-filesystem store.
// This is an example of a local-storage (local filesystem) provider:
config := &cloudstorage.Config{
Type: localfs.StoreType,
TokenSource: localfs.AuthFileSystem,
LocalFS: "/tmp/mockcloud",
TmpDir: "/tmp/localcache",
}
store, _ := cloudstorage.NewStore(config)
// Create a query to define the search path
q := cloudstorage.NewQuery("list-test/")
// Create an Iterator to list files
iter := store.Objects(context.Background(), q)
for {
o, err := iter.Next()
if err == iterator.Done {
break
}
log.Println("found object %v", o.Name())
}
Index ¶
- Constants
- Variables
- func Backoff(try int)
- func CachePathObj(cachepath, oname, storeid string) string
- func CleanETag(etag string) string
- func CleanupCacheFiles(maxage time.Duration, TmpDir string) (err error)
- func ContentType(name string) string
- func Copy(ctx context.Context, s Store, src, des Object) error
- func EnsureContextType(o string, md map[string]string) string
- func EnsureDir(filename string) error
- func Exists(filename string) bool
- func Move(ctx context.Context, s Store, src, des Object) error
- func Register(storeType string, provider StoreProvider)
- type AccessLevel
- type AuthMethod
- type Config
- type Filter
- type JwtConf
- type Object
- type ObjectIterator
- type ObjectPageIterator
- type Objects
- type ObjectsResponse
- type Query
- type Store
- type StoreCopy
- type StoreMove
- type StoreProvider
- type StoreReader
Constants ¶
const ( // StoreCacheFileExt = ".cache" StoreCacheFileExt = ".cache" // ContentTypeKey ContentTypeKey = "content_type" // MaxResults default number of objects to retrieve during a list-objects request, // if more objects exist, then they will need to be paged MaxResults = 3000 )
Variables ¶
var ( // ErrObjectNotFound Error of not finding a file(object) ErrObjectNotFound = fmt.Errorf("object not found") // ErrObjectExists error trying to create an already existing file. ErrObjectExists = fmt.Errorf("object already exists in backing store (use store.Get)") // ErrNotImplemented this feature is not implemented for this store ErrNotImplemented = fmt.Errorf("Not implemented") )
var ObjectSortFilter = func(objs Objects) Objects { sort.Stable(objs) return objs }
Functions ¶
func Backoff ¶
func Backoff(try int)
Backoff sleeps a random amount so we can. retry failed requests using a randomized exponential backoff: wait a random period between [0..1] seconds and retry; if that fails, wait a random period between [0..2] seconds and retry; if that fails, wait a random period between [0..4] seconds and retry, and so on, with an upper bounds to the wait period being 16 seconds. http://play.golang.org/p/l9aUHgiR8J
func CachePathObj ¶
CachePathObj check the cache path.
func CleanETag ¶
CleanETag transforms a string into the full etag spec, removing extra quote-marks, whitespace from etag.
per Etag spec https://tools.ietf.org/html/rfc7232#section-2.3 the etag value (<ETAG VALUE>) may: - W/"<ETAG VALUE>" - "<ETAG VALUE>" - ""
func CleanupCacheFiles ¶
CleanupCacheFiles cleans up old store cache files if your process crashes all it's old cache files, the local copies of the cloudfiles, will left behind. This function is a convenience func to help clean up those old files.
I suggest you call this behind a package var sync.Once struct, so its only called at the startup of your application.
func ContentType ¶
ContentType check content type of file by looking at extension (.html, .png) uses package mime for global types. Use mime.AddExtensionType to add new global types.
func EnsureContextType ¶
EnsureContextType read Type of metadata
func Register ¶
func Register(storeType string, provider StoreProvider)
Register adds a store type provider.
Types ¶
type AccessLevel ¶
type AccessLevel int
AccessLevel is the level of permissions on files
const ( // ReadOnly File Permissions Levels ReadOnly AccessLevel = 0 ReadWrite AccessLevel = 1 )
type Config ¶
type Config struct {
// Type is StoreType [gcs,localfs,s3,azure]
Type string
// AuthMethod the methods of authenticating store. Ie, where/how to
// find auth tokens.
AuthMethod AuthMethod
// Cloud Bucket Project
Project string
// Region is the cloud region
Region string
// Bucket is the "path" or named bucket in cloud
Bucket string
// the page size to use with api requests (default 1000)
PageSize int
// used by JWTKeySource
JwtConf *JwtConf
// JwtFile is the file-path to local auth-token file.
JwtFile string `json:"jwtfile,omitempty"`
// BaseUrl is the base-url path for customizing regions etc. IE
// AWS has different url paths per region on some situations.
BaseUrl string `json:"baseurl,omitempty"`
// Permissions scope
Scope string `json:"scope,omitempty"`
// LocalFS is filesystem path to use for the local files
// for Type=localfs
LocalFS string `json:"localfs,omitempty"`
// The filesystem path to save locally cached files as they are
// being read/written from cloud and need a staging area.
TmpDir string `json:"tmpdir,omitempty"`
// Settings are catch-all-bag to allow per-implementation over-rides
Settings gou.JsonHelper `json:"settings,omitempty"`
// LogPrefix Logging Prefix/Context message
LogPrefix string
}
Config the cloud store config settings.
type JwtConf ¶
type JwtConf struct {
// Unfortuneately we departed from the standard jwt service account field-naming
// for reasons we forgot. So, during load, we convert from bad->correct format.
PrivateKeyDeprecated string `json:"private_keybase64,omitempty"`
KeyTypeDeprecated string `json:"keytype,omitempty"`
// Jwt Service Account Fields
ProjectID string `json:"project_id,omitempty"`
PrivateKeyID string `json:"private_key_id,omitempty"`
PrivateKey string `json:"private_key,omitempty"`
ClientEmail string `json:"client_email,omitempty"`
ClientID string `json:"client_id,omitempty"`
Type string `json:"type,omitempty"`
// Scopes is list of what scope to use when the token is created.
// for example https://github.com/google/google-api-go-client/blob/0d3983fb069cb6651353fc44c5cb604e263f2a93/storage/v1/storage-gen.go#L54
Scopes []string `json:"scopes,omitempty"`
}
JwtConf For use with google/google_jwttransporter.go Which can be used by the google go sdk's. This struct is based on the Google Jwt files json for service accounts.
type Object ¶
type Object interface {
// Name of object/file.
Name() string
// String is default descriptor.
String() string
// Updated timestamp.
Updated() time.Time
// MetaData is map of arbitrary name/value pairs about object.
MetaData() map[string]string
// SetMetaData allows you to set key/value pairs.
SetMetaData(meta map[string]string)
// StorageSource is the type of store.
StorageSource() string
// Open copies the remote file to a local cache and opens the cached version
// for read/writing. Calling Close/Sync will push the copy back to the
// backing store.
Open(readonly AccessLevel) (*os.File, error)
// Release will remove the locally cached copy of the file. You most call Close
// before releasing. Release will call os.Remove(local_copy_file) so opened
// filehandles need to be closed.
Release() error
// Implement io.ReadWriteCloser Open most be called before using these
// functions.
Read(p []byte) (n int, err error)
Write(p []byte) (n int, err error)
Sync() error
Close() error
// File returns the cached/local copy of the file
File() *os.File
// Delete removes the object from the cloud store and local cache.
Delete() error
}
Object is a handle to a cloud stored file/object. Calling Open will pull the remote file onto your local filesystem for reading/writing. Calling Sync/Close will push the local copy backup to the cloud store.
type ObjectIterator ¶
type ObjectIterator interface {
// Next gets next object, returns google.golang.org/api/iterator iterator.Done error.
Next() (Object, error)
// Close this down (and or context.Close)
Close()
}
ObjectIterator interface to page through objects See go doc for examples https://github.com/GoogleCloudPlatform/google-cloud-go/wiki/Iterator-Guidelines
func NewObjectPageIterator ¶
func NewObjectPageIterator(ctx context.Context, s Store, q Query) ObjectIterator
NewObjectPageIterator create an iterator that wraps the store List interface.
type ObjectPageIterator ¶
type ObjectPageIterator struct {
// contains filtered or unexported fields
}
ObjectPageIterator iterator to facilitate easy paging through store.List() method to read all Objects that matched query.
func (*ObjectPageIterator) Next ¶
func (it *ObjectPageIterator) Next() (Object, error)
Next iterator to go to next object or else returns error for done.
type Objects ¶
type Objects []Object
Objects are just a collection of Object(s). Used as the results for store.List commands.
func ObjectsAll ¶
func ObjectsAll(iter ObjectIterator) (Objects, error)
ObjectsAll get all objects for an iterator.
type ObjectsResponse ¶
ObjectsResponse for paged object apis.
func NewObjectsResponse ¶
func NewObjectsResponse() *ObjectsResponse
func ObjectResponseFromIter ¶
func ObjectResponseFromIter(iter ObjectIterator) (*ObjectsResponse, error)
ObjectResponseFromIter get all objects for an iterator.
type Query ¶
type Query struct {
Delimiter string // Delimiter is most likely "/"
Prefix string // prefix (directory) to search for or object name if one file
Marker string // Next Page Marker if provided is a start next page fetch bookmark.
ShowHidden bool // Show hidden files?
Filters []Filter // Applied to the result sets to filter out Objects (i.e. remove objects by extension)
PageSize int // PageSize defaults to global, or you can supply an override
}
Query used to query the cloud source. The primary query is a prefix query like `ls /my-csv-files/baseball/*`. This is the Request, and includes the PageSize, cursor/next token as well.
func NewQueryForFolders ¶
NewQueryForFolders create a query for finding Folders under given path.
func (*Query) AddFilter ¶
AddFilter adds a post prefix query, that can be used to alter results set from the prefix query.
func (*Query) ApplyFilters ¶
ApplyFilters is called as the last step in store.List() to filter out the results before they are returned.
type Store ¶
type Store interface {
StoreReader
// NewWriter returns a io.Writer that writes to a Cloud object
// associated with this backing Store object.
//
// A new object will be created if an object with this name already exists.
// Otherwise any previous object with the same name will be replaced.
// The object will not be available (and any previous object will remain)
// until Close has been called
NewWriter(o string, metadata map[string]string) (io.WriteCloser, error)
// NewWriter but with context.
NewWriterWithContext(ctx context.Context, o string, metadata map[string]string) (io.WriteCloser, error)
// NewObject creates a new empty object backed by the cloud store
// This new object isn't' synced/created in the backing store
// until the object is Closed/Sync'ed.
NewObject(o string) (Object, error)
// Delete removes the object from the cloud store.
Delete(ctx context.Context, o string) error
}
Store interface to define the Storage Interface abstracting the GCS, S3, LocalFile interfaces
type StoreCopy ¶
type StoreCopy interface {
// Copy from object, to object
Copy(ctx context.Context, src, dst Object) error
}
StoreCopy Optional interface to fast path copy. Many of the cloud providers don't actually copy bytes. Rather they allow a "pointer" that is a fast copy.
type StoreMove ¶
type StoreMove interface {
// Move from object location, to object location.
Move(ctx context.Context, src, dst Object) error
}
StoreMove Optional interface to fast path move. Many of the cloud providers don't actually copy bytes.
type StoreProvider ¶
StoreProvider a provider function for creating New Stores
type StoreReader ¶
type StoreReader interface {
// Type is he Store Type [google, s3, azure, localfs, etc]
Type() string
// Client gets access to the underlying native Client for Google, S3, etc
Client() interface{}
// Get returns an object (file) from the cloud store. The object
// isn't opened already, see Object.Open()
// ObjectNotFound will be returned if the object is not found.
Get(ctx context.Context, o string) (Object, error)
// Objects returns an object Iterator to allow paging through object
// which keeps track of page cursors. Query defines the specific set
// of filters to apply to request.
Objects(ctx context.Context, q Query) (ObjectIterator, error)
// List file/objects filter by given query. This just wraps the object-iterator
// returning full list of objects.
List(ctx context.Context, q Query) (*ObjectsResponse, error)
// Folders creates list of folders
Folders(ctx context.Context, q Query) ([]string, error)
// NewReader creates a new Reader to read the contents of the object.
// ErrObjectNotFound will be returned if the object is not found.
NewReader(o string) (io.ReadCloser, error)
// NewReader with context (for cancelation, etc)
NewReaderWithContext(ctx context.Context, o string) (io.ReadCloser, error)
// String default descriptor.
String() string
}
StoreReader interface to define the Storage Interface abstracting the GCS, S3, LocalFile, etc interfaces
Source Files
Directories