swift

package module
v1.0.36 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 13, 2017 License: MIT Imports: 22 Imported by: 0

README

Swift

This package provides an easy to use library for interfacing with Swift / Openstack Object Storage / Rackspace cloud files from the Go Language

See here for package docs

http://godoc.org/github.com/ncw/swift

Build Status GoDoc

Install

Use go to install the library

go get github.com/ncw/swift

Usage

See here for full package docs

Here is a short example from the docs

import "github.com/ncw/swift"

// Create a connection
c := swift.Connection{
    UserName: "user",
    ApiKey:   "key",
    AuthUrl:  "auth_url",
    Domain:   "domain",  // Name of the domain (v3 auth only)
    Tenant:   "tenant",  // Name of the tenant (v2 auth only)
}
// Authenticate
err := c.Authenticate()
if err != nil {
    panic(err)
}
// List all the containers
containers, err := c.ContainerNames(nil)
fmt.Println(containers)
// etc...

Additions

The rs sub project contains a wrapper for the Rackspace specific CDN Management interface.

Testing

To run the tests you can either use an embedded fake Swift server either use a real Openstack Swift server or a Rackspace Cloud files account.

When using a real Swift server, you need to set these environment variables before running the tests

export SWIFT_API_USER='user'
export SWIFT_API_KEY='key'
export SWIFT_AUTH_URL='https://url.of.auth.server/v1.0'

And optionally these if using v2 authentication

export SWIFT_TENANT='TenantName'
export SWIFT_TENANT_ID='TenantId'

And optionally these if using v3 authentication

export SWIFT_TENANT='TenantName'
export SWIFT_TENANT_ID='TenantId'
export SWIFT_API_DOMAIN_ID='domain id'
export SWIFT_API_DOMAIN='domain name'

And optionally these if using v3 trust

export SWIFT_TRUST_ID='TrustId'

And optionally this if you want to skip server certificate validation

export SWIFT_AUTH_INSECURE=1

And optionally this to configure the connect channel timeout, in seconds

export SWIFT_CONNECTION_CHANNEL_TIMEOUT=60

And optionally this to configure the data channel timeout, in seconds

export SWIFT_DATA_CHANNEL_TIMEOUT=60

Then run the tests with go test

License

This is free software under the terms of MIT license (check COPYING file included in this package).

Contact and support

The project website is at:

There you can file bug reports, ask for help or contribute patches.

Authors

Contributors

Documentation

Overview

Package swift provides an easy to use interface to Swift / Openstack Object Storage / Rackspace Cloud Files

Standard Usage

Most of the work is done through the Container*() and Object*() methods.

All methods are safe to use concurrently in multiple go routines.

Object Versioning

As defined by http://docs.openstack.org/api/openstack-object-storage/1.0/content/Object_Versioning-e1e3230.html#d6e983 one can create a container which allows for version control of files. The suggested method is to create a version container for holding all non-current files, and a current container for holding the latest version that the file points to. The container and objects inside it can be used in the standard manner, however, pushing a file multiple times will result in it being copied to the version container and the new file put in it's place. If the current file is deleted, the previous file in the version container will replace it. This means that if a file is updated 5 times, it must be deleted 5 times to be completely removed from the system.

Rackspace Sub Module

This module specifically allows the enabling/disabling of Rackspace Cloud File CDN management on a container. This is specific to the Rackspace API and not Swift/Openstack, therefore it has been placed in a submodule. One can easily create a RsConnection and use it like the standard Connection to access and manipulate containers and objects.

Index

Examples

Constants

View Source 
const (
	// Use public URL as storage URL
	EndpointTypePublic = EndpointType("public")

	// Use internal URL as storage URL
	EndpointTypeInternal = EndpointType("internal")

	// Use admin URL as storage URL
	EndpointTypeAdmin = EndpointType("admin")
)
View Source 
const (
	DefaultUserAgent = "goswift/1.0"         // Default user agent
	DefaultRetries   = 3                     // Default number of retries on token expiry
	TimeFormat       = "2006-01-02T15:04:05" // Python date format for json replies parsed as UTC
	UploadTar        = "tar"                 // Data format specifier for Connection.BulkUpload().
	UploadTarGzip    = "tar.gz"              // Data format specifier for Connection.BulkUpload().
	UploadTarBzip2   = "tar.bz2"             // Data format specifier for Connection.BulkUpload().

)

Variables

View Source 
var (
	// Specific Errors you might want to check for equality
	NotModified         = newError(304, "Not Modified")
	BadRequest          = newError(400, "Bad Request")
	AuthorizationFailed = newError(401, "Authorization Failed")
	ContainerNotFound   = newError(404, "Container Not Found")
	ContainerNotEmpty   = newError(409, "Container Not Empty")
	ObjectNotFound      = newError(404, "Object Not Found")
	ObjectCorrupted     = newError(422, "Object Corrupted")
	TimeoutError        = newError(408, "Timeout when reading or writing data")
	Forbidden           = newError(403, "Operation forbidden")
	TooLargeObject      = newError(413, "Too Large Object")
	RateLimit           = newError(498, "Rate Limit")

	// Mappings for container errors
	ContainerErrorMap = errorMap{
		400: BadRequest,
		403: Forbidden,
		404: ContainerNotFound,
		409: ContainerNotEmpty,
		498: RateLimit,
	}
)
View Source 
var NotLargeObject = errors.New("Not a large object")

NotLargeObject is returned if an operation is performed on an object which isn't large.

View Source 
var SLONotSupported = errors.New("SLO not supported")

Functions

func FloatStringToTime 

func FloatStringToTime(s string) (t time.Time, err error)

FloatStringToTime converts a floating point number string to a time.Time

The string is floating point number of seconds since the epoch (Unix time). The number should be in fixed point format (not exponential), eg "1354040105.123456789" which represents the time "2012-11-27T18:15:05.123456789Z"

Some care is taken to preserve all the accuracy in the time.Time (which wouldn't happen with a naive conversion through float64) so a round trip conversion won't change the data.

If an error is returned then time will be returned as the zero time.

func TimeToFloatString 

func TimeToFloatString(t time.Time) string

TimeToFloatString converts a time.Time object to a floating point string

The string is floating point number of seconds since the epoch (Unix time). The number is in fixed point format (not exponential), eg "1354040105.123456789" which represents the time "2012-11-27T18:15:05.123456789Z". Trailing zeros will be dropped from the output.

Some care is taken to preserve all the accuracy in the time.Time (which wouldn't happen with a naive conversion through float64) so a round trip conversion won't change the data.

Types

type Account 

type Account struct {
	BytesUsed  int64 // total number of bytes used
	Containers int64 // total number of containers
	Objects    int64 // total number of objects
}

Account contains information about this account.

type Authenticator added in v1.0.2 

type Authenticator interface {
	// Request creates an http.Request for the auth - return nil if not needed
	Request(*Connection) (*http.Request, error)
	// Response parses the http.Response
	Response(resp *http.Response) error
	// The public storage URL - set Internal to true to read
	// internal/service net URL
	StorageUrl(Internal bool) string
	// The access token
	Token() string
	// The CDN url if available
	CdnUrl() string
}

Auth defines the operations needed to authenticate with swift

This encapsulates the different authentication schemes in use

type BulkDeleteResult added in v1.0.3 

type BulkDeleteResult struct {
	NumberNotFound int64            // # of objects not found.
	NumberDeleted  int64            // # of deleted objects.
	Errors         map[string]error // Mapping between object name and an error.
	Headers        Headers          // Response HTTP headers.
}

BulkDeleteResult stores results of BulkDelete().

Individual errors may (or may not) be returned by Errors. Errors is a map whose keys are a full path of where the object was to be deleted, and whose values are Error objects. A full path of object looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".

type BulkUploadResult added in v1.0.4 

type BulkUploadResult struct {
	NumberCreated int64            // # of created objects.
	Errors        map[string]error // Mapping between object name and an error.
	Headers       Headers          // Response HTTP headers.
}

BulkUploadResult stores results of BulkUpload().

Individual errors may (or may not) be returned by Errors. Errors is a map whose keys are a full path of where an object was to be created, and whose values are Error objects. A full path of object looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".

type Connection 

type Connection struct {
	// Parameters - fill these in before calling Authenticate
	// They are all optional except UserName, ApiKey and AuthUrl
	Domain         string            // User's domain name
	DomainId       string            // User's domain Id
	UserName       string            // UserName for api
	UserId         string            // User Id
	ApiKey         string            // Key for api access
	AuthUrl        string            // Auth URL
	Retries        int               // Retries on error (default is 3)
	UserAgent      string            // Http User agent (default goswift/1.0)
	ConnectTimeout time.Duration     // Connect channel timeout (default 10s)
	Timeout        time.Duration     // Data channel timeout (default 60s)
	Region         string            // Region to use eg "LON", "ORD" - default is use first region (v2,v3 auth only)
	AuthVersion    int               // Set to 1, 2 or 3 or leave at 0 for autodetect
	Internal       bool              // Set this to true to use the the internal / service network
	Tenant         string            // Name of the tenant (v2,v3 auth only)
	TenantId       string            // Id of the tenant (v2,v3 auth only)
	EndpointType   EndpointType      // Endpoint type (v2,v3 auth only) (default is public URL unless Internal is set)
	TenantDomain   string            // Name of the tenant's domain (v3 auth only), only needed if it differs from the user domain
	TenantDomainId string            // Id of the tenant's domain (v3 auth only), only needed if it differs the from user domain
	TrustId        string            // Id of the trust (v3 auth only)
	Transport      http.RoundTripper `json:"-" xml:"-"` // Optional specialised http.Transport (eg. for Google Appengine)
	// These are filled in after Authenticate is called as are the defaults for above
	StorageUrl string
	AuthToken  string

	Auth Authenticator `json:"-" xml:"-"` // the current authenticator
	// contains filtered or unexported fields
}

Connection holds the details of the connection to the swift server.

You need to provide UserName, ApiKey and AuthUrl when you create a connection then call Authenticate on it.

The auth version in use will be detected from the AuthURL - you can override this with the AuthVersion parameter.

If using v2 auth you can also set Region in the Connection structure. If you don't set Region you will get the default region which may not be what you want.

For reference some common AuthUrls looks like this: 

Rackspace US        https://auth.api.rackspacecloud.com/v1.0
Rackspace UK        https://lon.auth.api.rackspacecloud.com/v1.0
Rackspace v2        https://identity.api.rackspacecloud.com/v2.0
Memset Memstore UK  https://auth.storage.memset.com/v1.0
Memstore v2         https://auth.storage.memset.com/v2.0

When using Google Appengine you must provide the Connection with an appengine-specific Transport: 

import (
	"appengine/urlfetch"
	"fmt"
	"github.com/ncw/swift"
)

func handler(w http.ResponseWriter, r *http.Request) {
	ctx := appengine.NewContext(r)
	tr := urlfetch.Transport{Context: ctx}
	c := swift.Connection{
		UserName:  "user",
		ApiKey:    "key",
		AuthUrl:   "auth_url",
		Transport: tr,
	}
	_ := c.Authenticate()
	containers, _ := c.ContainerNames(nil)
	fmt.Fprintf(w, "containers: %q", containers)
}

If you don't supply a Transport, one is made which relies on http.ProxyFromEnvironment (http://golang.org/pkg/net/http/#ProxyFromEnvironment). This means that the connection will respect the HTTP proxy specified by the environment variables $HTTP_PROXY and $NO_PROXY.

Example 
package main

import (
	"fmt"

	"github.com/ncw/swift"
)

func main() {
	// Create a v1 auth connection
	c := &swift.Connection{
		// This should be your username
		UserName: "user",
		// This should be your api key
		ApiKey: "key",
		// This should be a v1 auth url, eg
		//  Rackspace US        https://auth.api.rackspacecloud.com/v1.0
		//  Rackspace UK        https://lon.auth.api.rackspacecloud.com/v1.0
		//  Memset Memstore UK  https://auth.storage.memset.com/v1.0
		AuthUrl: "auth_url",
	}

	// Authenticate
	err := c.Authenticate()
	if err != nil {
		panic(err)
	}
	// List all the containers
	containers, err := c.ContainerNames(nil)
	fmt.Println(containers)
	// etc...

	// ------ or alternatively create a v2 connection ------

	// Create a v2 auth connection
	c = &swift.Connection{
		// This is the sub user for the storage - eg "admin"
		UserName: "user",
		// This should be your api key
		ApiKey: "key",
		// This should be a version2 auth url, eg
		//  Rackspace v2        https://identity.api.rackspacecloud.com/v2.0
		//  Memset Memstore v2  https://auth.storage.memset.com/v2.0
		AuthUrl: "v2_auth_url",
		// Region to use - default is use first region if unset
		Region: "LON",
		// Name of the tenant - this is likely your username
		Tenant: "jim",
	}

	// as above...
}

Output:

func (*Connection) Account 

func (c *Connection) Account() (info Account, headers Headers, err error)

Account returns info about the account in an Account struct.

func (*Connection) AccountUpdate 

func (c *Connection) AccountUpdate(h Headers) error

AccountUpdate adds, replaces or remove account metadata.

Add or update keys by mentioning them in the Headers.

Remove keys by setting them to an empty string.

func (*Connection) ApplyEnvironment added in v1.0.34 

func (c *Connection) ApplyEnvironment() (err error)

ApplyEnvironment reads environment variables and applies them to the Connection structure. It won't overwrite any parameters which are already set in the Connection struct.

To make a new Connection object entirely from the environment you would do: 

c := new(Connection)
err := c.ApplyEnvironment()
if err != nil { log.Fatal(err) }

The naming of these variables follows the official Openstack naming scheme so it should be compatible with OpenStack rc files.

For v1 authentication (obsolete) 

ST_AUTH - Auth URL
ST_USER - UserName for api
ST_KEY - Key for api access

For v2 authentication 

OS_AUTH_URL - Auth URL
OS_USERNAME - UserName for api
OS_PASSWORD - Key for api access
OS_TENANT_NAME - Name of the tenant
OS_TENANT_ID   - Id of the tenant
OS_REGION_NAME - Region to use - default is use first region

For v3 authentication 

OS_AUTH_URL - Auth URL
OS_USERNAME - UserName for api
OS_USER_ID - User Id
OS_PASSWORD - Key for api access
OS_USER_DOMAIN_NAME - User's domain name
OS_USER_DOMAIN_ID - User's domain Id
OS_PROJECT_NAME - Name of the project
OS_PROJECT_DOMAIN_NAME - Name of the tenant's domain, only needed if it differs from the user domain
OS_PROJECT_DOMAIN_ID - Id of the tenant's domain, only needed if it differs the from user domain
OS_TRUST_ID - If of the trust
OS_REGION_NAME - Region to use - default is use first region

Other 

OS_ENDPOINT_TYPE - Endpoint type public, internal or admin
ST_AUTH_VERSION - Choose auth version - 1, 2 or 3 or leave at 0 for autodetect

For manual authentication 

OS_STORAGE_URL - storage URL from alternate authentication
OS_AUTH_TOKEN - Auth Token from alternate authentication

Library specific 

GOSWIFT_RETRIES - Retries on error (default is 3)
GOSWIFT_USER_AGENT - HTTP User agent (default goswift/1.0)
GOSWIFT_CONNECT_TIMEOUT - Connect channel timeout with unit, eg "10s", "100ms" (default "10s")
GOSWIFT_TIMEOUT - Data channel timeout with unit, eg "10s", "100ms" (default "60s")
GOSWIFT_INTERNAL - Set this to "true" to use the the internal network (obsolete - use OS_ENDPOINT_TYPE)

func (*Connection) Authenticate 

func (c *Connection) Authenticate() (err error)

Authenticate connects to the Swift server.

If you don't call it before calling one of the connection methods then it will be called for you on the first access.

func (*Connection) Authenticated 

func (c *Connection) Authenticated() bool

Authenticated returns a boolean to show if the current connection is authenticated.

Doesn't actually check the credentials against the server.

func (*Connection) BulkDelete added in v1.0.3 

func (c *Connection) BulkDelete(container string, objectNames []string) (result BulkDeleteResult, err error)

BulkDelete deletes multiple objectNames from container in one operation.

Some servers may not accept bulk-delete requests since bulk-delete is an optional feature of swift - these will return the Forbidden error.

See also: * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-bulk-delete.html * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Bulk_Delete-d1e2338.html

func (*Connection) BulkUpload added in v1.0.4 

func (c *Connection) BulkUpload(uploadPath string, dataStream io.Reader, format string, h Headers) (result BulkUploadResult, err error)

BulkUpload uploads multiple files in one operation.

uploadPath can be empty, a container name, or a pseudo-directory within a container. If uploadPath is empty, new containers may be automatically created.

Files are read from dataStream. The format of the stream is specified by the format parameter. Available formats are: * UploadTar - Plain tar stream. * UploadTarGzip - Gzip compressed tar stream. * UploadTarBzip2 - Bzip2 compressed tar stream.

Some servers may not accept bulk-upload requests since bulk-upload is an optional feature of swift - these will return the Forbidden error.

See also: * http://docs.openstack.org/trunk/openstack-object-storage/admin/content/object-storage-extract-archive.html * http://docs.rackspace.com/files/api/v1/cf-devguide/content/Extract_Archive-d1e2338.html

func (*Connection) Call added in v1.0.1 

func (c *Connection) Call(targetUrl string, p RequestOpts) (resp *http.Response, headers Headers, err error)

Call runs a remote command on the targetUrl, returns a response, headers and possible error.

operation is GET, HEAD etc container is the name of a container Any other parameters (if not None) are added to the targetUrl

Returns a response or an error. If response is returned then the resp.Body must be read completely and resp.Body.Close() must be called on it, unless noResponse is set in which case the body will be closed in this function

If "Content-Length" is set in p.Headers it will be used - this can be used to override the default chunked transfer encoding for uploads.

This will Authenticate if necessary, and re-authenticate if it receives a 401 error which means the token has expired

This method is exported so extensions can call it.

func (*Connection) Container 

func (c *Connection) Container(container string) (info Container, headers Headers, err error)

Container returns info about a single container including any metadata in the headers.

func (*Connection) ContainerCreate 

func (c *Connection) ContainerCreate(container string, h Headers) error

ContainerCreate creates a container.

If you don't want to add Headers just pass in nil

No error is returned if it already exists but the metadata if any will be updated.

func (*Connection) ContainerDelete 

func (c *Connection) ContainerDelete(container string) error

ContainerDelete deletes a container.

May return ContainerDoesNotExist or ContainerNotEmpty

func (*Connection) ContainerNames 

func (c *Connection) ContainerNames(opts *ContainersOpts) ([]string, error)

ContainerNames returns a slice of names of containers in this account.

func (*Connection) ContainerNamesAll 

func (c *Connection) ContainerNamesAll(opts *ContainersOpts) ([]string, error)

ContainerNamesAll is like ContainerNamess but it returns all the Containers

It calls ContainerNames multiple times using the Marker parameter

It has a default Limit parameter but you may pass in your own

func (*Connection) ContainerUpdate 

func (c *Connection) ContainerUpdate(container string, h Headers) error

ContainerUpdate adds, replaces or removes container metadata.

Add or update keys by mentioning them in the Metadata.

Remove keys by setting them to an empty string.

Container metadata can only be read with Container() not with Containers().

func (*Connection) Containers 

func (c *Connection) Containers(opts *ContainersOpts) ([]Container, error)

Containers returns a slice of structures with full information as described in Container.

func (*Connection) ContainersAll 

func (c *Connection) ContainersAll(opts *ContainersOpts) ([]Container, error)

ContainersAll is like Containers but it returns all the Containers

It calls Containers multiple times using the Marker parameter

It has a default Limit parameter but you may pass in your own

func (*Connection) DynamicLargeObjectCreate added in v1.0.26 

func (c *Connection) DynamicLargeObjectCreate(opts *LargeObjectOpts) (LargeObjectFile, error)

DynamicLargeObjectCreate creates or truncates an existing dynamic large object returning a writeable object. This sets opts.Flags to an appropriate value before calling DynamicLargeObjectCreateFile

func (*Connection) DynamicLargeObjectCreateFile added in v1.0.26 

func (c *Connection) DynamicLargeObjectCreateFile(opts *LargeObjectOpts) (LargeObjectFile, error)

DynamicLargeObjectCreateFile creates a dynamic large object returning an object which satisfies io.Writer, io.Seeker, io.Closer and io.ReaderFrom. The flags are as passes to the largeObjectCreate method.

func (*Connection) DynamicLargeObjectDelete added in v1.0.26 

func (c *Connection) DynamicLargeObjectDelete(container string, path string) error

DynamicLargeObjectDelete deletes a dynamic large object and all of its segments.

func (*Connection) DynamicLargeObjectMove added in v1.0.26 

func (c *Connection) DynamicLargeObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) error

DynamicLargeObjectMove moves a dynamic large object from srcContainer, srcObjectName to dstContainer, dstObjectName

func (*Connection) LargeObjectDelete added in v1.0.26 

func (c *Connection) LargeObjectDelete(container string, objectName string) error

LargeObjectDelete deletes the large object named by container, path

func (*Connection) LargeObjectGetSegments added in v1.0.26 

func (c *Connection) LargeObjectGetSegments(container string, path string) (string, []Object, error)

LargeObjectGetSegments returns all the segments that compose an object If the object is a Dynamic Large Object (DLO), it just returns the objects that have the prefix as indicated by the manifest. If the object is a Static Large Object (SLO), it retrieves the JSON content of the manifest and return all the segments of it.

func (*Connection) Object 

func (c *Connection) Object(container string, objectName string) (info Object, headers Headers, err error)

Object returns info about a single object including any metadata in the header.

May return ObjectNotFound.

Use headers.ObjectMetadata() to read the metadata in the Headers.

func (*Connection) ObjectCopy 

func (c *Connection) ObjectCopy(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string, h Headers) (headers Headers, err error)

ObjectCopy does a server side copy of an object to a new position

All metadata is preserved. If metadata is set in the headers then it overrides the old metadata on the copied object.

The destination container must exist before the copy.

You can use this to copy an object to itself - this is the only way to update the content type of an object.

func (*Connection) ObjectCreate 

func (c *Connection) ObjectCreate(container string, objectName string, checkHash bool, Hash string, contentType string, h Headers) (file *ObjectCreateFile, err error)

ObjectCreate creates or updates the object in the container. It returns an io.WriteCloser you should write the contents to. You MUST call Close() on it and you MUST check the error return from Close().

If checkHash is True then it will calculate the MD5 Hash of the file as it is being uploaded and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted on Close()

If you know the MD5 hash of the object ahead of time then set the Hash parameter and it will be sent to the server (as an Etag header) and the server will check the MD5 itself after the upload, and this will return ObjectCorrupted on Close() if it is incorrect.

If you don't want any error protection (not recommended) then set checkHash to false and Hash to "".

If contentType is set it will be used, otherwise one will be guessed from objectName using mime.TypeByExtension

func (*Connection) ObjectDelete 

func (c *Connection) ObjectDelete(container string, objectName string) error

ObjectDelete deletes the object.

May return ObjectNotFound if the object isn't found

func (*Connection) ObjectGet 

func (c *Connection) ObjectGet(container string, objectName string, contents io.Writer, checkHash bool, h Headers) (headers Headers, err error)

ObjectGet gets the object into the io.Writer contents.

Returns the headers of the response.

If checkHash is true then it will calculate the md5sum of the file as it is being received and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted.

headers["Content-Type"] will give the content type if desired.

func (*Connection) ObjectGetBytes 

func (c *Connection) ObjectGetBytes(container string, objectName string) (contents []byte, err error)

ObjectGetBytes returns an object as a []byte.

This is a simplified interface which checks the MD5

func (*Connection) ObjectGetString 

func (c *Connection) ObjectGetString(container string, objectName string) (contents string, err error)

ObjectGetString returns an object as a string.

This is a simplified interface which checks the MD5

func (*Connection) ObjectMove 

func (c *Connection) ObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) (err error)

ObjectMove does a server side move of an object to a new position

This is a convenience method which calls ObjectCopy then ObjectDelete

All metadata is preserved.

The destination container must exist before the copy.

func (*Connection) ObjectNames 

func (c *Connection) ObjectNames(container string, opts *ObjectsOpts) ([]string, error)

ObjectNames returns a slice of names of objects in a given container.

func (*Connection) ObjectNamesAll 

func (c *Connection) ObjectNamesAll(container string, opts *ObjectsOpts) ([]string, error)

ObjectNamesAll is like ObjectNames but it returns all the Objects

It calls ObjectNames multiple times using the Marker parameter

It has a default Limit parameter but you may pass in your own

func (*Connection) ObjectOpen 

func (c *Connection) ObjectOpen(container string, objectName string, checkHash bool, h Headers) (file *ObjectOpenFile, headers Headers, err error)

ObjectOpen returns an ObjectOpenFile for reading the contents of the object. This satisfies the io.ReadCloser and the io.Seeker interfaces.

You must call Close() on contents when finished

Returns the headers of the response.

If checkHash is true then it will calculate the md5sum of the file as it is being received and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted. It will also check the length returned. No checking will be done if you don't read all the contents.

Note that objects with X-Object-Manifest or X-Static-Large-Object set won't ever have their md5sum's checked as the md5sum reported on the object is actually the md5sum of the md5sums of the parts. This isn't very helpful to detect a corrupted download as the size of the parts aren't known without doing more operations. If you want to ensure integrity of an object with a manifest then you will need to download everything in the manifest separately.

headers["Content-Type"] will give the content type if desired.

func (*Connection) ObjectPut 

func (c *Connection) ObjectPut(container string, objectName string, contents io.Reader, checkHash bool, Hash string, contentType string, h Headers) (headers Headers, err error)

ObjectPut creates or updates the path in the container from contents. contents should be an open io.Reader which will have all its contents read.

This is a low level interface.

If checkHash is True then it will calculate the MD5 Hash of the file as it is being uploaded and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted.

If you know the MD5 hash of the object ahead of time then set the Hash parameter and it will be sent to the server (as an Etag header) and the server will check the MD5 itself after the upload, and this will return ObjectCorrupted if it is incorrect.

If you don't want any error protection (not recommended) then set checkHash to false and Hash to "".

If contentType is set it will be used, otherwise one will be guessed from objectName using mime.TypeByExtension

func (*Connection) ObjectPutBytes 

func (c *Connection) ObjectPutBytes(container string, objectName string, contents []byte, contentType string) (err error)

ObjectPutBytes creates an object from a []byte in a container.

This is a simplified interface which checks the MD5.

func (*Connection) ObjectPutString 

func (c *Connection) ObjectPutString(container string, objectName string, contents string, contentType string) (err error)

ObjectPutString creates an object from a string in a container.

This is a simplified interface which checks the MD5

func (*Connection) ObjectTempUrl added in v1.0.16 

func (c *Connection) ObjectTempUrl(container string, objectName string, secretKey string, method string, expires time.Time) string

ObjectTempUrl returns a temporary URL for an object

func (*Connection) ObjectUpdate 

func (c *Connection) ObjectUpdate(container string, objectName string, h Headers) error

ObjectUpdate adds, replaces or removes object metadata.

Add or Update keys by mentioning them in the Metadata. Use Metadata.ObjectHeaders and Headers.ObjectMetadata to convert your Metadata to and from normal HTTP headers.

This removes all metadata previously added to the object and replaces it with that passed in so to delete keys, just don't mention them the headers you pass in.

Object metadata can only be read with Object() not with Objects().

This can also be used to set headers not already assigned such as X-Delete-At or X-Delete-After for expiring objects.

You cannot use this to change any of the object's other headers such as Content-Type, ETag, etc.

Refer to copying an object when you need to update metadata or other headers such as Content-Type or CORS headers.

May return ObjectNotFound.

func (*Connection) ObjectUpdateContentType 

func (c *Connection) ObjectUpdateContentType(container string, objectName string, contentType string) (err error)

ObjectUpdateContentType updates the content type of an object

This is a convenience method which calls ObjectCopy

All other metadata is preserved.

func (*Connection) Objects 

func (c *Connection) Objects(container string, opts *ObjectsOpts) ([]Object, error)

Objects returns a slice of Object with information about each object in the container.

If Delimiter is set in the opts then PseudoDirectory may be set, with ContentType 'application/directory'. These are not real objects but represent directories of objects which haven't had an object created for them.

func (*Connection) ObjectsAll 

func (c *Connection) ObjectsAll(container string, opts *ObjectsOpts) ([]Object, error)

ObjectsAll is like Objects but it returns an unlimited number of Objects in a slice

It calls Objects multiple times using the Marker parameter

func (*Connection) ObjectsWalk 

func (c *Connection) ObjectsWalk(container string, opts *ObjectsOpts, walkFn ObjectsWalkFn) error

ObjectsWalk is uses to iterate through all the objects in chunks as returned by Objects or ObjectNames using the Marker and Limit parameters in the ObjectsOpts.

Pass in a closure `walkFn` which calls Objects or ObjectNames with the *ObjectsOpts passed to it and does something with the results.

Errors will be returned from this function

It has a default Limit parameter but you may pass in your own

Example 
c, rollback := makeConnection(nil)
defer rollback()

objects := make([]string, 0)
err := c.ObjectsWalk(container, nil, func(opts *swift.ObjectsOpts) (interface{}, error) {
	newObjects, err := c.ObjectNames(container, opts)
	if err == nil {
		objects = append(objects, newObjects...)
	}
	return newObjects, err
})
fmt.Println("Found all the objects", objects, err)

Output:

func (*Connection) QueryInfo added in v1.0.19 

func (c *Connection) QueryInfo() (infos SwiftInfo, err error)

Discover Swift configuration by doing a request against /info

func (*Connection) StaticLargeObjectCreate added in v1.0.26 

func (c *Connection) StaticLargeObjectCreate(opts *LargeObjectOpts) (LargeObjectFile, error)

StaticLargeObjectCreate creates or truncates an existing static large object returning a writeable object. This sets opts.Flags to an appropriate value before calling StaticLargeObjectCreateFile

func (*Connection) StaticLargeObjectCreateFile added in v1.0.26 

func (c *Connection) StaticLargeObjectCreateFile(opts *LargeObjectOpts) (LargeObjectFile, error)

StaticLargeObjectCreateFile creates a static large object returning an object which satisfies io.Writer, io.Seeker, io.Closer and io.ReaderFrom. The flags are as passed to the largeObjectCreate method.

func (*Connection) StaticLargeObjectDelete added in v1.0.26 

func (c *Connection) StaticLargeObjectDelete(container string, path string) error

StaticLargeObjectDelete deletes a static large object and all of its segments.

func (*Connection) StaticLargeObjectMove added in v1.0.26 

func (c *Connection) StaticLargeObjectMove(srcContainer string, srcObjectName string, dstContainer string, dstObjectName string) error

StaticLargeObjectMove moves a static large object from srcContainer, srcObjectName to dstContainer, dstObjectName

func (*Connection) UnAuthenticate 

func (c *Connection) UnAuthenticate()

UnAuthenticate removes the authentication from the Connection.

func (*Connection) VersionContainerCreate added in v1.0.1 

func (c *Connection) VersionContainerCreate(current, version string) error

VersionContainerCreate is a helper method for creating and enabling version controlled containers.

It builds the current object container, the non-current object version container, and enables versioning.

If the server doesn't support versioning then it will return Forbidden however it will have created both the containers at that point.

Example 
c, rollback := makeConnection(nil)
defer rollback()

// Use the helper method to create the current and versions container.
if err := c.VersionContainerCreate("cds", "cd-versions"); err != nil {
	fmt.Print(err.Error())
}

Output:

func (*Connection) VersionDisable added in v1.0.1 

func (c *Connection) VersionDisable(current string) error

VersionDisable disables versioning on the current container.

Example 
c, rollback := makeConnection(nil)
defer rollback()

// Disable versioning on a container.  Note that this does not delete the versioning container.
c.VersionDisable("movies")

Output:

func (*Connection) VersionEnable added in v1.0.1 

func (c *Connection) VersionEnable(current, version string) error

VersionEnable enables versioning on the current container with version as the tracking container.

May return Forbidden if this isn't supported by the server

Example 
c, rollback := makeConnection(nil)
defer rollback()

// Build the containers manually and enable them.
if err := c.ContainerCreate("movie-versions", nil); err != nil {
	fmt.Print(err.Error())
}
if err := c.ContainerCreate("movies", nil); err != nil {
	fmt.Print(err.Error())
}
if err := c.VersionEnable("movies", "movie-versions"); err != nil {
	fmt.Print(err.Error())
}

// Access the primary container as usual with ObjectCreate(), ObjectPut(), etc.
// etc...

Output:

func (*Connection) VersionObjectList added in v1.0.1 

func (c *Connection) VersionObjectList(version, object string) ([]string, error)

VersionObjectList returns a list of older versions of the object.

Objects are returned in the format <length><object_name>/<timestamp>

type Container 

type Container struct {
	Name  string // Name of the container
	Count int64  // Number of objects in the container
	Bytes int64  // Total number of bytes used in the container
}

Container contains information about a container

type ContainersOpts 

type ContainersOpts struct {
	Limit     int     // For an integer value n, limits the number of results to at most n values.
	Prefix    string  // Given a string value x, return container names matching the specified prefix.
	Marker    string  // Given a string value x, return container names greater in value than the specified marker.
	EndMarker string  // Given a string value x, return container names less in value than the specified marker.
	Headers   Headers // Any additional HTTP headers - can be nil
}

ContainersOpts is options for Containers() and ContainerNames()

type CustomEndpointAuthenticator added in v1.0.24 

type CustomEndpointAuthenticator interface {
	StorageUrlForEndpoint(endpointType EndpointType) string
}

type DynamicLargeObjectCreateFile added in v1.0.26 

type DynamicLargeObjectCreateFile struct {
	// contains filtered or unexported fields
}

DynamicLargeObjectCreateFile represents an open static large object

func (*DynamicLargeObjectCreateFile) Close added in v1.0.26 

func (file *DynamicLargeObjectCreateFile) Close() error

Close satisfies the io.Closer interface

func (*DynamicLargeObjectCreateFile) Flush added in v1.0.27 

func (file *DynamicLargeObjectCreateFile) Flush() error

func (*DynamicLargeObjectCreateFile) Seek added in v1.0.27 

func (file *DynamicLargeObjectCreateFile) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next write operation

func (*DynamicLargeObjectCreateFile) Size added in v1.0.27 

func (file *DynamicLargeObjectCreateFile) Size() int64

func (*DynamicLargeObjectCreateFile) Write added in v1.0.26 

func (file *DynamicLargeObjectCreateFile) Write(buf []byte) (int, error)

Write satisfies the io.Writer interface

type EndpointType added in v1.0.24 

type EndpointType string

type Error 

type Error struct {
	StatusCode int // HTTP status code if relevant or 0 if not
	Text       string
}

Error - all errors generated by this package are of this type. Other error may be passed on from library functions though.

func (*Error) Error 

func (e *Error) Error() string

Error satisfy the error interface.

type Headers 

type Headers map[string]string

Headers stores HTTP headers (can only have one of each header like Swift).

func (Headers) AccountMetadata 

func (h Headers) AccountMetadata() Metadata

AccountMetadata converts Headers from account to a Metadata.

The keys in the Metadata will be converted to lower case.

func (Headers) ContainerMetadata 

func (h Headers) ContainerMetadata() Metadata

ContainerMetadata converts Headers from container to a Metadata.

The keys in the Metadata will be converted to lower case.

func (Headers) IsLargeObject added in v1.0.27 

func (headers Headers) IsLargeObject() bool

func (Headers) IsLargeObjectDLO added in v1.0.27 

func (headers Headers) IsLargeObjectDLO() bool

func (Headers) IsLargeObjectSLO added in v1.0.27 

func (headers Headers) IsLargeObjectSLO() bool

func (Headers) Metadata 

func (h Headers) Metadata(metaPrefix string) Metadata

Metadata gets the Metadata starting with the metaPrefix out of the Headers.

The keys in the Metadata will be converted to lower case

func (Headers) ObjectMetadata 

func (h Headers) ObjectMetadata() Metadata

ObjectMetadata converts Headers from object to a Metadata.

The keys in the Metadata will be converted to lower case.

type LargeObjectFile added in v1.0.27 

type LargeObjectFile interface {
	io.Writer
	io.Seeker
	io.Closer
	Size() int64
	Flush() error
}

type LargeObjectOpts added in v1.0.26 

type LargeObjectOpts struct {
	Container        string  // Name of container to place object
	ObjectName       string  // Name of object
	Flags            int     // Creation flags
	CheckHash        bool    // If set Check the hash
	Hash             string  // If set use this hash to check
	ContentType      string  // Content-Type of the object
	Headers          Headers // Additional headers to upload the object with
	ChunkSize        int64   // Size of chunks of the object, defaults to 10MB if not set
	MinChunkSize     int64   // Minimum chunk size, automatically set for SLO's based on info
	SegmentContainer string  // Name of the container to place segments
	SegmentPrefix    string  // Prefix to use for the segments
	NoBuffer         bool    // Prevents using a bufio.Writer to write segments
}

LargeObjectOpts describes how a large object should be created

type Metadata 

type Metadata map[string]string

Metadata stores account, container or object metadata.

func (Metadata) AccountHeaders 

func (m Metadata) AccountHeaders() Headers

AccountHeaders converts the Metadata for the account.

func (Metadata) ContainerHeaders 

func (m Metadata) ContainerHeaders() Headers

ContainerHeaders converts the Metadata for the container.

func (Metadata) GetModTime 

func (m Metadata) GetModTime() (t time.Time, err error)

Read a modification time (mtime) from a Metadata object

This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time (as read using os.Stat) for an object. It is stored using the key 'mtime', which for example when written to an object will be 'X-Object-Meta-Mtime'.

If an error is returned then time will be returned as the zero time.

func (Metadata) Headers 

func (m Metadata) Headers(metaPrefix string) Headers

Headers convert the Metadata starting with the metaPrefix into a Headers.

The keys in the Metadata will be converted from lower case to http Canonical (see http.CanonicalHeaderKey).

func (Metadata) ObjectHeaders 

func (m Metadata) ObjectHeaders() Headers

ObjectHeaders converts the Metadata for the object.

func (Metadata) SetModTime 

func (m Metadata) SetModTime(t time.Time)

Write an modification time (mtime) to a Metadata object

This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time (as read using os.Stat) for an object. It is stored using the key 'mtime', which for example when written to an object will be 'X-Object-Meta-Mtime'.

type Object 

type Object struct {
	Name               string     `json:"name"`          // object name
	ContentType        string     `json:"content_type"`  // eg application/directory
	Bytes              int64      `json:"bytes"`         // size in bytes
	ServerLastModified string     `json:"last_modified"` // Last modified time, eg '2011-06-30T08:20:47.736680' as a string supplied by the server
	LastModified       time.Time  // Last modified time converted to a time.Time
	Hash               string     `json:"hash"` // MD5 hash, eg "d41d8cd98f00b204e9800998ecf8427e"
	PseudoDirectory    bool       // Set when using delimiter to show that this directory object does not really exist
	SubDir             string     `json:"subdir"` // returned only when using delimiter to mark "pseudo directories"
	ObjectType         ObjectType // type of this object
}

Object contains information about an object

type ObjectCreateFile 

type ObjectCreateFile struct {
	// contains filtered or unexported fields
}

ObjectCreateFile represents a swift object open for writing

func (*ObjectCreateFile) Close 

func (file *ObjectCreateFile) Close() error

Close the object and checks the md5sum if it was required.

Also returns any other errors from the server (eg container not found) so it is very important to check the errors on this method.

func (*ObjectCreateFile) Headers added in v1.0.29 

func (file *ObjectCreateFile) Headers() (Headers, error)

Headers returns the response headers from the created object if the upload has been completed. The Close() method must be called on an ObjectCreateFile before this method.

func (*ObjectCreateFile) Write 

func (file *ObjectCreateFile) Write(p []byte) (n int, err error)

Write bytes to the object - see io.Writer

type ObjectOpenFile 

type ObjectOpenFile struct {
	// contains filtered or unexported fields
}

ObjectOpenFile represents a swift object open for reading

func (*ObjectOpenFile) Close 

func (file *ObjectOpenFile) Close() (err error)

Close the object and checks the length and md5sum if it was required and all the object was read

func (*ObjectOpenFile) Length added in v1.0.5 

func (file *ObjectOpenFile) Length() (int64, error)

Length gets the objects content length either from a cached copy or from the server.

func (*ObjectOpenFile) Read 

func (file *ObjectOpenFile) Read(p []byte) (n int, err error)

Read bytes from the object - see io.Reader

func (*ObjectOpenFile) Seek 

func (file *ObjectOpenFile) Seek(offset int64, whence int) (newPos int64, err error)

Seek sets the offset for the next Read to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. Seek returns the new offset and an Error, if any.

Seek uses HTTP Range headers which, if the file pointer is moved, will involve reopening the HTTP connection.

Note that you can't seek to the end of a file or beyond; HTTP Range requests don't support the file pointer being outside the data, unlike os.File

Seek(0, 1) will return the current file pointer.

type ObjectType added in v1.0.26 

type ObjectType int

ObjectType is the type of the swift object, regular, static large, or dynamic large. 

const (
	RegularObjectType ObjectType = iota
	StaticLargeObjectType
	DynamicLargeObjectType
)

Values that ObjectType can take

type ObjectsOpts 

type ObjectsOpts struct {
	Limit     int     // For an integer value n, limits the number of results to at most n values.
	Marker    string  // Given a string value x, return object names greater in value than the  specified marker.
	EndMarker string  // Given a string value x, return object names less in value than the specified marker
	Prefix    string  // For a string value x, causes the results to be limited to object names beginning with the substring x.
	Path      string  // For a string value x, return the object names nested in the pseudo path
	Delimiter rune    // For a character c, return all the object names nested in the container
	Headers   Headers // Any additional HTTP headers - can be nil
}

ObjectOpts is options for Objects() and ObjectNames()

type ObjectsWalkFn 

type ObjectsWalkFn func(*ObjectsOpts) (interface{}, error)

A closure defined by the caller to iterate through all objects

Call Objects or ObjectNames from here with the *ObjectOpts passed in

Do whatever is required with the results then return them

type RequestOpts added in v1.0.1 

type RequestOpts struct {
	Container  string
	ObjectName string
	Operation  string
	Parameters url.Values
	Headers    Headers
	ErrorMap   errorMap
	NoResponse bool
	Body       io.Reader
	Retries    int
	// if set this is called on re-authentication to refresh the targetUrl
	OnReAuth func() (string, error)
}

RequestOpts contains parameters for Connection.storage.

type StaticLargeObjectCreateFile added in v1.0.26 

type StaticLargeObjectCreateFile struct {
	// contains filtered or unexported fields
}

StaticLargeObjectCreateFile represents an open static large object

func (*StaticLargeObjectCreateFile) Close added in v1.0.26 

func (file *StaticLargeObjectCreateFile) Close() error

func (*StaticLargeObjectCreateFile) Flush added in v1.0.27 

func (file *StaticLargeObjectCreateFile) Flush() error

func (*StaticLargeObjectCreateFile) Seek added in v1.0.27 

func (file *StaticLargeObjectCreateFile) Seek(offset int64, whence int) (int64, error)

Seek sets the offset for the next write operation

func (*StaticLargeObjectCreateFile) Size added in v1.0.27 

func (file *StaticLargeObjectCreateFile) Size() int64

func (*StaticLargeObjectCreateFile) Write added in v1.0.26 

func (file *StaticLargeObjectCreateFile) Write(buf []byte) (int, error)

Write satisfies the io.Writer interface

type SwiftInfo added in v1.0.19 

type SwiftInfo map[string]interface{}

SwiftInfo contains the JSON object returned by Swift when the /info route is queried. The object contains, among others, the Swift version, the enabled middlewares and their configuration

func (SwiftInfo) SLOMinSegmentSize added in v1.0.27 

func (i SwiftInfo) SLOMinSegmentSize() int64

func (SwiftInfo) SupportsBulkDelete added in v1.0.27 

func (i SwiftInfo) SupportsBulkDelete() bool

func (SwiftInfo) SupportsSLO added in v1.0.27 

func (i SwiftInfo) SupportsSLO() bool

Source Files

View all Source files

Directories

Path Synopsis
This implements a very basic Swift server Everything is stored in memory This comes from the https://github.com/mitchellh/goamz and was adapted for Swift
 This implements a very basic Swift server Everything is stored in memory This comes from the https://github.com/mitchellh/goamz and was adapted for Swift

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.