s3backupstorage

README

Recently added options to enable usage of an S3 appliance: Cloudian
HyperStore:
        --s3_backup_aws_endpoint <host:port> (port is required)
        --s3_backup_force_path_style=true/false
        --s3_backup_log_level <level> can be one of: LogOff, LogDebug, LogDebugWithSigning, LogDebugWithHTTPBody, LogDebugWithRequestRetries, LogDebugWithRequestErrors.  Default: LogOff

By default the s3 client will try to connect to
<path>.<region>.amazonaws.com.  Adjusting the endpoint will allow this
to be changed.

Given the way the FQDN is configured the TLS certificate may not match the
server's "base" (<region>.<end_point_address>) due to the leading <path>
so setting --s3_backup_force_path_style=true will force the s3 client to
connect to <region>.<endpoint> and then make a request using the full
path within the http calls.

--s3backup_log_level enables more verbose logging of the S3 calls.

Documentation

Overview

Package s3backupstorage implements the BackupStorage interface for AWS S3.

AWS access credentials are configured via standard AWS means, such as: - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables - credentials file at ~/.aws/credentials - if running on an EC2 instance, an IAM role See details at http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs

Index

• type ClosedConnectionRetryer
  • func (retryer *ClosedConnectionRetryer) GetInitialToken() (releaseToken func(error) error)
  • func (retryer *ClosedConnectionRetryer) GetRetryToken(ctx context.Context, opErr error) (releaseToken func(error) error, err error)
  • func (retryer *ClosedConnectionRetryer) IsErrorRetryable(err error) bool
  • func (retryer *ClosedConnectionRetryer) MaxAttempts() int
  • func (retryer *ClosedConnectionRetryer) RetryDelay(attempt int, opErr error) (time.Duration, error)
• type S3BackupHandle
  • func (bh *S3BackupHandle) AbortBackup(ctx context.Context) error
  • func (bh *S3BackupHandle) AddFile(ctx context.Context, filename string, filesize int64) (io.WriteCloser, error)
  • func (bh *S3BackupHandle) Directory() string
  • func (bh *S3BackupHandle) EndBackup(ctx context.Context) error
  • func (bh *S3BackupHandle) Error() error
  • func (bh *S3BackupHandle) HasErrors() bool
  • func (bh *S3BackupHandle) Name() string
  • func (bh *S3BackupHandle) ReadFile(ctx context.Context, filename string) (io.ReadCloser, error)
  • func (bh *S3BackupHandle) RecordError(err error)
• type S3BackupStorage
  • func (bs *S3BackupStorage) Close() error
  • func (bs *S3BackupStorage) ListBackups(ctx context.Context, dir string) ([]backupstorage.BackupHandle, error)
  • func (bs *S3BackupStorage) RemoveBackup(ctx context.Context, dir, name string) error
  • func (bs *S3BackupStorage) StartBackup(ctx context.Context, dir, name string) (backupstorage.BackupHandle, error)
  • func (bs *S3BackupStorage) WithParams(params backupstorage.Params) backupstorage.BackupStorage
• type S3ServerSideEncryption

Types

type ClosedConnectionRetryer

type ClosedConnectionRetryer struct {
	// contains filtered or unexported fields
}

ClosedConnectionRetryer implements the aws.Retryer interface and is used to retry closed connection errors during MultipartUpload operations. It is a simplified version of the RetryableConnectionError implementation, which always retry on any type of connection error.

func (*ClosedConnectionRetryer) GetInitialToken

func (retryer *ClosedConnectionRetryer) GetInitialToken() (releaseToken func(error) error)

GetInitialToken returns the initial attempt token that can increment the retry token pool if the attempt is successful.

func (*ClosedConnectionRetryer) GetRetryToken

func (retryer *ClosedConnectionRetryer) GetRetryToken(ctx context.Context, opErr error) (releaseToken func(error) error, err error)

GetRetryToken attempts to deduct the retry cost from the retry token pool. Returning the token release function, or error.

func (*ClosedConnectionRetryer) IsErrorRetryable

func (retryer *ClosedConnectionRetryer) IsErrorRetryable(err error) bool

IsErrorRetryable returns true if the error should be retried. We first try to see if the error is due to the use of a closed connection, if it is, we retry, and if not, we default to what the aws.Retryer would do.

func (*ClosedConnectionRetryer) MaxAttempts

func (retryer *ClosedConnectionRetryer) MaxAttempts() int

MaxAttempts returns the maximum number of attempts that can be made for an attempt before failing. A value of 0 implies that the attempt should be retried until it succeeds if the errors are retryable.

func (*ClosedConnectionRetryer) RetryDelay

func (retryer *ClosedConnectionRetryer) RetryDelay(attempt int, opErr error) (time.Duration, error)

RetryDelay returns the delay that should be used before retrying the attempt. Will return error if the delay could not be determined.

type S3BackupHandle

type S3BackupHandle struct {
	// contains filtered or unexported fields
}

S3BackupHandle implements the backupstorage.BackupHandle interface.

func (*S3BackupHandle) AbortBackup

func (bh *S3BackupHandle) AbortBackup(ctx context.Context) error

AbortBackup is part of the backupstorage.BackupHandle interface.

func (*S3BackupHandle) AddFile

func (bh *S3BackupHandle) AddFile(ctx context.Context, filename string, filesize int64) (io.WriteCloser, error)

AddFile is part of the backupstorage.BackupHandle interface.

func (*S3BackupHandle) Directory

func (bh *S3BackupHandle) Directory() string

Directory is part of the backupstorage.BackupHandle interface.

func (*S3BackupHandle) EndBackup

func (bh *S3BackupHandle) EndBackup(ctx context.Context) error

EndBackup is part of the backupstorage.BackupHandle interface.

func (*S3BackupHandle) Error

func (bh *S3BackupHandle) Error() error

Error is part of the concurrency.ErrorRecorder interface.

func (*S3BackupHandle) HasErrors

func (bh *S3BackupHandle) HasErrors() bool

HasErrors is part of the concurrency.ErrorRecorder interface.

func (*S3BackupHandle) Name

func (bh *S3BackupHandle) Name() string

Name is part of the backupstorage.BackupHandle interface.

func (*S3BackupHandle) ReadFile

func (bh *S3BackupHandle) ReadFile(ctx context.Context, filename string) (io.ReadCloser, error)

ReadFile is part of the backupstorage.BackupHandle interface.

func (*S3BackupHandle) RecordError

func (bh *S3BackupHandle) RecordError(err error)

RecordError is part of the concurrency.ErrorRecorder interface.

type S3BackupStorage

type S3BackupStorage struct {
	// contains filtered or unexported fields
}

S3BackupStorage implements the backupstorage.BackupStorage interface.

func (*S3BackupStorage) Close

func (bs *S3BackupStorage) Close() error

Close is part of the backupstorage.BackupStorage interface.

func (*S3BackupStorage) ListBackups

func (bs *S3BackupStorage) ListBackups(ctx context.Context, dir string) ([]backupstorage.BackupHandle, error)

ListBackups is part of the backupstorage.BackupStorage interface.

func (*S3BackupStorage) RemoveBackup

func (bs *S3BackupStorage) RemoveBackup(ctx context.Context, dir, name string) error

RemoveBackup is part of the backupstorage.BackupStorage interface.

func (*S3BackupStorage) StartBackup

func (bs *S3BackupStorage) StartBackup(ctx context.Context, dir, name string) (backupstorage.BackupHandle, error)

StartBackup is part of the backupstorage.BackupStorage interface.

func (*S3BackupStorage) WithParams

func (bs *S3BackupStorage) WithParams(params backupstorage.Params) backupstorage.BackupStorage

type S3ServerSideEncryption

type S3ServerSideEncryption struct {
	// contains filtered or unexported fields
}