obj

package
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 13, 2019 License: Apache-2.0 Imports: 43 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	StorageBackendEnvVar = "STORAGE_BACKEND"
	PachRootEnvVar       = "PACH_ROOT"
)

Environment variables for determining storage backend and pathing

View Source 
const (
	Minio     = "MINIO"
	Amazon    = "AMAZON"
	Google    = "GOOGLE"
	Microsoft = "MICROSOFT"
	Local     = "LOCAL"
)

Valid object storage backends

View Source 
const (
	GoogleBucketEnvVar = "GOOGLE_BUCKET"
	GoogleCredEnvVar   = "GOOGLE_CRED"
)

Google environment variables

View Source 
const (
	MicrosoftContainerEnvVar = "MICROSOFT_CONTAINER"
	MicrosoftIDEnvVar        = "MICROSOFT_ID"
	MicrosoftSecretEnvVar    = "MICROSOFT_SECRET"
)

Microsoft environment variables

View Source 
const (
	MinioBucketEnvVar    = "MINIO_BUCKET"
	MinioEndpointEnvVar  = "MINIO_ENDPOINT"
	MinioIDEnvVar        = "MINIO_ID"
	MinioSecretEnvVar    = "MINIO_SECRET"
	MinioSecureEnvVar    = "MINIO_SECURE"
	MinioSignatureEnvVar = "MINIO_SIGNATURE"
)

Minio environment variables

View Source 
const (
	AmazonRegionEnvVar       = "AMAZON_REGION"
	AmazonBucketEnvVar       = "AMAZON_BUCKET"
	AmazonIDEnvVar           = "AMAZON_ID"
	AmazonSecretEnvVar       = "AMAZON_SECRET"
	AmazonTokenEnvVar        = "AMAZON_TOKEN"
	AmazonVaultAddrEnvVar    = "AMAZON_VAULT_ADDR"
	AmazonVaultRoleEnvVar    = "AMAZON_VAULT_ROLE"
	AmazonVaultTokenEnvVar   = "AMAZON_VAULT_TOKEN"
	AmazonDistributionEnvVar = "AMAZON_DISTRIBUTION"
	CustomEndpointEnvVar     = "CUSTOM_ENDPOINT"
)

Amazon environment variables

Variables

View Source 
var EnvVarToSecretKey = map[string]string{
	GoogleBucketEnvVar:       "google-bucket",
	GoogleCredEnvVar:         "google-cred",
	MicrosoftContainerEnvVar: "microsoft-container",
	MicrosoftIDEnvVar:        "microsoft-id",
	MicrosoftSecretEnvVar:    "microsoft-secret",
	MinioBucketEnvVar:        "minio-bucket",
	MinioEndpointEnvVar:      "minio-endpoint",
	MinioIDEnvVar:            "minio-id",
	MinioSecretEnvVar:        "minio-secret",
	MinioSecureEnvVar:        "minio-secure",
	MinioSignatureEnvVar:     "minio-signature",
	AmazonRegionEnvVar:       "amazon-region",
	AmazonBucketEnvVar:       "amazon-bucket",
	AmazonIDEnvVar:           "amazon-id",
	AmazonSecretEnvVar:       "amazon-secret",
	AmazonTokenEnvVar:        "amazon-token",
	AmazonVaultAddrEnvVar:    "amazon-vault-addr",
	AmazonVaultRoleEnvVar:    "amazon-vault-role",
	AmazonVaultTokenEnvVar:   "amazon-vault-token",
	AmazonDistributionEnvVar: "amazon-distribution",
	CustomEndpointEnvVar:     "custom-endpoint",
}

EnvVarToSecretKey is an environment variable name to secret key mapping This is being used to temporarily bridge the gap as we transition to a model where object storage access in the workers is based on environment variables and a library rather than mounting a secret to a sidecar container which accesses object storage

Functions

func BlockPathFromEnv added in v1.8.0 

func BlockPathFromEnv(block *pfs.Block) (string, error)

BlockPathFromEnv gets the path to an object storage block based on environment variables.

func DisableMonkeyTest added in v1.9.0 

func DisableMonkeyTest()

DisableMonkeyTest disables sporadic request failures.

func EnableMonkeyTest added in v1.9.0 

func EnableMonkeyTest()

EnableMonkeyTest enables sporadic request failures.

func InitMonkeyTest added in v1.9.0 

func InitMonkeyTest(seed int64)

InitMonkeyTest sets up this package for monkey testing. Object storage clients will be wrapped with a client that sporadically fails requests.

func IsMonkeyError added in v1.9.0 

func IsMonkeyError(err error) bool

IsMonkeyError checks if an error was caused by a monkey client.

func IsRetryable added in v1.3.14 

func IsRetryable(client Client, err error) bool

IsRetryable determines if an operation should be retried given an error

func NewExponentialBackOffConfig added in v1.1.0 

func NewExponentialBackOffConfig() *backoff.ExponentialBackOff

NewExponentialBackOffConfig creates an exponential back-off config with longer wait times than the default.

func StorageRootFromEnv added in v1.8.0 

func StorageRootFromEnv() (string, error)

StorageRootFromEnv gets the storage root based on environment variables.

func TestStorage added in v1.8.7 

func TestStorage(ctx context.Context, c Client) error

TestStorage is a defensive method for checking to make sure that storage is properly configured.

Types

type AmazonCreds added in v1.7.1 

type AmazonCreds struct {
	// Direct credentials. Only applicable if Pachyderm is given its own permanent
	// AWS credentials
	ID     string // Access Key ID
	Secret string // Secret Access Key
	Token  string // Access token (if using temporary security credentials

	// Vault options (if getting AWS credentials from Vault)
	VaultAddress string // normally addresses come from env, but don't have vault service name
	VaultRole    string
	VaultToken   string
}

AmazonCreds are options that are applicable specifically to Pachd's credentials in an AWS deployment

type BackoffReadCloser added in v1.1.0 

type BackoffReadCloser struct {
	// contains filtered or unexported fields
}

BackoffReadCloser retries with exponential backoff in the case of failures

func (*BackoffReadCloser) Close added in v1.1.0 

func (b *BackoffReadCloser) Close() error

Close closes the ReaderCloser contained in b.

func (*BackoffReadCloser) Read added in v1.1.0 

func (b *BackoffReadCloser) Read(data []byte) (int, error)

type BackoffWriteCloser added in v1.1.0 

type BackoffWriteCloser struct {
	// contains filtered or unexported fields
}

BackoffWriteCloser retries with exponential backoff in the case of failures

func (*BackoffWriteCloser) Close added in v1.1.0 

func (b *BackoffWriteCloser) Close() error

Close closes the WriteCloser contained in b.

func (*BackoffWriteCloser) Write added in v1.1.0 

func (b *BackoffWriteCloser) Write(data []byte) (int, error)

type Client 

type Client interface {
	// Writer returns a writer which writes to an object.
	// It should error if the object already exists or we don't have sufficient
	// permissions to write it.
	Writer(ctx context.Context, name string) (io.WriteCloser, error)
	// Reader returns a reader which reads from an object.
	// If `size == 0`, the reader should read from the offset till the end of the object.
	// It should error if the object doesn't exist or we don't have sufficient
	// permission to read it.
	Reader(ctx context.Context, name string, offset uint64, size uint64) (io.ReadCloser, error)
	// Delete deletes an object.
	// It should error if the object doesn't exist or we don't have sufficient
	// permission to delete it.
	Delete(ctx context.Context, name string) error
	// Walk calls `fn` with the names of objects which can be found under `prefix`.
	Walk(ctx context.Context, prefix string, fn func(name string) error) error
	// Exsits checks if a given object already exists
	Exists(ctx context.Context, name string) bool
	// IsRetryable determines if an operation should be retried given an error
	IsRetryable(err error) bool
	// IsNotExist returns true if err is a non existence error
	IsNotExist(err error) bool
	// IsIgnorable returns true if the error can be ignored
	IsIgnorable(err error) bool
}

Client is an interface to object storage.

func NewAmazonClient 

func NewAmazonClient(region, bucket string, creds *AmazonCreds, distribution string, endpoint string, reversed ...bool) (Client, error)

NewAmazonClient creates an amazon client with the following credentials: 

bucket - S3 bucket name
distribution - cloudfront distribution ID
id     - AWS access key id
secret - AWS secret access key
token  - AWS access token
region - AWS region
endpoint - Custom endpoint (generally used for S3 compatible object stores)

func NewAmazonClientFromEnv added in v1.8.0 

func NewAmazonClientFromEnv() (Client, error)

NewAmazonClientFromEnv creates a Amazon client based on environment variables.

func NewAmazonClientFromSecret added in v1.3.2 

func NewAmazonClientFromSecret(bucket string, reversed ...bool) (Client, error)

NewAmazonClientFromSecret constructs an amazon client by reading credentials from a mounted AmazonSecret. You may pass "" for bucket in which case it will read the bucket from the secret.

func NewClientFromEnv added in v1.8.0 

func NewClientFromEnv(storageRoot string) (c Client, err error)

NewClientFromEnv creates a client based on environment variables.

func NewClientFromSecret added in v1.9.0 

func NewClientFromSecret(storageRoot string) (c Client, err error)

NewClientFromSecret creates a client based on mounted secret files.

func NewClientFromURLAndSecret added in v1.3.2 

func NewClientFromURLAndSecret(url *ObjectStoreURL, reversed ...bool) (c Client, err error)

NewClientFromURLAndSecret constructs a client by parsing `URL` and then constructing the correct client for that URL using secrets.

func NewGoogleClient 

func NewGoogleClient(bucket string, opts []option.ClientOption) (Client, error)

NewGoogleClient creates a google client with the given bucket name.

func NewGoogleClientFromEnv added in v1.8.0 

func NewGoogleClientFromEnv() (Client, error)

NewGoogleClientFromEnv creates a Google client based on environment variables.

func NewGoogleClientFromSecret added in v1.3.2 

func NewGoogleClientFromSecret(bucket string) (Client, error)

NewGoogleClientFromSecret creates a google client by reading credentials from a mounted GoogleSecret. You may pass "" for bucket in which case it will read the bucket from the secret.

func NewLocalClient added in v1.6.0 

func NewLocalClient(root string) (Client, error)

NewLocalClient returns a Client that stores data on the local file system

func NewMicrosoftClient added in v1.2.3 

func NewMicrosoftClient(container string, accountName string, accountKey string) (Client, error)

NewMicrosoftClient creates a microsoft client: 

container   - Azure Blob Container name
accountName - Azure Storage Account name
accountKey  - Azure Storage Account key

func NewMicrosoftClientFromEnv added in v1.8.0 

func NewMicrosoftClientFromEnv() (Client, error)

NewMicrosoftClientFromEnv creates a Microsoft client based on environment variables.

func NewMicrosoftClientFromSecret added in v1.3.2 

func NewMicrosoftClientFromSecret(container string) (Client, error)

NewMicrosoftClientFromSecret creates a microsoft client by reading credentials from a mounted MicrosoftSecret. You may pass "" for container in which case it will read the container from the secret.

func NewMinioClient added in v1.3.5 

func NewMinioClient(endpoint, bucket, id, secret string, secure, isS3V2 bool) (Client, error)

NewMinioClient creates an s3 compatible client with the following credentials: 

endpoint - S3 compatible endpoint
bucket - S3 bucket name
id     - AWS access key id
secret - AWS secret access key
secure - Set to true if connection is secure.
isS3V2 - Set to true if client follows S3V2

func NewMinioClientFromEnv added in v1.8.0 

func NewMinioClientFromEnv() (Client, error)

NewMinioClientFromEnv creates a Minio client based on environment variables.

func NewMinioClientFromSecret added in v1.3.5 

func NewMinioClientFromSecret(bucket string) (Client, error)

NewMinioClientFromSecret constructs an s3 compatible client by reading credentials from a mounted AmazonSecret. You may pass "" for bucket in which case it will read the bucket from the secret.

func TracingObjClient added in v1.7.12 

func TracingObjClient(provider string, c Client) Client

TracingObjClient wraps the given object client 'c', adding tracing to all calls made by the returned interface

type ObjectStoreURL added in v1.5.3 

type ObjectStoreURL struct {
	// The object store, e.g. s3, gcs, as...
	Store string
	// The "bucket" (in AWS parlance) or the "container" (in Azure parlance).
	Bucket string
	// The object itself.
	Object string
}

ObjectStoreURL represents a parsed URL to an object in an object store.

func ParseURL added in v1.5.3 

func ParseURL(urlStr string) (*ObjectStoreURL, error)

ParseURL parses an URL into ObjectStoreURL.

type RetryError added in v1.1.0 

type RetryError struct {
	Err               string
	TimeTillNextRetry string
	BytesProcessed    int
}

RetryError is used to log retry attempts.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.