storageccl

Documentation

Index

• Constants
• func ExportStorageConfFromURI(path string) (roachpb.ExportStorage, error)
• type ExportFileWriter
• type ExportStorage
  • func ExportStorageFromURI(ctx context.Context, uri string) (ExportStorage, error)
  • func MakeExportStorage(ctx context.Context, dest roachpb.ExportStorage) (ExportStorage, error)
• type KeyRewriter
  • func (kr KeyRewriter) RewriteKey(key []byte) ([]byte, bool)

Constants

const (
	// S3AccessKeyParam is the query parameter for access_key in an S3 URI.
	S3AccessKeyParam = "AWS_ACCESS_KEY_ID"
	// S3SecretParam is the query parameter for the 'secret' in an S3 URI.
	S3SecretParam = "AWS_SECRET_ACCESS_KEY"

	// AzureAccountNameParam is the query parameter for account_name in an azure URI.
	AzureAccountNameParam = "AZURE_ACCOUNT_NAME"
	// AzureAccountKeyParam is the query parameter for account_key in an azure URI.
	AzureAccountKeyParam = "AZURE_ACCOUNT_KEY"
)

Functions

func ExportStorageConfFromURI

func ExportStorageConfFromURI(path string) (roachpb.ExportStorage, error)

ExportStorageConfFromURI generates an ExportStorage config from a URI string.

Types

type ExportFileWriter

type ExportFileWriter interface {
	// LocalFile returns the path to a local path to which a caller should write.
	LocalFile() string
	// Finish indicates that no further writes to the local file are expected and
	// that the implementation should store the content (copy it, upload, etc) if
	// that has not already been done in a streaming fashion (e.g. via a pipe).
	Finish() error
	// Cleanup removes any temporary files or resources that were made on behalf
	// of this writer. If `Finish` has not been called, any writes to
	// `LocalFile` will be lost. Implementations of `Cleanup` are required to be
	// idempotent and should log any errors.
	Cleanup()
}

ExportFileWriter provides a local file or pipe that can be written to before calling Finish() to store the content of said file. As some writers may be non-Go (e.g. the RocksDB SSTable Writer), this is intentionally the path to a file, not an io.Writer, and the required call to `Finish` gives implementations the opportunity to move/copy/upload/etc as needed.

type ExportStorage

type ExportStorage interface {
	io.Closer
	// Conf should return the serializable configuration required to reconstruct
	// this ExportStorage implementation.
	Conf() roachpb.ExportStorage
	// PutFile is used to prepare to write a file to the storage.
	// See ExportFileWriter.
	PutFile(ctx context.Context, basename string) (ExportFileWriter, error)
	// ReadFile should return a Reader for requested name.
	ReadFile(ctx context.Context, basename string) (io.ReadCloser, error)
	// FetchFile returns the path to a local file containing the content of
	// the requested filename. Implementations may wish to use the `fetchFile`
	// helper for copying the content of ReadFile to a temporary file.
	FetchFile(ctx context.Context, basename string) (string, error)
	// Delete removes the named file from the store.
	Delete(ctx context.Context, basename string) error
}

ExportStorage handles reading and writing files in an export.

func ExportStorageFromURI

func ExportStorageFromURI(ctx context.Context, uri string) (ExportStorage, error)

ExportStorageFromURI returns an ExportStorage for the given URI.

func MakeExportStorage

func MakeExportStorage(ctx context.Context, dest roachpb.ExportStorage) (ExportStorage, error)

MakeExportStorage creates an ExportStorage from the given config.

type KeyRewriter

type KeyRewriter []roachpb.KeyRewrite

KeyRewriter is an matcher for an ordered list of pairs of byte prefix rewrite rules. For dependency reasons, the implementation of the matching is here, but the interesting constructor is in sqlccl.

func (KeyRewriter) RewriteKey

func (kr KeyRewriter) RewriteKey(key []byte) ([]byte, bool)

RewriteKey modifies key using the first matching rule and returns it. If no rules matched, returns false and the original input key.