storageccl

Documentation

Index

• Constants
• Variables
• func ExportStorageConfFromURI(path string) (roachpb.ExportStorage, error)
• func MakeLocalStorageURI(path string) (string, error)
• func SHA512ChecksumData(data []byte) ([]byte, error)
• func SanitizeExportStorageURI(path string) (string, error)
• type ExportStorage
  • func MakeExportStorage(ctx context.Context, dest roachpb.ExportStorage) (ExportStorage, error)
• type KeyRewriter
  • func MakeKeyRewriter(rekeys []roachpb.ImportRequest_TableRekey) (*KeyRewriter, error)
  • func (kr *KeyRewriter) RewriteKey(key []byte) ([]byte, bool, error)
  • func (kr *KeyRewriter) RewriteSpan(span roachpb.Span) (roachpb.Span, error)

Constants

const (
	// S3AccessKeyParam is the query parameter for access_key in an S3 URI.
	S3AccessKeyParam = "AWS_ACCESS_KEY_ID"
	// S3SecretParam is the query parameter for the 'secret' in an S3 URI.
	S3SecretParam = "AWS_SECRET_ACCESS_KEY"

	// AzureAccountNameParam is the query parameter for account_name in an azure URI.
	AzureAccountNameParam = "AZURE_ACCOUNT_NAME"
	// AzureAccountKeyParam is the query parameter for account_key in an azure URI.
	AzureAccountKeyParam = "AZURE_ACCOUNT_KEY"
)

const ExportRequestLimit = 5

ExportRequestLimit is the number of Export requests that can run at once. Each extracts data from RocksDB to a temp file and then uploads it to cloud storage. In order to not exhaust the disk or memory, or saturate the network, limit the number of these that can be run in parallel. This number was chosen by a guess. If SST files are likely to not be over 200MB, then 5 parallel workers hopefully won't use more than 1GB of space in the temp directory. It could be improved by more measured heuristics.

Variables

var AddSSTableEnabled = settings.RegisterBoolSetting(
	"kv.import.experimental_addsstable.enabled",
	"set to true to use the AddSSTable command in Import or false to use WriteBatch",
	true,
)

AddSSTableEnabled wraps "kv.import.experimental_addsstable.enabled".

Functions

func ExportStorageConfFromURI

func ExportStorageConfFromURI(path string) (roachpb.ExportStorage, error)

ExportStorageConfFromURI generates an ExportStorage config from a URI string.

func MakeLocalStorageURI

func MakeLocalStorageURI(path string) (string, error)

MakeLocalStorageURI converts a local path (absolute or relative) to a valid nodelocal URI.

func SHA512ChecksumData

func SHA512ChecksumData(data []byte) ([]byte, error)

SHA512ChecksumData returns the SHA512 checksum of data.

func SanitizeExportStorageURI

func SanitizeExportStorageURI(path string) (string, error)

SanitizeExportStorageURI returns the export storage URI with sensitive credentials stripped.

Types

type ExportStorage

type ExportStorage interface {
	io.Closer

	// Conf should return the serializable configuration required to reconstruct
	// this ExportStorage implementation.
	Conf() roachpb.ExportStorage

	// ReadFile should return a Reader for requested name.
	ReadFile(ctx context.Context, basename string) (io.ReadCloser, error)

	// WriteFile should write the content to requested name.
	WriteFile(ctx context.Context, basename string, content io.ReadSeeker) error

	// Delete removes the named file from the store.
	Delete(ctx context.Context, basename string) error
}

ExportStorage provides functions to read and write files in some storage, namely various cloud storage providers, for example to store backups. Generally an implementation is instantiated pointing to some base path or prefix and then gets and puts files using the various methods to interact with individual files contained within that path or prefix. However, implementations must also allow callers to provide the full path to a given file as the "base" path, and then read or write it with the methods below by simply passing an empty filename. Implementations that use stdlib's `path.Join` to concatenate their base path with the provided filename will find its semantics well suited to this -- it elides empty components and does not append surplus slashes.

func MakeExportStorage

func MakeExportStorage(ctx context.Context, dest roachpb.ExportStorage) (ExportStorage, error)

MakeExportStorage creates an ExportStorage from the given config.

type KeyRewriter

type KeyRewriter struct {
	// contains filtered or unexported fields
}

KeyRewriter rewrites old table IDs to new table IDs. It is able to descend into interleaved keys, and is able to function on partial keys for spans and splits.

func MakeKeyRewriter

func MakeKeyRewriter(rekeys []roachpb.ImportRequest_TableRekey) (*KeyRewriter, error)

MakeKeyRewriter creates a KeyRewriter. This includes a simple []byte prefix rewriter to rewrite table IDs including prefix ends, and table descriptor data to traverse interleaved keys to child tables.

func (*KeyRewriter) RewriteKey

func (kr *KeyRewriter) RewriteKey(key []byte) ([]byte, bool, error)

RewriteKey modifies key (possibly in place), changing all table IDs to their new value, including any interleaved table children and prefix ends. This function works by inspecting the key for table and index IDs, then uses the corresponding table and index descriptors to determine if interleaved data is present and if it is, to find the next prefix of an interleaved child, then calls itself recursively until all interleaved children have been rekeyed.

func (*KeyRewriter) RewriteSpan

func (kr *KeyRewriter) RewriteSpan(span roachpb.Span) (roachpb.Span, error)

RewriteSpan returns a new span with both Key and EndKey rewritten using RewriteKey. An error is returned if either was not matched for rewrite.