storage

package
v0.0.0-...-503c688 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 8, 2023 License: Apache-2.0, Apache-2.0 Imports: 52 Imported by: 0

Documentation

Index

Constants

View Source 
const (

	// LocalURIPrefix represents the local storage prefix.
	LocalURIPrefix = "file://"
)

Variables

This section is empty.

Functions

func DefineFlags 

func DefineFlags(flags *pflag.FlagSet)

DefineFlags adds flags to the flag set corresponding to all backend options.

func ExtractQueryParameters 

func ExtractQueryParameters(u *url.URL, options interface{})

ExtractQueryParameters moves the query parameters of the URL into the options using reflection.

The options must be a pointer to a struct which contains only string or bool fields (more types will be supported in the future), and tagged for JSON serialization.

All of the URL's query parameters will be removed after calling this method.

func FormatBackendURL 

func FormatBackendURL(backend *backuppb.StorageBackend) (u url.URL)

FormatBackendURL obtains the raw URL which can be used the reconstruct the backend. The returned URL does not contain options for further configurating the backend. This is to avoid exposing secret tokens.

func HiddenFlagsForStream 

func HiddenFlagsForStream(flags *pflag.FlagSet)

HiddenFlagsForStream hidden flags for stream cmd.

func InterceptDecompressReader 

func InterceptDecompressReader(fileReader io.ReadSeekCloser, compressType CompressType) (io.ReadSeekCloser, error)

InterceptDecompressReader intercepts the reader and wraps it with a decompress reader on the given io.ReadSeekCloser. Note that the returned io.ReadSeekCloser does not have the property that Seek(0, io.SeekCurrent) equals total bytes Read() if the decompress reader is used.

func ParseBackend 

func ParseBackend(rawURL string, options *BackendOptions) (*backuppb.StorageBackend, error)

ParseBackend constructs a structured backend description from the storage URL.

func ParseBackendFromURL 

func ParseBackendFromURL(u *url.URL, options *BackendOptions) (*backuppb.StorageBackend, error)

ParseBackendFromURL constructs a structured backend description from the *url.URL.

func ParseRawURL 

func ParseRawURL(rawURL string) (*url.URL, error)

ParseRawURL parse raw url to url object.

Types

type AzblobBackendOptions 

type AzblobBackendOptions struct {
	Endpoint    string `json:"endpoint" toml:"endpoint"`
	AccountName string `json:"account-name" toml:"account-name"`
	AccountKey  string `json:"account-key" toml:"account-key"`
	AccessTier  string `json:"access-tier" toml:"access-tier"`
}

AzblobBackendOptions is the options for Azure Blob storage.

type AzureBlobStorage 

type AzureBlobStorage struct {
	// contains filtered or unexported fields
}

AzureBlobStorage is a storage engine that stores data in Azure Blob Storage.

func (*AzureBlobStorage) Create 

func (s *AzureBlobStorage) Create(_ context.Context, name string) (ExternalFileWriter, error)

Create implements the StorageWriter interface.

func (*AzureBlobStorage) DeleteFile 

func (s *AzureBlobStorage) DeleteFile(ctx context.Context, name string) error

DeleteFile deletes the file with the given name.

func (*AzureBlobStorage) FileExists 

func (s *AzureBlobStorage) FileExists(ctx context.Context, name string) (bool, error)

FileExists checks if a file exists in Azure Blob Storage.

func (*AzureBlobStorage) Open 

func (s *AzureBlobStorage) Open(ctx context.Context, name string) (ExternalFileReader, error)

Open implements the StorageReader interface.

func (*AzureBlobStorage) ReadFile 

func (s *AzureBlobStorage) ReadFile(ctx context.Context, name string) ([]byte, error)

ReadFile reads a file from Azure Blob Storage.

func (*AzureBlobStorage) Rename 

func (s *AzureBlobStorage) Rename(ctx context.Context, oldFileName, newFileName string) error

Rename implements the StorageWriter interface.

func (*AzureBlobStorage) URI 

func (s *AzureBlobStorage) URI() string

URI implements the StorageReader interface.

func (*AzureBlobStorage) WalkDir 

func (s *AzureBlobStorage) WalkDir(ctx context.Context, opt *WalkOption, fn func(path string, size int64) error) error

WalkDir implements the StorageReader interface.

func (*AzureBlobStorage) WriteFile 

func (s *AzureBlobStorage) WriteFile(ctx context.Context, name string, data []byte) error

WriteFile writes a file to Azure Blob Storage.

type BackendOptions 

type BackendOptions struct {
	S3     S3BackendOptions     `json:"s3" toml:"s3"`
	GCS    GCSBackendOptions    `json:"gcs" toml:"gcs"`
	Azblob AzblobBackendOptions `json:"azblob" toml:"azblob"`
}

BackendOptions further configures the storage backend not expressed by the storage URL.

func (*BackendOptions) ParseFromFlags 

func (options *BackendOptions) ParseFromFlags(flags *pflag.FlagSet) error

ParseFromFlags obtains the backend options from the flag set.

type BytesWriter 

type BytesWriter struct {
	// contains filtered or unexported fields
}

BytesWriter is a Writer implementation on top of bytes.Buffer that is useful for testing.

func NewBufferWriter 

func NewBufferWriter() *BytesWriter

NewBufferWriter creates a Writer that simply writes to a buffer (useful for testing).

func (*BytesWriter) Bytes 

func (u *BytesWriter) Bytes() []byte

Bytes delegates to bytes.Buffer.

func (*BytesWriter) Close 

func (*BytesWriter) Close(_ context.Context) error

Close delegates to bytes.Buffer.

func (*BytesWriter) Reset 

func (u *BytesWriter) Reset()

Reset delegates to bytes.Buffer.

func (*BytesWriter) String 

func (u *BytesWriter) String() string

String delegates to bytes.Buffer.

func (*BytesWriter) Write 

func (u *BytesWriter) Write(_ context.Context, p []byte) (int, error)

Write delegates to bytes.Buffer.

type ClientBuilder 

type ClientBuilder interface {
	// Example of serviceURL: https://<your_storage_account>.blob.core.windows.net
	GetServiceClient() (azblob.ServiceClient, error)
	GetAccountName() string
}

ClientBuilder provides common method to build a service client.

type CompressType 

type CompressType uint8

CompressType represents the type of compression. 

const (
	// NoCompression won't compress given bytes.
	NoCompression CompressType = iota
	// Gzip will compress given bytes in gzip format.
	Gzip
	// Snappy will compress given bytes in snappy format.
	Snappy
	// Zstd will compress given bytes in zstd format.
	Zstd
)

type ExternalFileReader 

type ExternalFileReader interface {
	io.ReadSeekCloser
}

ExternalFileReader represents the streaming external file reader.

func NewLimitedInterceptReader 

func NewLimitedInterceptReader(fileReader ExternalFileReader, compressType CompressType, n int64) (ExternalFileReader, error)

type ExternalFileWriter 

type ExternalFileWriter interface {
	// Write writes to buffer and if chunk is filled will upload it
	Write(ctx context.Context, p []byte) (int, error)
	// Close writes final chunk and completes the upload
	Close(ctx context.Context) error
}

ExternalFileWriter represents the streaming external file writer.

func NewUploaderWriter 

func NewUploaderWriter(writer ExternalFileWriter, chunkSize int, compressType CompressType) ExternalFileWriter

NewUploaderWriter wraps the Writer interface over an uploader.

type ExternalStorage 

type ExternalStorage interface {
	// WriteFile writes a complete file to storage, similar to os.WriteFile, but WriteFile should be atomic
	WriteFile(ctx context.Context, name string, data []byte) error
	// ReadFile reads a complete file from storage, similar to os.ReadFile
	ReadFile(ctx context.Context, name string) ([]byte, error)
	// FileExists return true if file exists
	FileExists(ctx context.Context, name string) (bool, error)
	// DeleteFile delete the file in storage
	DeleteFile(ctx context.Context, name string) error
	// Open a Reader by file path. path is relative path to storage base path
	Open(ctx context.Context, path string) (ExternalFileReader, error)
	// WalkDir traverse all the files in a dir.
	//
	// fn is the function called for each regular file visited by WalkDir.
	// The argument `path` is the file path that can be used in `Open`
	// function; the argument `size` is the size in byte of the file determined
	// by path.
	WalkDir(ctx context.Context, opt *WalkOption, fn func(path string, size int64) error) error

	// URI returns the base path as a URI
	URI() string

	// Create opens a file writer by path. path is relative path to storage base path
	Create(ctx context.Context, path string) (ExternalFileWriter, error)
	// Rename file name from oldFileName to newFileName
	Rename(ctx context.Context, oldFileName, newFileName string) error
}

ExternalStorage represents a kind of file system storage.

func Create 

func Create(ctx context.Context, backend *backuppb.StorageBackend, sendCreds bool) (ExternalStorage, error)

Create creates ExternalStorage.

Please consider using `New` in the future.

func New 

func New(ctx context.Context, backend *backuppb.StorageBackend, opts *ExternalStorageOptions) (ExternalStorage, error)

New creates an ExternalStorage with options.

func WithCompression 

func WithCompression(inner ExternalStorage, compressionType CompressType) ExternalStorage

WithCompression returns an ExternalStorage with compress option

type ExternalStorageOptions 

type ExternalStorageOptions struct {
	// SendCredentials marks whether to send credentials downstream.
	//
	// This field should be set to false if the credentials are provided to
	// downstream via external key managers, e.g. on K8s or cloud provider.
	SendCredentials bool

	// NoCredentials means that no cloud credentials are supplied to BR
	NoCredentials bool

	// HTTPClient to use. The created storage may ignore this field if it is not
	// directly using HTTP (e.g. the local storage).
	HTTPClient *http.Client

	// CheckPermissions check the given permission in New() function.
	// make sure we can access the storage correctly before execute tasks.
	CheckPermissions []Permission

	// S3Retryer is the retryer for create s3 storage, if it is nil,
	// defaultS3Retryer() will be used.
	S3Retryer request.Retryer

	// CheckObjectLockOptions check the s3 bucket has enabled the ObjectLock.
	// if enabled. it will send the options to tikv.
	CheckS3ObjectLockOptions bool
}

ExternalStorageOptions are backend-independent options provided to New.

type GCSBackendOptions 

type GCSBackendOptions struct {
	Endpoint        string `json:"endpoint" toml:"endpoint"`
	StorageClass    string `json:"storage-class" toml:"storage-class"`
	PredefinedACL   string `json:"predefined-acl" toml:"predefined-acl"`
	CredentialsFile string `json:"credentials-file" toml:"credentials-file"`
}

GCSBackendOptions are options for configuration the GCS storage.

type GCSStorage 

type GCSStorage struct {
	// contains filtered or unexported fields
}

GCSStorage defines some standard operations for BR/Lightning on the GCS storage. It implements the `ExternalStorage` interface.

func NewGCSStorage 

func NewGCSStorage(ctx context.Context, gcs *backuppb.GCS, opts *ExternalStorageOptions) (*GCSStorage, error)

NewGCSStorage creates a GCS external storage implementation.

func (*GCSStorage) Create 

func (s *GCSStorage) Create(ctx context.Context, name string) (ExternalFileWriter, error)

Create implements ExternalStorage interface.

func (*GCSStorage) DeleteFile 

func (s *GCSStorage) DeleteFile(ctx context.Context, name string) error

DeleteFile delete the file in storage

func (*GCSStorage) FileExists 

func (s *GCSStorage) FileExists(ctx context.Context, name string) (bool, error)

FileExists return true if file exists.

func (*GCSStorage) GetBucketHandle 

func (s *GCSStorage) GetBucketHandle() *storage.BucketHandle

GetBucketHandle gets the handle to the GCS API on the bucket.

func (*GCSStorage) GetOptions 

func (s *GCSStorage) GetOptions() *backuppb.GCS

GetOptions gets the external storage operations for the GCS.

func (*GCSStorage) Open 

func (s *GCSStorage) Open(ctx context.Context, path string) (ExternalFileReader, error)

Open a Reader by file path.

func (*GCSStorage) ReadFile 

func (s *GCSStorage) ReadFile(ctx context.Context, name string) ([]byte, error)

ReadFile reads the file from the storage and returns the contents.

func (*GCSStorage) Rename 

func (s *GCSStorage) Rename(ctx context.Context, oldFileName, newFileName string) error

Rename file name from oldFileName to newFileName.

func (*GCSStorage) URI 

func (s *GCSStorage) URI() string

func (*GCSStorage) WalkDir 

func (s *GCSStorage) WalkDir(ctx context.Context, opt *WalkOption, fn func(string, int64) error) error

WalkDir traverse all the files in a dir.

fn is the function called for each regular file visited by WalkDir. The first argument is the file path that can be used in `Open` function; the second argument is the size in byte of the file determined by path.

func (*GCSStorage) WriteFile 

func (s *GCSStorage) WriteFile(ctx context.Context, name string, data []byte) error

WriteFile writes data to a file to storage.

type HDFSStorage 

type HDFSStorage struct {
	// contains filtered or unexported fields
}

HDFSStorage represents HDFS storage.

func NewHDFSStorage 

func NewHDFSStorage(remote string) *HDFSStorage

NewHDFSStorage creates a new HDFS storage.

func (*HDFSStorage) Create 

func (*HDFSStorage) Create(_ context.Context, _ string) (ExternalFileWriter, error)

Create opens a file writer by path. path is relative path to storage base path

func (*HDFSStorage) DeleteFile 

func (*HDFSStorage) DeleteFile(_ context.Context, _ string) error

DeleteFile delete the file in storage

func (*HDFSStorage) FileExists 

func (s *HDFSStorage) FileExists(_ context.Context, name string) (bool, error)

FileExists return true if file exists

func (*HDFSStorage) Open 

func (*HDFSStorage) Open(_ context.Context, _ string) (ExternalFileReader, error)

Open a Reader by file path. path is relative path to storage base path

func (*HDFSStorage) ReadFile 

func (*HDFSStorage) ReadFile(_ context.Context, _ string) ([]byte, error)

ReadFile reads a complete file from storage, similar to os.ReadFile

func (*HDFSStorage) Rename 

func (*HDFSStorage) Rename(_ context.Context, _, _ string) error

Rename a file name from oldFileName to newFileName.

func (*HDFSStorage) URI 

func (s *HDFSStorage) URI() string

URI returns the base path as a URI

func (*HDFSStorage) WalkDir 

func (*HDFSStorage) WalkDir(_ context.Context, _ *WalkOption, _ func(path string, size int64) error) error

WalkDir traverse all the files in a dir.

fn is the function called for each regular file visited by WalkDir. The argument `path` is the file path that can be used in `Open` function; the argument `size` is the size in byte of the file determined by path.

func (*HDFSStorage) WriteFile 

func (s *HDFSStorage) WriteFile(_ context.Context, name string, data []byte) error

WriteFile writes a complete file to storage, similar to os.WriteFile

type LocalStorage 

type LocalStorage struct {
	// contains filtered or unexported fields
}

LocalStorage represents local file system storage.

export for using in tests.

func NewLocalStorage 

func NewLocalStorage(base string) (*LocalStorage, error)

NewLocalStorage return a LocalStorage at directory `base`.

export for test.

func (*LocalStorage) Create 

func (l *LocalStorage) Create(_ context.Context, name string) (ExternalFileWriter, error)

Create implements ExternalStorage interface.

func (*LocalStorage) DeleteFile 

func (l *LocalStorage) DeleteFile(_ context.Context, name string) error

DeleteFile deletes the file.

func (*LocalStorage) FileExists 

func (l *LocalStorage) FileExists(_ context.Context, name string) (bool, error)

FileExists implement ExternalStorage.FileExists.

func (*LocalStorage) Open 

func (l *LocalStorage) Open(_ context.Context, path string) (ExternalFileReader, error)

Open a Reader by file path, path is a relative path to base path.

func (*LocalStorage) ReadFile 

func (l *LocalStorage) ReadFile(_ context.Context, name string) ([]byte, error)

ReadFile reads the file from the storage and returns the contents.

func (*LocalStorage) Rename 

func (l *LocalStorage) Rename(_ context.Context, oldFileName, newFileName string) error

Rename implements ExternalStorage interface.

func (*LocalStorage) URI 

func (l *LocalStorage) URI() string

URI returns the base path as an URI with a file:/// prefix.

func (*LocalStorage) WalkDir 

func (l *LocalStorage) WalkDir(_ context.Context, opt *WalkOption, fn func(string, int64) error) error

WalkDir traverse all the files in a dir.

fn is the function called for each regular file visited by WalkDir. The first argument is the file path that can be used in `Open` function; the second argument is the size in byte of the file determined by path.

func (*LocalStorage) WriteFile 

func (l *LocalStorage) WriteFile(_ context.Context, name string, data []byte) error

WriteFile writes data to a file to storage.

type MemStorage 

type MemStorage struct {
	// contains filtered or unexported fields
}

MemStorage represents a in-memory storage.

func NewMemStorage 

func NewMemStorage() *MemStorage

NewMemStorage creates a new in-memory storage.

func (*MemStorage) Create 

func (s *MemStorage) Create(ctx context.Context, name string) (ExternalFileWriter, error)

Create creates a file and returning a writer to write data into. When the writer is closed, the data is stored in the file. It implements the `ExternalStorage` interface

func (*MemStorage) DeleteFile 

func (s *MemStorage) DeleteFile(ctx context.Context, name string) error

DeleteFile delete the file in storage It implements the `ExternalStorage` interface

func (*MemStorage) FileExists 

func (s *MemStorage) FileExists(ctx context.Context, name string) (bool, error)

FileExists return true if file exists. It implements the `ExternalStorage` interface

func (*MemStorage) Open 

func (s *MemStorage) Open(ctx context.Context, filePath string) (ExternalFileReader, error)

Open opens a Reader by file path. It implements the `ExternalStorage` interface

func (*MemStorage) ReadFile 

func (s *MemStorage) ReadFile(ctx context.Context, name string) ([]byte, error)

ReadFile reads the storage file. It implements the `ExternalStorage` interface

func (*MemStorage) Rename 

func (s *MemStorage) Rename(ctx context.Context, oldFileName, newFileName string) error

Rename renames a file name to another file name. It implements the `ExternalStorage` interface

func (*MemStorage) URI 

func (*MemStorage) URI() string

URI returns the URI of the storage.

func (*MemStorage) WalkDir 

func (s *MemStorage) WalkDir(ctx context.Context, opt *WalkOption, fn func(string, int64) error) error

WalkDir traverse all the files in a dir. It implements the `ExternalStorage` interface

func (*MemStorage) WriteFile 

func (s *MemStorage) WriteFile(ctx context.Context, name string, data []byte) error

WriteFile file to storage. It implements the `ExternalStorage` interface

type Permission 

type Permission string

Permission represents the permission we need to check in create storage. 

const (
	// AccessBuckets represents bucket access permission
	// it replace the origin skip-check-path.
	AccessBuckets Permission = "AccessBucket"

	// ListObjects represents listObjects permission
	ListObjects Permission = "ListObjects"
	// GetObject represents GetObject permission
	GetObject Permission = "GetObject"
	// PutObject represents PutObject permission
	PutObject Permission = "PutObject"
)

type RangeInfo 

type RangeInfo struct {
	// Start is the absolute position of the first byte of the byte range,
	// starting from 0.
	Start int64
	// End is the absolute position of the last byte of the byte range. This end
	// offset is inclusive, e.g. if the Size is 1000, the maximum value of End
	// would be 999.
	End int64
	// Size is the total size of the original file.
	Size int64
}

RangeInfo represents the an HTTP Content-Range header value of the form `bytes [Start]-[End]/[Size]`.

func ParseRangeInfo 

func ParseRangeInfo(info *string) (ri RangeInfo, err error)

ParseRangeInfo parses the Content-Range header and returns the offsets.

type ReadSeekCloser 

type ReadSeekCloser interface {
	io.Reader
	io.Seeker
	io.Closer
}

ReadSeekCloser is the interface that groups the basic Read, Seek and Close methods.

type S3BackendOptions 

type S3BackendOptions struct {
	Endpoint              string `json:"endpoint" toml:"endpoint"`
	Region                string `json:"region" toml:"region"`
	StorageClass          string `json:"storage-class" toml:"storage-class"`
	Sse                   string `json:"sse" toml:"sse"`
	SseKmsKeyID           string `json:"sse-kms-key-id" toml:"sse-kms-key-id"`
	ACL                   string `json:"acl" toml:"acl"`
	AccessKey             string `json:"access-key" toml:"access-key"`
	SecretAccessKey       string `json:"secret-access-key" toml:"secret-access-key"`
	SessionToken          string `json:"session-token" toml:"session-token"`
	Provider              string `json:"provider" toml:"provider"`
	ForcePathStyle        bool   `json:"force-path-style" toml:"force-path-style"`
	UseAccelerateEndpoint bool   `json:"use-accelerate-endpoint" toml:"use-accelerate-endpoint"`
	RoleARN               string `json:"role-arn" toml:"role-arn"`
	ExternalID            string `json:"external-id" toml:"external-id"`
	ObjectLockEnabled     bool   `json:"object-lock-enabled" toml:"object-lock-enabled"`
}

S3BackendOptions contains options for s3 storage.

func (*S3BackendOptions) Apply 

func (options *S3BackendOptions) Apply(s3 *backuppb.S3) error

Apply apply s3 options on backuppb.S3.

type S3Storage 

type S3Storage struct {
	// contains filtered or unexported fields
}

S3Storage defines some standard operations for BR/Lightning on the S3 storage. It implements the `ExternalStorage` interface.

func NewS3Storage 

func NewS3Storage(ctx context.Context, backend *backuppb.S3, opts *ExternalStorageOptions) (obj *S3Storage, errRet error)

NewS3Storage initialize a new s3 storage for metadata.

func NewS3StorageForTest 

func NewS3StorageForTest(svc s3iface.S3API, options *backuppb.S3) *S3Storage

NewS3StorageForTest creates a new S3Storage for testing only.

func (*S3Storage) Create 

func (rs *S3Storage) Create(ctx context.Context, name string) (ExternalFileWriter, error)

Create creates multi upload request.

func (*S3Storage) CreateUploader 

func (rs *S3Storage) CreateUploader(ctx context.Context, name string) (ExternalFileWriter, error)

CreateUploader create multi upload request.

func (*S3Storage) DeleteFile 

func (rs *S3Storage) DeleteFile(ctx context.Context, file string) error

DeleteFile delete the file in s3 storage

func (*S3Storage) FileExists 

func (rs *S3Storage) FileExists(ctx context.Context, file string) (bool, error)

FileExists check if file exists on s3 storage.

func (*S3Storage) GetOptions 

func (rs *S3Storage) GetOptions() *backuppb.S3

GetOptions gets the external storage operations for the S3.

func (*S3Storage) GetS3APIHandle 

func (rs *S3Storage) GetS3APIHandle() s3iface.S3API

GetS3APIHandle gets the handle to the S3 API.

func (*S3Storage) IsObjectLockEnabled 

func (rs *S3Storage) IsObjectLockEnabled() bool

func (*S3Storage) Open 

func (rs *S3Storage) Open(ctx context.Context, path string) (ExternalFileReader, error)

Open a Reader by file path.

func (*S3Storage) ReadFile 

func (rs *S3Storage) ReadFile(ctx context.Context, file string) ([]byte, error)

ReadFile reads the file from the storage and returns the contents.

func (*S3Storage) Rename 

func (rs *S3Storage) Rename(ctx context.Context, oldFileName, newFileName string) error

Rename implements ExternalStorage interface.

func (*S3Storage) URI 

func (rs *S3Storage) URI() string

URI returns s3://<base>/<prefix>.

func (*S3Storage) WalkDir 

func (rs *S3Storage) WalkDir(ctx context.Context, opt *WalkOption, fn func(string, int64) error) error

WalkDir traverse all the files in a dir.

fn is the function called for each regular file visited by WalkDir. The first argument is the file path that can be used in `Open` function; the second argument is the size in byte of the file determined by path.

func (*S3Storage) WriteFile 

func (rs *S3Storage) WriteFile(ctx context.Context, file string, data []byte) error

WriteFile writes data to a file to storage.

type S3Uploader 

type S3Uploader struct {
	// contains filtered or unexported fields
}

S3Uploader does multi-part upload to s3.

func (*S3Uploader) Close 

func (u *S3Uploader) Close(ctx context.Context) error

Close complete multi upload request.

func (*S3Uploader) Write 

func (u *S3Uploader) Write(ctx context.Context, data []byte) (int, error)

UploadPart update partial data to s3, we should call CreateMultipartUpload to start it, and call CompleteMultipartUpload to finish it.

type Uploader 

type Uploader interface {
	// UploadPart upload part of file data to storage
	UploadPart(ctx context.Context, data []byte) error
	// CompleteUpload make the upload data to a complete file
	CompleteUpload(ctx context.Context) error
}

Uploader upload file with chunks.

type WalkOption 

type WalkOption struct {
	// walk on SubDir of specify directory
	SubDir string
	// ObjPrefix used fo prefix search in storage. Note that only part of storage
	// support it.
	// It can save lots of time when we want find specify prefix objects in storage.
	// For example. we have 10000 <Hash>.sst files and 10 backupmeta.(\d+) files.
	// we can use ObjPrefix = "backupmeta" to retrieve all meta files quickly.
	ObjPrefix string
	// ListCount is the number of entries per page.
	//
	// In cloud storages such as S3 and GCS, the files listed and sent in pages.
	// Typically a page contains 1000 files, and if a folder has 3000 descendant
	// files, one would need 3 requests to retrieve all of them. This parameter
	// controls this size. Note that both S3 and GCS limits the maximum to 1000.
	//
	// Typically you want to leave this field unassigned (zero) to use the
	// default value (1000) to minimize the number of requests, unless you want
	// to reduce the possibility of timeout on an extremely slow connection, or
	// perform testing.
	ListCount int64
}

WalkOption is the option of storage.WalkDir.

type Writer 

type Writer interface {
	// Write writes to buffer and if chunk is filled will upload it
	Write(ctx context.Context, p []byte) (int, error)
	// Close writes final chunk and completes the upload
	Close(ctx context.Context) error
}

Writer is like io.Writer but with Context, create a new writer on top of Uploader with NewUploaderWriter.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.