util

package
v1.4.0-alpha.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 22, 2022 License: Apache-2.0 Imports: 52 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (

	// DefaultVersion is the default tikv and br version
	DefaultVersion = "4.0"
)

Functions

func ConstructBRGlobalOptionsForBackup 

func ConstructBRGlobalOptionsForBackup(backup *v1alpha1.Backup) ([]string, error)

ConstructBRGlobalOptionsForBackup constructs BR global options for backup and also return the remote path.

func ConstructBRGlobalOptionsForRestore 

func ConstructBRGlobalOptionsForRestore(restore *v1alpha1.Restore) ([]string, error)

ConstructBRGlobalOptionsForRestore constructs BR global options for restore.

func ConstructDumplingOptionsForBackup added in v1.1.3 

func ConstructDumplingOptionsForBackup(backup *v1alpha1.Backup) []string

ConstructDumplingOptionsForBackup constructs dumpling options for backup

func ConstructRcloneArgs added in v1.1.8 

func ConstructRcloneArgs(conf string, opts []string, command, source, dest string, verboseLog bool) []string

ConstructRcloneArgs constructs the rclone args

func EnsureDirectoryExist 

func EnsureDirectoryExist(dirName string) error

EnsureDirectoryExist create directory if does not exist

func GenStorageArgsForFlag 

func GenStorageArgsForFlag(provider v1alpha1.StorageProvider, flag string) ([]string, error)

genStorageArgs returns the arg for --flag option and the remote/local path for br, default flag is storage. TODO: add unit test

func GetBRArchiveSize added in v1.1.7 

func GetBRArchiveSize(meta *kvbackup.BackupMeta) uint64

GetBRArchiveSize returns the total size of the backup archive.

func GetBRMetaData added in v1.1.7 

func GetBRMetaData(ctx context.Context, provider v1alpha1.StorageProvider) (*kvbackup.BackupMeta, error)

GetBRMetaData get backup metadata from cloud storage

func GetCommitTsFromBRMetaData added in v1.1.3 

func GetCommitTsFromBRMetaData(ctx context.Context, provider v1alpha1.StorageProvider) (uint64, error)

GetCommitTsFromBRMetaData get backup position from `EndVersion` in BR backup meta

func GetCommitTsFromMetadata added in v1.1.3 

func GetCommitTsFromMetadata(backupPath string) (string, error)

GetCommitTsFromMetadata get commitTs from mydumper's metadata file

metadata file format is as follows: 

Started dump at: 2019-06-13 10:00:04
SHOW MASTER STATUS:
	Log: tidb-binlog
	Pos: 409054741514944513
	GTID:

Finished dump at: 2019-06-13 10:00:04

func GetContextForTerminationSignals added in v1.1.10 

func GetContextForTerminationSignals(op string) (context.Context, context.CancelFunc)

GetContextForTerminationSignals get a context for some termination signals, and the context will become done after any of these signals triggered.

func GetOptionValueFromEnv 

func GetOptionValueFromEnv(option, envPrefix string) string

GetOptionValueFromEnv get option's value from environment variable. If unset, return empty string.

func GetOptions 

func GetOptions(provider v1alpha1.StorageProvider) []string

GetOptions gets the rclone options

func GetStoragePath added in v1.1.8 

func GetStoragePath(backup *v1alpha1.Backup) (string, error)

GetStoragePath generate the path of a specific storage

func IsDirExist 

func IsDirExist(path string) bool

IsDirExist return true if path exist and is a dir, other cases return false

func IsFileExist 

func IsFileExist(file string) bool

IsFileExist return true if file exist and is a regular file, other cases return false

func NewCRCli 

func NewCRCli(kubeconfig string) (versioned.Interface, error)

NewCRCli create a CR cli Interface

func NewEventRecorder 

func NewEventRecorder(kubeCli kubernetes.Interface, source string) record.EventRecorder

NewEventRecorder return the specify source's recoder

func NewKubeAndCRCli 

func NewKubeAndCRCli(kubeconfig string) (kubernetes.Interface, versioned.Interface, error)

NewKubeAndCRCli create both kube cli and CR cli

func NewKubeCli 

func NewKubeCli(kubeconfig string) (kubernetes.Interface, error)

NewKubeCli create a kubeCli Interface

func NormalizeBucketURI 

func NormalizeBucketURI(bucket string) string

NormalizeBucketURI normal bucket URL for rclone, e.g. s3://bucket -> s3:bucket

func ParseRestoreProgress added in v1.4.0 

func ParseRestoreProgress(line string) (step, progress string)

ParseRestoreProgress parse restore progress and return restore step and progress

func RetriableOnAnyError added in v1.3.7 

func RetriableOnAnyError(err error) bool

func RetryOnError added in v1.3.7 

func RetryOnError(ctx context.Context, attempts int, sleep time.Duration,
	retriable func(error) bool, fn func() error) error

RetryOnError allows the caller to retry fn in case the error returned by fn. sleep define the interval between two retries.

func Suffix 

func Suffix(version string) string

Suffix parses the major and minor version from the string and return the suffix

func ValidCmdFlags 

func ValidCmdFlags(cmdPath string, flagSet *pflag.FlagSet)

ValidCmdFlags verify that all flags are set

Types

type BatchDeleteObjectsResult added in v1.2.4 

type BatchDeleteObjectsResult struct {
	Deleted []string
	Errors  []ObjectError
}

func BatchDeleteObjectsConcurrently added in v1.2.4 

func BatchDeleteObjectsConcurrently(ctx context.Context, bucket *blob.Bucket, objs []*blob.ListObject, concurrency int) *BatchDeleteObjectsResult

BatchDeleteObjectsConcurrently delete objects by multiple goroutine concurrently

func BatchDeleteObjectsOfS3 added in v1.2.4 

func BatchDeleteObjectsOfS3(ctx context.Context, s3cli s3iface.S3API, objs []*blob.ListObject, bucket string, prefix string, concurrency int) *BatchDeleteObjectsResult

BatchDeleteObjectsOfS3 delete objects by batch delete api

type GenericOptions 

type GenericOptions struct {
	Namespace string
	// ResourceName can be the name of a backup or restore resource
	ResourceName   string
	TLSClient      bool
	TLSCluster     bool
	SkipClientCA   bool
	Host           string
	Port           int32
	Password       string
	User           string
	TiKVVersion    string
	Mode           string
	SubCommand     string
	CommitTS       string
	TruncateUntil  string
	PitrRestoredTs string
}

GenericOptions contains the generic input arguments to the backup/restore command

func (*GenericOptions) GetDSN 

func (bo *GenericOptions) GetDSN(enabledTLSClient bool) (string, error)

func (*GenericOptions) GetTikvGCLifeTime 

func (bo *GenericOptions) GetTikvGCLifeTime(ctx context.Context, db *sql.DB) (string, error)

func (*GenericOptions) SetTikvGCLifeTime 

func (bo *GenericOptions) SetTikvGCLifeTime(ctx context.Context, db *sql.DB, gcTime string) error

func (*GenericOptions) String 

func (bo *GenericOptions) String() string

type MockDriver added in v1.3.7 

type MockDriver struct {
	driver.Bucket

	Type v1alpha1.BackupStorageType
	// contains filtered or unexported fields
}

MockDriver implement driver.Bucket

func (*MockDriver) As added in v1.3.7 

func (d *MockDriver) As(i interface{}) bool

func (*MockDriver) Delete added in v1.3.7 

func (d *MockDriver) Delete(_ context.Context, key string) error

func (*MockDriver) ErrorCode added in v1.3.7 

func (d *MockDriver) ErrorCode(err error) gcerrors.ErrorCode

func (*MockDriver) ListPaged added in v1.3.7 

func (d *MockDriver) ListPaged(ctx context.Context, opts *driver.ListOptions) (*driver.ListPage, error)

func (*MockDriver) SetDelete added in v1.3.7 

func (d *MockDriver) SetDelete(fn func(key string) error)

func (*MockDriver) SetListPaged added in v1.3.7 

func (d *MockDriver) SetListPaged(objs []*driver.ListObject, rerr error)

type ObjectError added in v1.2.4 

type ObjectError struct {
	Key string
	Err error
}

type PageIterator added in v1.2.4 

type PageIterator struct {
	// contains filtered or unexported fields
}

PageIterator iterates a page of objects via 'ListIterator'

func (*PageIterator) Next added in v1.2.4 

func (i *PageIterator) Next(ctx context.Context, pageSize int) ([]*blob.ListObject, error)

Next list a page of objects

If err == io.EOF, all objects of bucket have been read. If err == nil, a page of objects have been read. Otherwise, err occurs and return objects have been read.

TODO: use blob.ListPage after upgrade gocloud.dev denpendency to 0.21

type StorageBackend added in v1.2.4 

type StorageBackend struct {
	*blob.Bucket
	// contains filtered or unexported fields
}

StorageBackend provide a generic storage backend

func NewStorageBackend added in v1.1.8 

func NewStorageBackend(provider v1alpha1.StorageProvider) (*StorageBackend, error)

NewStorageBackend creates new storage backend, now supports S3/GCS/Azblob/Local

func (*StorageBackend) AsGCS added in v1.2.4 

func (b *StorageBackend) AsGCS() (*storage.Client, bool)

func (*StorageBackend) AsS3 added in v1.2.4 

func (b *StorageBackend) AsS3() (*s3.S3, bool)

func (*StorageBackend) BatchDeleteObjects added in v1.2.4 

func (b *StorageBackend) BatchDeleteObjects(ctx context.Context, objs []*blob.ListObject, opt v1alpha1.BatchDeleteOption) *BatchDeleteObjectsResult

BatchDeleteObjects delete mutli objects

Depending on storage type, it use function 'BatchDeleteObjectsOfS3' or 'BatchDeleteObjectsConcurrently'

func (*StorageBackend) GetBucket added in v1.2.4 

func (b *StorageBackend) GetBucket() string

GetBucket return bucket name

If provider is S3/GCS/Azblob, return bucket. Otherwise return empty string

func (*StorageBackend) GetPrefix added in v1.2.4 

func (b *StorageBackend) GetPrefix() string

GetPrefix return prefix

func (*StorageBackend) ListPage added in v1.2.4 

func (b *StorageBackend) ListPage(opts *blob.ListOptions) *PageIterator

func (*StorageBackend) StorageType added in v1.2.4 

func (b *StorageBackend) StorageType() v1alpha1.BackupStorageType

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.