stream

package
v1.1.0-beta.0...-a878e1f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 19, 2024 License: Apache-2.0, Apache-2.0 Imports: 37 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	DefaultCF = "default"
	WriteCF   = "write"
)

Default columnFamily and write columnFamily

View Source 
const (
	RewriteStatusPreConstructMap = iota // represents construct map status.
	RewriteStatusRestoreKV              // represents restore meta kv status.
)
View Source 
const DATE_FORMAT = "2006-01-02 15:04:05.999999999 -0700"
View Source 
const WildCard = "*"

Variables

This section is empty.

Functions

func BuildObserveDataRanges 

func BuildObserveDataRanges(
	storage kv.Storage,
	filterStr []string,
	tableFilter filter.Filter,
	backupTS uint64,
) ([]kv.KeyRange, error)

BuildObserveDataRanges builds key ranges to observe data KV.

func BuildObserveMetaRange 

func BuildObserveMetaRange() *kv.KeyRange

BuildObserveMetaRange specifies build key ranges to observe meta KV(contains all of metas)

func DecodeKVEntry 

func DecodeKVEntry(buff []byte) (k, v []byte, length uint32, err error)

DecodeKVEntry decodes kv-entry from buff. If error returned is nil, it can get key/value and the length occupied in buff.

func EncodeKVEntry 

func EncodeKVEntry(k, v []byte) []byte

EncodeKVEntry gets a entry-event from input: k, v.

func FastUnmarshalMetaData 

func FastUnmarshalMetaData(
	ctx context.Context,
	s storage.ExternalStorage,
	metaDataWorkerPoolSize uint,
	fn func(path string, rawMetaData []byte) error,
) error

FastUnmarshalMetaData used a 128 worker pool to speed up read metadata content from external_storage.

func FormatDate 

func FormatDate(ts time.Time) string

func FromSchemaMaps 

func FromSchemaMaps(dbMaps []*backuppb.PitrDBMap) map[UpstreamID]*DBReplace

func GetStreamBackupGlobalCheckpointPrefix 

func GetStreamBackupGlobalCheckpointPrefix() string

func GetStreamBackupMetaPrefix 

func GetStreamBackupMetaPrefix() string

func IsMetaDBKey 

func IsMetaDBKey(key []byte) bool

func IsMetaDDLJobHistoryKey 

func IsMetaDDLJobHistoryKey(key []byte) bool

func MaybeDBOrDDLJobHistoryKey 

func MaybeDBOrDDLJobHistoryKey(key []byte) bool

func MaybeQPS 

func MaybeQPS(ctx context.Context, mgr PDInfoProvider) (float64, error)

MaybeQPS get a number like the QPS of last seconds for each store via the prometheus interface. TODO: this is a temporary solution(aha, like in a Hackthon), 

we MUST find a better way for providing this information.

Types

type ContentRef 

type ContentRef struct {
	// contains filtered or unexported fields
}

type DBReplace 

type DBReplace struct {
	Name     string
	DbID     DownstreamID
	TableMap map[UpstreamID]*TableReplace
}

DBReplace specifies database information mapping from up-stream cluster to up-stream cluster.

func NewDBReplace 

func NewDBReplace(name string, newID DownstreamID) *DBReplace

NewDBReplace creates a DBReplace struct.

type DelRangeParams 

type DelRangeParams struct {
	JobID    int64
	ElemID   int64
	StartKey string
	EndKey   string
}

type DownstreamID 

type DownstreamID = int64

type EventIterator 

type EventIterator struct {
	// contains filtered or unexported fields
}

EventIterator is used for reading kv-event from buffer by iterator.

func (*EventIterator) GetError 

func (ei *EventIterator) GetError() error

GetError gets the error in iterator. it returns nil if it has nothing mistaken happened

func (*EventIterator) Key 

func (ei *EventIterator) Key() []byte

Key gets the key in kv-event if valid() == true

func (*EventIterator) Next 

func (ei *EventIterator) Next()

Next specifies the next iterative element.

func (*EventIterator) Valid 

func (ei *EventIterator) Valid() bool

Valid checks whether the iterator is valid.

func (*EventIterator) Value 

func (ei *EventIterator) Value() []byte

Value gets the value in kv-event if valid() == true

type Iterator 

type Iterator interface {
	Next()
	Valid() bool
	Key() []byte
	Value() []byte
	GetError() error
}

Iterator specifies a read iterator Interface.

func NewEventIterator 

func NewEventIterator(buff []byte) Iterator

NewEventIterator creates a Iterator to read kv-event.

type MetadataHelper 

type MetadataHelper struct {
	// contains filtered or unexported fields
}

MetadataHelper make restore/truncate compatible with metadataV1 and metadataV2.

func NewMetadataHelper 

func NewMetadataHelper() *MetadataHelper

func (*MetadataHelper) InitCacheEntry 

func (m *MetadataHelper) InitCacheEntry(path string, ref int)

func (*MetadataHelper) Marshal 

func (*MetadataHelper) Marshal(meta *backuppb.Metadata) ([]byte, error)

For truncate command. Marshal metadata to reupload to external storage. The metadata must be unmarshal by `ParseToMetadataHard`

func (*MetadataHelper) ParseToMetadata 

func (*MetadataHelper) ParseToMetadata(rawMetaData []byte) (*backuppb.Metadata, error)

func (*MetadataHelper) ParseToMetadataHard 

func (*MetadataHelper) ParseToMetadataHard(rawMetaData []byte) (*backuppb.Metadata, error)

Only for deleting, after MetadataV1 is deprecated, we can remove it. Hard means convert to MetaDataV2 deeply.

func (*MetadataHelper) ReadFile 

func (m *MetadataHelper) ReadFile(
	ctx context.Context,
	path string,
	offset uint64,
	length uint64,
	compressionType backuppb.CompressionType,
	storage storage.ExternalStorage,
) ([]byte, error)

type PDInfoProvider 

type PDInfoProvider interface {
	GetPDClient() pd.Client
	GetTLSConfig() *tls.Config
}

type PreDelRangeQuery 

type PreDelRangeQuery struct {
	Sql        string
	ParamsList []DelRangeParams
}

type RawMetaKey 

type RawMetaKey struct {
	Key   []byte
	Field []byte
	Ts    uint64
}

RawMetaKey specified a transaction meta key.

func ParseTxnMetaKeyFrom 

func ParseTxnMetaKeyFrom(txnKey kv.Key) (*RawMetaKey, error)

ParseTxnMetaKeyFrom gets a `RawMetaKey` struct by parsing transaction meta key.

func (*RawMetaKey) EncodeMetaKey 

func (k *RawMetaKey) EncodeMetaKey() kv.Key

EncodeMetaKey Encodes RawMetaKey into a transaction key

func (*RawMetaKey) UpdateField 

func (k *RawMetaKey) UpdateField(field []byte)

UpdateField updates `Field` field in `RawMetaKey` struct.

func (*RawMetaKey) UpdateKey 

func (k *RawMetaKey) UpdateKey(key []byte)

UpdateKey updates `key` field in `RawMetaKey` struct.

func (*RawMetaKey) UpdateTS 

func (k *RawMetaKey) UpdateTS(ts uint64)

UpdateTS updates `Ts` field in `RawMetaKey` struct.

type RawWriteCFValue 

type RawWriteCFValue struct {
	// contains filtered or unexported fields
}

RawWriteCFValue represents the value in write columnFamily. Detail see line: https://github.com/tikv/tikv/blob/release-6.5/components/txn_types/src/write.rs#L70

func (*RawWriteCFValue) EncodeTo 

func (v *RawWriteCFValue) EncodeTo() []byte

EncodeTo encodes the RawWriteCFValue to get encoded value.

func (*RawWriteCFValue) GetShortValue 

func (v *RawWriteCFValue) GetShortValue() []byte

UpdateShortValue gets the shortValue field.

func (*RawWriteCFValue) GetStartTs 

func (v *RawWriteCFValue) GetStartTs() uint64

func (*RawWriteCFValue) GetWriteType 

func (v *RawWriteCFValue) GetWriteType() byte

func (*RawWriteCFValue) HasShortValue 

func (v *RawWriteCFValue) HasShortValue() bool

HasShortValue checks whether short value is stored in write cf.

func (*RawWriteCFValue) IsDelete 

func (v *RawWriteCFValue) IsDelete() bool

IsRollback checks whether the value in cf is a `delete` record.

func (*RawWriteCFValue) IsRollback 

func (v *RawWriteCFValue) IsRollback() bool

IsRollback checks whether the value in cf is a `rollback` record.

func (*RawWriteCFValue) ParseFrom 

func (v *RawWriteCFValue) ParseFrom(data []byte) error

ParseFrom decodes the value to get the struct `RawWriteCFValue`.

func (*RawWriteCFValue) UpdateShortValue 

func (v *RawWriteCFValue) UpdateShortValue(value []byte)

UpdateShortValue updates the shortValue field.

type RewriteStatus 

type RewriteStatus int

type SchemasReplace 

type SchemasReplace struct {
	DbMap map[UpstreamID]*DBReplace

	TiflashRecorder *tiflashrec.TiFlashRecorder
	RewriteTS       uint64        // used to rewrite commit ts in meta kv.
	TableFilter     filter.Filter // used to filter schema/table

	AfterTableRewritten func(deleted bool, tableInfo *model.TableInfo)
	// contains filtered or unexported fields
}

SchemasReplace specifies schemas information mapping from up-stream cluster to up-stream cluster.

func NewSchemasReplace 

func NewSchemasReplace(
	dbMap map[UpstreamID]*DBReplace,
	needConstructIdMap bool,
	tiflashRecorder *tiflashrec.TiFlashRecorder,
	restoreTS uint64,
	tableFilter filter.Filter,
	genID func(ctx context.Context) (int64, error),
	genIDs func(ctx context.Context, n int) ([]int64, error),
	recordDeleteRange func(*PreDelRangeQuery),
) *SchemasReplace

NewSchemasReplace creates a SchemasReplace struct.

func (*SchemasReplace) GetIngestRecorder 

func (sr *SchemasReplace) GetIngestRecorder() *ingestrec.IngestRecorder

func (*SchemasReplace) IsPreConsturctMapStatus 

func (sr *SchemasReplace) IsPreConsturctMapStatus() bool

IsPreConsturctMapStatus checks the status is PreConsturctMap.

func (*SchemasReplace) IsRestoreKVStatus 

func (sr *SchemasReplace) IsRestoreKVStatus() bool

IsRestoreKVStatus checks the status is RestoreKV.

func (*SchemasReplace) NeedConstructIdMap 

func (sr *SchemasReplace) NeedConstructIdMap() bool

func (*SchemasReplace) RewriteKvEntry 

func (sr *SchemasReplace) RewriteKvEntry(e *kv.Entry, cf string) (*kv.Entry, error)

RewriteKvEntry uses to rewrite tableID/dbID in entry.key and entry.value

func (*SchemasReplace) SetPreConstructMapStatus 

func (sr *SchemasReplace) SetPreConstructMapStatus()

SetPreConstructMapStatus sets the PreConstructMap status.

func (*SchemasReplace) SetRestoreKVStatus 

func (sr *SchemasReplace) SetRestoreKVStatus()

SetRestoreKVStatus sets the RestoreKV status.

func (*SchemasReplace) TidySchemaMaps 

func (sr *SchemasReplace) TidySchemaMaps() []*backuppb.PitrDBMap

TidySchemaMaps produces schemas id maps from up-stream to down-stream.

type StatusController 

type StatusController struct {
	// contains filtered or unexported fields
}

StatusController is the controller type (or context type) for the command `stream status`.

func NewStatusController 

func NewStatusController(meta *MetaDataClient, mgr PDInfoProvider, view TaskPrinter) *StatusController

NewStatusContorller make a status controller via some resource accessors.

func (*StatusController) Close 

func (ctl *StatusController) Close() error

func (*StatusController) PrintStatusOfTask 

func (ctl *StatusController) PrintStatusOfTask(ctx context.Context, name string) error

PrintStatusOfTask prints the status of tasks with the name. When the name is *, print all tasks.

type TableReplace 

type TableReplace struct {
	Name         string
	TableID      DownstreamID
	PartitionMap map[UpstreamID]DownstreamID
	IndexMap     map[UpstreamID]DownstreamID
}

TableReplace specifies table information mapping from up-stream cluster to up-stream cluster.

func NewTableReplace 

func NewTableReplace(name string, newID DownstreamID) *TableReplace

NewTableReplace creates a TableReplace struct.

type TaskPrinter 

type TaskPrinter interface {
	AddTask(t TaskStatus)
	PrintTasks()
}

func PrintTaskByTable 

func PrintTaskByTable(c glue.ConsoleOperations) TaskPrinter

PrintTaskByTable make a TaskPrinter, which prints the task by the `Table` implement of the console.

func PrintTaskWithJSON 

func PrintTaskWithJSON(c glue.ConsoleOperations) TaskPrinter

PrintTaskWithJSON make a TaskPrinter, which prints tasks as json to the console directly.

type TaskStatus 

type TaskStatus struct {
	// Info is the stream task information.
	Info backuppb.StreamBackupTaskInfo

	// Checkpoints collects the checkpoints.
	Checkpoints []Checkpoint
	// Total QPS of the task in recent seconds.
	QPS float64
	// Last error reported by the store.
	LastErrors map[uint64]backuppb.StreamBackupError
	// contains filtered or unexported fields
}

func (TaskStatus) GetMinStoreCheckpoint 

func (t TaskStatus) GetMinStoreCheckpoint() Checkpoint

GetCheckpoint calculates the checkpoint of the task.

type UpstreamID 

type UpstreamID = int64

type WriteType 

type WriteType = byte

WriteType defines a write type. 

const (
	WriteTypeLock     WriteType = 'L'
	WriteTypeRollback WriteType = 'R'
	WriteTypeDelete   WriteType = 'D'
	WriteTypePut      WriteType = 'P'
)

WriteType

func WriteTypeFrom 

func WriteTypeFrom(t byte) (WriteType, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.