kv

package
v1.1.0-beta.0...-97e7b5c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 13, 2024 License: Apache-2.0 Imports: 44 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ExtraHandleColumnInfo = model.NewExtraHandleColInfo()

ExtraHandleColumnInfo is the column info of extra handle column.

Functions

func ClearRow 

func ClearRow(row encode.Row)

ClearRow recycles the memory used by the row.

func GetActualDatum 

func GetActualDatum(encoder encode.Encoder, col *table.Column, rowID int64,
	inputDatum *types.Datum) (types.Datum, error)

GetActualDatum export getActualDatum function.

func GetAutoRecordID 

func GetAutoRecordID(d types.Datum, target *types.FieldType) int64

GetAutoRecordID returns the record ID for an auto-increment field. get record value for auto-increment field

See: 

https://github.com/pingcap/tidb/blob/47f0f15b14ed54fc2222f3e304e29df7b05e6805/executor/insert_common.go#L781-L852

func GetEncoderIncrementalID 

func GetEncoderIncrementalID(encoder encode.Encoder, id int64) int64

GetEncoderIncrementalID return Auto increment id.

func IsAutoIncCol 

func IsAutoIncCol(colInfo *model.ColumnInfo) bool

IsAutoIncCol return true if the column is auto increment column.

func MakeRowFromKvPairs 

func MakeRowFromKvPairs(pairs []common.KvPair) encode.Row

MakeRowFromKvPairs converts a KvPair slice into a Row instance. This is mainly used for testing only. The resulting Row instance should only be used for the importer backend.

func MakeRowsFromKvPairs 

func MakeRowsFromKvPairs(pairs []common.KvPair) encode.Rows

MakeRowsFromKvPairs converts a KvPair slice into a Rows instance. This is mainly used for testing only. The resulting Rows instance should only be used for the importer backend.

func NewPanickingAllocators 

func NewPanickingAllocators(sepAutoInc bool, base int64) autoid.Allocators

NewPanickingAllocators creates a PanickingAllocator shared by all allocation types. we use this to collect the max id(either _tidb_rowid or auto_increment id or auto_random) used during import, and we will use this info to do ALTER TABLE xxx AUTO_RANDOM_BASE or AUTO_INCREMENT on post-process phase. TODO: support save all bases in checkpoint.

func NewTableKVEncoder 

func NewTableKVEncoder(
	config *encode.EncodingConfig,
	metrics *metric.Metrics,
) (encode.Encoder, error)

NewTableKVEncoder creates a new tableKVEncoder.

func Row2KvPairs 

func Row2KvPairs(row encode.Row) []common.KvPair

Row2KvPairs converts a Row instance constructed from MakeRowFromKvPairs back into a slice of KvPair. This method panics if the Row is not constructed in such way.

func Rows2KvPairs 

func Rows2KvPairs(rows encode.Rows) []common.KvPair

Rows2KvPairs converts a Rows instance constructed from MakeRowsFromKvPairs back into a slice of KvPair. This method panics if the Rows is not constructed in such way.

Types

type AutoIDConverterFn 

type AutoIDConverterFn func(int64) int64

AutoIDConverterFn is a function to convert auto id.

type BaseKVEncoder 

type BaseKVEncoder struct {
	GenCols    []GeneratedCol
	SessionCtx *Session

	Columns         []*table.Column
	AutoRandomColID int64
	// convert auto id for shard rowid or auto random id base on row id generated by lightning
	AutoIDFn AutoIDConverterFn
	// contains filtered or unexported fields
}

BaseKVEncoder encodes a row into a KV pair.

func NewBaseKVEncoder 

func NewBaseKVEncoder(config *encode.EncodingConfig) (*BaseKVEncoder, error)

NewBaseKVEncoder creates a new BaseKVEncoder.

func (*BaseKVEncoder) AddRecord 

func (e *BaseKVEncoder) AddRecord(record []types.Datum) (kv.Handle, error)

AddRecord adds a record into encoder

func (*BaseKVEncoder) EvalGeneratedColumns 

func (e *BaseKVEncoder) EvalGeneratedColumns(record []types.Datum,
	cols []*table.Column) (errCol *model.ColumnInfo, err error)

EvalGeneratedColumns evaluates the generated columns.

func (*BaseKVEncoder) GetOrCreateRecord 

func (e *BaseKVEncoder) GetOrCreateRecord() []types.Datum

GetOrCreateRecord returns a record slice from the cache if possible, otherwise creates a new one.

func (*BaseKVEncoder) IsAutoRandomCol 

func (e *BaseKVEncoder) IsAutoRandomCol(col *model.ColumnInfo) bool

IsAutoRandomCol checks if the column is auto random column.

func (*BaseKVEncoder) LogEvalGenExprFailed 

func (e *BaseKVEncoder) LogEvalGenExprFailed(row []types.Datum, colInfo *model.ColumnInfo, err error) error

LogEvalGenExprFailed logs the error when evaluating the generated column expression failed.

func (*BaseKVEncoder) LogKVConvertFailed 

func (e *BaseKVEncoder) LogKVConvertFailed(row []types.Datum, j int, colInfo *model.ColumnInfo, err error) error

LogKVConvertFailed logs the error when converting a row to KV pair failed.

func (*BaseKVEncoder) ProcessColDatum 

func (e *BaseKVEncoder) ProcessColDatum(col *table.Column, rowID int64, inputDatum *types.Datum) (types.Datum, error)

ProcessColDatum processes the datum of a column.

func (*BaseKVEncoder) Record2KV 

func (e *BaseKVEncoder) Record2KV(record, originalRow []types.Datum, rowID int64) (*Pairs, error)

Record2KV converts a row into a KV pair.

func (*BaseKVEncoder) TableAllocators 

func (e *BaseKVEncoder) TableAllocators() autoid.Allocators

TableAllocators returns the allocators of the table

func (*BaseKVEncoder) TableMeta 

func (e *BaseKVEncoder) TableMeta() *model.TableInfo

TableMeta returns the meta of the table

func (*BaseKVEncoder) TruncateWarns 

func (e *BaseKVEncoder) TruncateWarns()

TruncateWarns resets the warnings in session context.

type BytesBuf 

type BytesBuf struct {
	// contains filtered or unexported fields
}

BytesBuf bytes buffer.

type GeneratedCol 

type GeneratedCol struct {
	// index of the column in the table
	Index int
	Expr  expression.Expression
}

GeneratedCol generated column info.

func CollectGeneratedColumns 

func CollectGeneratedColumns(se *Session, meta *model.TableInfo, cols []*table.Column) ([]GeneratedCol, error)

CollectGeneratedColumns collects all expressions required to evaluate the results of all generated columns. The returning slice is in evaluation order.

type GroupedPairs 

type GroupedPairs map[int64][]common.KvPair

GroupedPairs is a map from index ID to KvPairs.

func (GroupedPairs) Clear 

func (GroupedPairs) Clear() encode.Rows

Clear implements the encode.Rows interface. It just satisfies the type system and should never be called.

func (GroupedPairs) SplitIntoChunks 

func (GroupedPairs) SplitIntoChunks(int) []encode.Rows

SplitIntoChunks implements the encode.Rows interface. It just satisfies the type system and should never be called.

type MemBuf 

type MemBuf struct {
	sync.Mutex
	kv.MemBuffer
	// contains filtered or unexported fields
}

MemBuf used to store the data in memory.

func (*MemBuf) AllocateBuf 

func (mb *MemBuf) AllocateBuf(size int)

AllocateBuf allocates a byte buffer.

func (*MemBuf) Cleanup 

func (*MemBuf) Cleanup(_ kv.StagingHandle)

Cleanup the resources referenced by the StagingHandle. If the changes are not published by `Release`, they will be discarded.

func (*MemBuf) Delete 

func (*MemBuf) Delete(_ kv.Key) error

Delete implements the kv.MemBuffer interface.

func (*MemBuf) GetLocal 

func (mb *MemBuf) GetLocal(ctx context.Context, key []byte) ([]byte, error)

GetLocal implements the kv.MemBuffer interface.

func (*MemBuf) Recycle 

func (mb *MemBuf) Recycle(buf *BytesBuf)

Recycle recycles the byte buffer.

func (*MemBuf) Release 

func (*MemBuf) Release(_ kv.StagingHandle)

Release publish all modifications in the latest staging buffer to upper level.

func (*MemBuf) Set 

func (mb *MemBuf) Set(k kv.Key, v []byte) error

Set sets the key-value pair.

func (*MemBuf) SetWithFlags 

func (mb *MemBuf) SetWithFlags(k kv.Key, v []byte, _ ...kv.FlagsOp) error

SetWithFlags implements the kv.MemBuffer interface.

func (*MemBuf) Size 

func (mb *MemBuf) Size() int

Size returns sum of keys and values length.

func (*MemBuf) Staging 

func (*MemBuf) Staging() kv.StagingHandle

Staging creates a new staging buffer.

type Pairs 

type Pairs struct {
	Pairs    []common.KvPair
	BytesBuf *BytesBuf
	MemBuf   *MemBuf
}

Pairs implements the Encoder interface.

func (*Pairs) ClassifyAndAppend 

func (kvs *Pairs) ClassifyAndAppend(
	data *encode.Rows,
	dataChecksum *verification.KVChecksum,
	indices *encode.Rows,
	indexChecksum *verification.KVChecksum,
)

ClassifyAndAppend separates the key-value pairs into data and index key-value pairs.

func (*Pairs) Clear 

func (kvs *Pairs) Clear() encode.Rows

Clear clears the key-value pairs.

func (*Pairs) Size 

func (kvs *Pairs) Size() uint64

Size returns the total size of the key-value pairs.

type RowArrayMarshaller 

type RowArrayMarshaller []types.Datum

RowArrayMarshaller wraps a slice of types.Datum for logging the content into zap.

func (RowArrayMarshaller) MarshalLogArray 

func (row RowArrayMarshaller) MarshalLogArray(encoder zapcore.ArrayEncoder) error

MarshalLogArray implements the zapcore.ArrayMarshaler interface

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session is used to provide context for lightning.

func GetEncoderSe 

func GetEncoderSe(encoder encode.Encoder) *Session

GetEncoderSe return session.

func GetSession4test 

func GetSession4test(encoder encode.Encoder) *Session

GetSession4test is only used for test.

func NewSession 

func NewSession(options *encode.SessionOptions, logger log.Logger) (*Session, error)

NewSession creates a new Session.

func (*Session) Close 

func (s *Session) Close()

Close closes the session

func (*Session) GetColumnSize 

func (s *Session) GetColumnSize(tblID int64) (ret map[int64]int64)

GetColumnSize returns the size of each column.

func (*Session) GetExprCtx 

func (s *Session) GetExprCtx() exprctx.ExprContext

GetExprCtx returns the expression context

func (*Session) GetTableCtx 

func (s *Session) GetTableCtx() tblctx.MutateContext

GetTableCtx returns the table MutateContext.

func (*Session) SetUserVarVal 

func (s *Session) SetUserVarVal(name string, dt types.Datum)

SetUserVarVal sets the value of a user variable.

func (*Session) TakeKvPairs 

func (s *Session) TakeKvPairs() *Pairs

TakeKvPairs returns the current Pairs and resets the buffer.

func (*Session) Txn 

func (s *Session) Txn() kv.Transaction

Txn returns the internal txn.

func (*Session) UnsetUserVar 

func (s *Session) UnsetUserVar(varName string)

UnsetUserVar unsets a user variable.

type TableKVDecoder 

type TableKVDecoder struct {
	// contains filtered or unexported fields
}

TableKVDecoder is a KVDecoder that decodes the key-value pairs of a table.

func NewTableKVDecoder 

func NewTableKVDecoder(
	tbl table.Table,
	tableName string,
	options *encode.SessionOptions,
	logger log.Logger,
) (*TableKVDecoder, error)

NewTableKVDecoder creates a new TableKVDecoder.

func (*TableKVDecoder) DecodeHandleFromIndex 

func (t *TableKVDecoder) DecodeHandleFromIndex(indexInfo *model.IndexInfo, key, value []byte) (kv.Handle, error)

DecodeHandleFromIndex implements KVDecoder.DecodeHandleFromIndex.

func (*TableKVDecoder) DecodeHandleFromRowKey 

func (*TableKVDecoder) DecodeHandleFromRowKey(key []byte) (kv.Handle, error)

DecodeHandleFromRowKey implements KVDecoder.DecodeHandleFromRowKey.

func (*TableKVDecoder) DecodeRawRowData 

func (t *TableKVDecoder) DecodeRawRowData(h kv.Handle, value []byte) ([]types.Datum, map[int64]types.Datum, error)

DecodeRawRowData decodes raw row data into a datum slice and a (columnID:columnValue) map.

func (*TableKVDecoder) DecodeRawRowDataAsStr 

func (t *TableKVDecoder) DecodeRawRowDataAsStr(h kv.Handle, value []byte) (res string)

DecodeRawRowDataAsStr decodes raw row data into a string.

func (*TableKVDecoder) IterRawIndexKeys 

func (t *TableKVDecoder) IterRawIndexKeys(h kv.Handle, rawRow []byte, fn func([]byte) error) error

IterRawIndexKeys generates the raw index keys corresponding to the raw row, and then iterate them using `fn`. The input buffer will be reused.

func (*TableKVDecoder) Name 

func (t *TableKVDecoder) Name() string

Name implements KVDecoder.Name.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.