redo

Documentation

Overview

Package redo provide a redo log for cdc.

There are three types of log file: meta log file, row log file, ddl log file. meta file used to store common.LogMeta info (CheckPointTs, ResolvedTs), atomic updated is guaranteed. A rotated file writer is used for other log files. All files will flush to disk or upload to s3 if enabled every defaultFlushIntervalInMs 1000ms or file size larger than defaultMaxLogSize 64 MB by default. The log file name is formatted as CaptureID_ChangeFeedID_CreateTime_FileType_MaxCommitTSOfAllEventInTheFile.log if safely wrote or end up with .log.tmp is not. meta file name is like CaptureID_ChangeFeedID_meta.meta

Each log file contains batch of model.RowChangedEvent or model.DDLEvent records wrote into different file with defaultMaxLogSize 64 MB. If larger than 64 MB will auto rotated to a new file. A record has a length field and a logical Log data. The length field is a 64-bit packed structure holding the length of the remaining logical Log data in its lower 56 bits and its physical padding in the first three bits of the most significant byte. Each record is 8-byte aligned so that the length field is never torn.

When apply redo log from cli, will select files in the specific dir to open base on the startTs, endTs send from cli or download logs from s3 first is enabled, then sort the event records in each file base on commitTs and startTs, after sorted, the new sort file name should be as CaptureID_ChangeFeedID_CreateTime_FileType_MaxCommitTSOfAllEventInTheFile.log.sort.

Index

• func NewDDLManager(changefeedID model.ChangeFeedID, cfg *config.ConsistentConfig, ...) *ddlManager
• func NewDMLManager(changefeedID model.ChangeFeedID, cfg *config.ConsistentConfig) *dmlManager
• func NewDisabledDDLManager() *ddlManager
• func NewDisabledDMLManager() *dmlManager
• func NewDisabledMetaManager() *metaManager
• func NewMetaManager(changefeedID model.ChangeFeedID, cfg *config.ConsistentConfig, ...) *metaManager
• type DDLManager
• type DMLManager
• type MetaManager

Functions

func NewDDLManager

func NewDDLManager(
	changefeedID model.ChangeFeedID,
	cfg *config.ConsistentConfig, ddlStartTs model.Ts,
) *ddlManager

NewDDLManager creates a new ddl Manager.

func NewDMLManager

func NewDMLManager(changefeedID model.ChangeFeedID,
	cfg *config.ConsistentConfig,
) *dmlManager

NewDMLManager creates a new dml Manager.

func NewDisabledDDLManager

func NewDisabledDDLManager() *ddlManager

NewDisabledDDLManager creates a disabled ddl Manager.

func NewDisabledDMLManager

func NewDisabledDMLManager() *dmlManager

NewDisabledDMLManager creates a disabled dml Manager.

func NewDisabledMetaManager

func NewDisabledMetaManager() *metaManager

NewDisabledMetaManager creates a disabled Meta Manager.

func NewMetaManager

func NewMetaManager(
	changefeedID model.ChangeFeedID, cfg *config.ConsistentConfig, checkpoint model.Ts,
) *metaManager

NewMetaManager creates a new meta Manager.

Types

type DDLManager

type DDLManager interface {
	EmitDDLEvent(ctx context.Context, ddl *model.DDLEvent) error
	UpdateResolvedTs(ctx context.Context, resolvedTs uint64) error
	GetResolvedTs() model.Ts
	// contains filtered or unexported methods
}

DDLManager defines an interface that is used to manage ddl logs in owner.

type DMLManager

type DMLManager interface {
	AddTable(span tablepb.Span, startTs uint64)
	StartTable(span tablepb.Span, startTs uint64)
	RemoveTable(span tablepb.Span)
	UpdateResolvedTs(ctx context.Context, span tablepb.Span, resolvedTs uint64) error
	GetResolvedTs(span tablepb.Span) model.Ts
	EmitRowChangedEvents(
		ctx context.Context,
		span tablepb.Span,
		releaseRowsMemory func(),
		rows ...*model.RowChangedEvent,
	) error
	// contains filtered or unexported methods
}

DMLManager defines an interface that is used to manage dml logs in processor.

type MetaManager

type MetaManager interface {

	// UpdateMeta updates the checkpointTs and resolvedTs asynchronously.
	UpdateMeta(checkpointTs, resolvedTs model.Ts)
	// GetFlushedMeta returns the flushed meta.
	GetFlushedMeta() common.LogMeta
	// Cleanup deletes all redo logs, which are only called from the owner
	// when changefeed is deleted.
	Cleanup(ctx context.Context) error

	// Running return true if the meta manager is running or not.
	Running() bool
	// contains filtered or unexported methods
}

MetaManager defines an interface that is used to manage redo meta and gc logs in owner.