lightning

Documentation

Index

• func CheckpointRemove(ctx context.Context, cfg *config.Config, tableName string) error
• func CleanupMetas(ctx context.Context, cfg *config.Config, tableName string) error
• func SwitchMode(ctx context.Context, cfg *config.Config, tls *common.TLS, mode string) error
• type Lightning
  • func New(globalCfg *config.GlobalConfig) *Lightning
  • func (l *Lightning) GoServe() error
  • func (l *Lightning) Metrics() *metric.Metrics
  • func (l *Lightning) RunOnce(taskCtx context.Context, taskCfg *config.Config, glue glue.Glue) error
  • func (l *Lightning) RunOnceWithOptions(taskCtx context.Context, taskCfg *config.Config, opts ...Option) error
  • func (l *Lightning) RunServer() error
  • func (l *Lightning) Status() (finished int64, total int64)
  • func (l *Lightning) Stop()
• type Option
  • func WithCheckpointStorage(s storage.ExternalStorage, cpName string) Option
  • func WithDumpFileStorage(s storage.ExternalStorage) Option
  • func WithDupIndicator(b *atomic.Bool) Option
  • func WithGlue(g glue.Glue) Option
  • func WithLogger(logger *zap.Logger) Option
  • func WithPromFactory(f promutil.Factory) Option
  • func WithPromRegistry(r promutil.Registry) Option

Functions

func CheckpointRemove

func CheckpointRemove(ctx context.Context, cfg *config.Config, tableName string) error

func CleanupMetas

func CleanupMetas(ctx context.Context, cfg *config.Config, tableName string) error

func SwitchMode

func SwitchMode(ctx context.Context, cfg *config.Config, tls *common.TLS, mode string) error

Types

type Lightning

type Lightning struct {
	// contains filtered or unexported fields
}

func New

func New(globalCfg *config.GlobalConfig) *Lightning

func (*Lightning) GoServe

func (l *Lightning) GoServe() error

func (*Lightning) Metrics

func (l *Lightning) Metrics() *metric.Metrics

Metrics returns the metrics of lightning. it's inited during `run`, so might return nil.

func (*Lightning) RunOnce

func (l *Lightning) RunOnce(taskCtx context.Context, taskCfg *config.Config, glue glue.Glue) error

RunOnce is used by binary lightning and host when using lightning as a library.

• for binary lightning, taskCtx could be context.Background which means taskCtx wouldn't be canceled directly by its cancel function, but only by Lightning.Stop or HTTP DELETE using l.cancel. and glue could be nil to let lightning use a default glue later.
• for lightning as a library, taskCtx could be a meaningful context that get canceled outside, and glue could be a caller implemented glue.

deprecated: use RunOnceWithOptions instead.

func (*Lightning) RunOnceWithOptions

func (l *Lightning) RunOnceWithOptions(taskCtx context.Context, taskCfg *config.Config, opts ...Option) error

RunOnceWithOptions is used by binary lightning and host when using lightning as a library.

• for binary lightning, taskCtx could be context.Background which means taskCtx wouldn't be canceled directly by its cancel function, but only by Lightning.Stop or HTTP DELETE using l.cancel. No need to set Options
• for lightning as a library, taskCtx could be a meaningful context that get canceled outside, and there Options may be used:
• WithGlue: set a caller implemented glue. Otherwise, lightning will use a default glue later.
• WithDumpFileStorage: caller has opened an external storage for lightning. Otherwise, lightning will open a storage by config
• WithCheckpointStorage: caller has opened an external storage for lightning and want to save checkpoint in it. Otherwise, lightning will save checkpoint by the Checkpoint.DSN in config

func (*Lightning) RunServer

func (l *Lightning) RunServer() error

func (*Lightning) Status

func (l *Lightning) Status() (finished int64, total int64)

Status return the sum size of file which has been imported to TiKV and the total size of source file.

func (*Lightning) Stop

func (l *Lightning) Stop()

type Option

type Option func(*options)

func WithCheckpointStorage

func WithCheckpointStorage(s storage.ExternalStorage, cpName string) Option

WithCheckpointStorage sets the checkpoint name in external storage to a lightning task. Typically, the checkpoint name is set when lightning is integrated with dataflow engine by DM.

func WithDumpFileStorage

func WithDumpFileStorage(s storage.ExternalStorage) Option

WithDumpFileStorage sets the external storage to a lightning task. Typically, the external storage is set when lightning is integrated with dataflow engine by DM.

func WithDupIndicator

func WithDupIndicator(b *atomic.Bool) Option

WithDupIndicator sets a *bool to indicate duplicate detection has found duplicate data.

func WithGlue

func WithGlue(g glue.Glue) Option

WithGlue sets the glue to a lightning task. Typically, the glue is set when lightning is integrated with a TiDB.

func WithLogger

func WithLogger(logger *zap.Logger) Option

WithLogger sets the logger to a lightning task.

func WithPromFactory

func WithPromFactory(f promutil.Factory) Option

WithPromFactory sets the prometheus factory to a lightning task.

func WithPromRegistry

func WithPromRegistry(r promutil.Registry) Option

WithPromRegistry sets the prometheus registry to a lightning task. The task metrics will be registered to the registry before the task starts and unregistered after the task ends.