datasource

Documentation

Overview

Package datasource describes the interface that must be satisfied by any external data source accessible through the connector.

Index

• type DataSource
• type DataSourceMock
  • func (*DataSourceMock[T]) DescribeTable(_ context.Context, _ *zap.Logger, _ *api_service_protos.TDescribeTableRequest) (*api_service_protos.TDescribeTableResponse, error)
  • func (m *DataSourceMock[T]) ReadSplit(_ context.Context, _ *zap.Logger, _ *api_service_protos.TReadSplitsRequest, ...)
• type Factory
• type TypeMapper

Types

type DataSource

type DataSource[T paging.Acceptor] interface {
	// DescribeTable returns metadata about a table (or similar entity in non-relational data sources)
	// located within a particular database in a cluster of a certain type.
	DescribeTable(
		ctx context.Context,
		logger *zap.Logger,
		request *api_service_protos.TDescribeTableRequest,
	) (*api_service_protos.TDescribeTableResponse, error)

	// ReadSplit is a main method for reading data from the table.
	ReadSplit(
		ctx context.Context,
		logger *zap.Logger,
		request *api_service_protos.TReadSplitsRequest,
		split *api_service_protos.TSplit,
		sink paging.Sink[T],
	)
}

DataSource is an abstraction over external data storage that is available for data and metadata extraction. All new data sources must implement this interface. The types of data extracted from the data source are parametrized via [T paging.Acceptor] interface.

type DataSourceMock

type DataSourceMock[T paging.Acceptor] struct {
	mock.Mock
}

func (*DataSourceMock[T]) DescribeTable

func (*DataSourceMock[T]) DescribeTable(
	_ context.Context,
	_ *zap.Logger,
	_ *api_service_protos.TDescribeTableRequest,
) (*api_service_protos.TDescribeTableResponse, error)

func (*DataSourceMock[T]) ReadSplit

func (m *DataSourceMock[T]) ReadSplit(
	_ context.Context,
	_ *zap.Logger,
	_ *api_service_protos.TReadSplitsRequest,
	split *api_service_protos.TSplit,
	pagingWriter paging.Sink[T],
)

type Factory

type Factory[T paging.Acceptor] interface {
	Make(
		logger *zap.Logger,
		dataSourceType api_common.EGenericDataSourceKind,
	) (DataSource[T], error)
}

type TypeMapper

type TypeMapper interface {
	SQLTypeToYDBColumn(columnName, typeName string, rules *api_service_protos.TTypeMappingSettings) (*Ydb.Column, error)
}