etl

Documentation

Index

• func CreateSearchIndex(dbConn string, searchIndex string) error
• func ImportFile(collection config.GeoSpatialCollection, searchIndex string, filePath string, ...) error
• type Extract
• type Load
• type Transform

Functions

func CreateSearchIndex

func CreateSearchIndex(dbConn string, searchIndex string) error

CreateSearchIndex creates empty search index in target database

func ImportFile

func ImportFile(collection config.GeoSpatialCollection, searchIndex string, filePath string, table config.FeatureTable,
	pageSize int, dbConn string) error

ImportFile import source data into target search index using extract-transform-load principle

Types

type Extract

type Extract interface {

	// Extract raw records from source database to be transformed and loaded into target search index
	Extract(table config.FeatureTable, fields []string, where string, limit int, offset int) ([]t.RawRecord, error)

	// Close connection to source database
	Close()
}

Extract - the 'E' in ETL. Datasource agnostic interface to extract source data.

type Load

type Load interface {

	// Init the target database by creating an empty search index
	Init(index string) error

	// Load records into search index
	Load(records []t.SearchIndexRecord, index string) (int64, error)

	// Close connection to target database
	Close()
}

Load - the 'L' in ETL. Datasource agnostic interface to load data into target database.

type Transform

type Transform interface {

	// Transform each raw record in one or more search records depending on the given configuration
	Transform(records []t.RawRecord, collection config.GeoSpatialCollection) ([]t.SearchIndexRecord, error)
}

Transform - the 'T' in ETL. Logic to transform raw records to search index records