e2ehelper

Documentation

Index

• type DataFlowTester
  • func NewDataFlowTester(t *testing.T, pluginName string, pluginMeta core.PluginMeta) *DataFlowTester
  • func (t *DataFlowTester) CreateSnapshot(dst schema.Tabler, csvRelPath string, pkfields []string, targetfields []string)
  • func (t *DataFlowTester) FlushRawTable(rawTableName string)
  • func (t *DataFlowTester) FlushTabler(dst schema.Tabler)
  • func (t *DataFlowTester) ImportCsvIntoRawTable(csvRelPath string, tableName string)
  • func (t *DataFlowTester) Subtask(subtaskMeta core.SubTaskMeta, taskData interface{})
  • func (t *DataFlowTester) VerifyTable(dst schema.Tabler, csvRelPath string, pkfields []string, targetfields []string)

Examples

• DataFlowTester

Types

type DataFlowTester

type DataFlowTester struct {
	Cfg    *viper.Viper
	Db     *gorm.DB
	Dal    dal.Dal
	T      *testing.T
	Name   string
	Plugin core.PluginMeta
	Log    core.Logger
}

1. Create a folder under your plugin root folder. i.e. `plugins/gitlab/e2e/ to host all your e2e-tests`
2. Create a folder named `tables` to hold all data in `csv` format
3. Create e2e test-cases to cover all possible data-flow routes

Example code:

See [Gitlab Project Data Flow Test](plugins/gitlab/e2e/project_test.go) for detail

DataFlowTester use `N`

Example

var t *testing.T // stub

var gitlab core.PluginMeta
dataflowTester := NewDataFlowTester(t, "gitlab", gitlab)

taskData := &tasks.GitlabTaskData{
	Options: &tasks.GitlabOptions{
		ProjectId: 3472737,
	},
}

// import raw data table
dataflowTester.ImportCsv("./tables/_raw_gitlab_api_projects.csv", "_raw_gitlab_api_project")

// verify extraction
dataflowTester.FlushTable("_tool_gitlab_projects")
dataflowTester.Subtask(tasks.ExtractProjectMeta, taskData)
dataflowTester.VerifyTable(
	"_tool_gitlab_projects",
	"tables/_tool_gitlab_projects.csv",
	[]string{"gitlab_id"},
	[]string{
		"name",
		"description",
		"default_branch",
		"path_with_namespace",
		"web_url",
		"creator_id",
		"visibility",
		"open_issues_count",
		"star_count",
		"forked_from_project_id",
		"forked_from_project_web_url",
		"created_date",
		"updated_date",
		"_raw_data_params",
		"_raw_data_table",
		"_raw_data_id",
		"_raw_data_remark",
	},
)

func NewDataFlowTester

func NewDataFlowTester(t *testing.T, pluginName string, pluginMeta core.PluginMeta) *DataFlowTester

NewDataFlowTester create a *DataFlowTester to help developer test their subtasks data flow

func (*DataFlowTester) CreateSnapshot

func (t *DataFlowTester) CreateSnapshot(dst schema.Tabler, csvRelPath string, pkfields []string, targetfields []string)

CreateSnapshot reads rows from database and write them into .csv file.

func (*DataFlowTester) FlushRawTable

func (t *DataFlowTester) FlushRawTable(rawTableName string)

MigrateRawTableAndFlush migrate table and deletes all records from specified table

func (*DataFlowTester) FlushTabler

func (t *DataFlowTester) FlushTabler(dst schema.Tabler)

FlushTabler migrate table and deletes all records from specified table

func (*DataFlowTester) ImportCsvIntoRawTable

func (t *DataFlowTester) ImportCsvIntoRawTable(csvRelPath string, tableName string)

ImportCsvIntoRawTable imports records from specified csv file into target table, note that existing data would be deleted first.

func (*DataFlowTester) Subtask

func (t *DataFlowTester) Subtask(subtaskMeta core.SubTaskMeta, taskData interface{})

Subtask executes specified subtasks

func (*DataFlowTester) VerifyTable

func (t *DataFlowTester) VerifyTable(dst schema.Tabler, csvRelPath string, pkfields []string, targetfields []string)

VerifyTable reads rows from csv file and compare with records from database one by one. You must specified the Primary Key Fields with `pkfields` so DataFlowTester could select the exact record from database, as well as which fields to compare with by specifying `targetfields` parameter.