testhelper

Documentation

Index

• type CsvFileIterator
  • func NewCsvFileIterator(csvPath string) *CsvFileIterator
  • func (ci *CsvFileIterator) Close()
  • func (ci *CsvFileIterator) Fetch() map[string]interface{}
  • func (ci *CsvFileIterator) HasNext() bool
• type DataFlowTester
  • func NewDataFlowTester(t *testing.T, pluginName string, pluginMeta core.PluginMeta) *DataFlowTester
  • func (t *DataFlowTester) FlushTable(tableName string)
  • func (t *DataFlowTester) ImportCsv(csvRelPath string, tableName string)
  • func (t *DataFlowTester) Subtask(subtaskMeta core.SubTaskMeta, taskData interface{})
  • func (t *DataFlowTester) VerifyTable(tableName string, csvRelPath string, pkfields []string, targetfields []string)

Examples

• CsvFileIterator
• DataFlowTester

Types

type CsvFileIterator

type CsvFileIterator struct {
	// contains filtered or unexported fields
}

CsvFileIterator make iterating rows from csv file easier, it reads tuple from csv file and turn it into a `map[string]interface{}` for you.

Example CSV format (exported by dbeaver):

"id","name","json","created_at"
123,"foobar","{""url"": ""https://example.com""}","2022-05-05 09:56:43.438000000"

Example

iter := NewCsvFileIterator("/path/to/foobar.csv")
defer iter.Close()
for iter.HasNext() {
	row := iter.Fetch()
	println(row["name"]) // foobar
	println(row["json"]) // {"url": "https://example.com"}
}

func NewCsvFileIterator

func NewCsvFileIterator(csvPath string) *CsvFileIterator

NewCsvFileIterator create a `*CsvFileIterator` based on path to csv file

func (*CsvFileIterator) Close

func (ci *CsvFileIterator) Close()

Close releases resource

func (*CsvFileIterator) Fetch

func (ci *CsvFileIterator) Fetch() map[string]interface{}

Fetch returns current row

func (*CsvFileIterator) HasNext

func (ci *CsvFileIterator) HasNext() bool

HasNext returns a boolean to indicate whether there was any row to be `Fetch`

type DataFlowTester

type DataFlowTester struct {
	Cfg    *viper.Viper
	Db     *gorm.DB
	T      *testing.T
	Name   string
	Plugin core.PluginMeta
	Log    core.Logger
}

1. Create a folder under your plugin root folder. i.e. `plugins/gitlab/e2e/ to host all your e2e-tests`
2. Create a folder named `tables` to hold all data in `csv` format
3. Create e2e test-cases to cover all possible data-flow routes

Example code:

See [Gitlab Project Data Flow Test](plugins/gitlab/e2e/project_test.go) for detail

DataFlowTester use `N`

Example

var t *testing.T // stub

var gitlab core.PluginMeta
dataflowTester := NewDataFlowTester(t, "gitlab", gitlab)

taskData := &tasks.GitlabTaskData{
	Options: &tasks.GitlabOptions{
		ProjectId: 3472737,
	},
}

// import raw data table
dataflowTester.ImportCsv("./tables/_raw_gitlab_api_projects.csv", "_raw_gitlab_api_project")

// verify extraction
dataflowTester.FlushTable("_tool_gitlab_projects")
dataflowTester.Subtask(tasks.ExtractProjectMeta, taskData)
dataflowTester.VerifyTable(
	"_tool_gitlab_projects",
	"tables/_tool_gitlab_projects.csv",
	[]string{"gitlab_id"},
	[]string{
		"name",
		"description",
		"default_branch",
		"path_with_namespace",
		"web_url",
		"creator_id",
		"visibility",
		"open_issues_count",
		"star_count",
		"forked_from_project_id",
		"forked_from_project_web_url",
		"created_date",
		"updated_date",
		"_raw_data_params",
		"_raw_data_table",
		"_raw_data_id",
		"_raw_data_remark",
	},
)

func NewDataFlowTester

func NewDataFlowTester(t *testing.T, pluginName string, pluginMeta core.PluginMeta) *DataFlowTester

NewDataFlowTester create a *DataFlowTester to help developer test their subtasks data flow

func (*DataFlowTester) FlushTable

func (t *DataFlowTester) FlushTable(tableName string)

FlushTable deletes all records from specified table

func (*DataFlowTester) ImportCsv

func (t *DataFlowTester) ImportCsv(csvRelPath string, tableName string)

ImportCsv imports records from specified csv file into target table, note that existing data would be deleted first.

func (*DataFlowTester) Subtask

func (t *DataFlowTester) Subtask(subtaskMeta core.SubTaskMeta, taskData interface{})

Subtask executes specified subtasks

func (*DataFlowTester) VerifyTable

func (t *DataFlowTester) VerifyTable(tableName string, csvRelPath string, pkfields []string, targetfields []string)

VerifyTable reads rows from csv file and compare with records from database one by one. You must specified the Primary Key Fields with `pkfields` so DataFlowTester could select the exact record from database, as well as which fields to compare with by specifying `targetfields` parameter.