moru

README

package moru

Package moru provides functions to run the models created by goMortgage. Using moru is quite straightforward. The user provides

• A ClickHouse table that has the features required by the model.
• A pointer to the directory of the model created by goMortgage.

The input table, augmented by the model outputs, is saved back to ClickHouse.

Documentation

Overview

Package moru provides functions to run the models created by goMortgage.

Using moru is quite straightforward. The user provides

• A ClickHouse table that has the features required by the model.

• A pointer to the directory of the model created by goMortgage.

The input table, augmented by the model outputs, is saved back to ClickHouse.

Index

• func InsertTable(tableName string, pipe sea.Pipeline, conn *chutils.Connect) error
• func MakeTable(tableName, orderBy string, pipe sea.Pipeline, conn *chutils.Connect) error
• func NewPipe(table, orderBy string, models []ModelDef, startRow, batchSize int, ...) (sea.Pipeline, error)
• func Rows(tableName string, conn *chutils.Connect) int
• func Score(sourceTable, destTable, orderBy string, models []ModelDef, ...) error
• type ModelDef
• type ModelMap

Functions

func InsertTable

func InsertTable(tableName string, pipe sea.Pipeline, conn *chutils.Connect) error

InsertTable inserts the data in pipe into the ClickHouse table tableName. The table must already exist.

• tableName: name of table to insert into (table must exist)
• pipe: pipeline with data to insert
• conn: ClickHouse connector

func MakeTable

func MakeTable(tableName, orderBy string, pipe sea.Pipeline, conn *chutils.Connect) error

MakeTable makes ClickHouse table tableName based on the fields in pipe. MakeTable overwrites the table if it exists.

• tableName: name of ClickHouse table to create
• orderBy: comma-separated values of sourceTable that create a unique key
• pipe: Pipeline containing fields to create for the table
• conn: connector to ClickHouse

func NewPipe

func NewPipe(table, orderBy string, models []ModelDef, startRow, batchSize int, conn *chutils.Connect) (sea.Pipeline, error)

NewPipe creates a new data pipeline.

• table: name of the ClickHouse table with data to calculate model
• orderBy: comma-separated field list that produces a unique key
• models: model location and fields to create
• startRow: first row of table to pull for the pipeline
• batchSize: number of rows of table to pull
• conn: connector to ClickHouse

func Rows

func Rows(tableName string, conn *chutils.Connect) int

Rows returns the number of rows in a ClickHouse table.

• tableName: name of table to for row count
• conn: ClickHouse connector

func Score

func Score(sourceTable, destTable, orderBy string, models []ModelDef, batchSize, nWorker int, conn *chutils.Connect) error

Score creates destTable from sourceTable adding fitted values from one or more models.

• sourceTable: source ClickHouse table with features required by the model
• destTable: created ClickHouse table with sourceTable fields plus model outputs
• orderBy: comma-separated values of sourceTable that create a unique key
• models: model specifications (location, field names and columns)
• batchsize: number of rows to process as a group
• nWorker: number of concurrent processes
• conn: ClickHouse connector

Set sea.Verbose = false to suppress messages during run.

Types

type ModelDef

type ModelDef struct {
	Location string   // directory with the goMortgage model
	Fields   ModelMap // map of fields names to columns of model output
}

ModelDef defines a model and the fields to calculate from it.

type ModelMap

type ModelMap map[string][]int

ModelMap maps the outputs of the model to fields in the output table. Examples:

• map["yhat"] = []int{0}

  creates a new field "yhat" that is the first column of the model output.

• map["yhat01"] = []int{0,1}

  creates a new field "yhat01" that is the sum of the first two columns of the model output.

If the model is a regression, use the first example as a template (i.e. column 0 is the output).