common_http_transform

README

Common HTTP Transform

This is a common library to be used in creating transform services. The MIMIRO data hub allows two kinds of transforms as part of a job. The internal javascript transform and the http transform.

To utilise the HTTP transform requires an external microservice to be running that receives an array of entities and returns a new array based on some processing. That processing typically involves calls to external systems that are used to enrich data about a given entity. Examples would be a call to a weather service to update locations for current weather status.

These HTTP transform services can be written in any language and the endpoint they provide is configurable. However, to remove all the boiler plate code around the web, config, log setup, entity parsing etc, we have created this common library.

The common library can be imported into a standalone transform service module. The transform service then only needs to implement the TransformService interface and provide a factory function to create a new instance. The common library then provides functions to host and expose that core logic service as an HTTP service.

Here is the entry to add to go.mod to include this common library. Note, check the release version against the latest release.

github.com/mimiro-io/common-http-transform v0.1.0

To implement the TransformService interface, follow this example. This example is an identity transform, in that it returns the same as it receives.

// SampleTransform is an example implementation of the Transform interface
// Note: store things like URL endpoints, or security credentials on here (obtained from config)
type SampleTransform struct {
	config  *ct.Config
	logger  ct.Logger
	metrics ct.Metrics
}

// no shutdown required
func (dl *SampleTransform) Stop(_ context.Context) error { return nil }

func (dl *SampleTransform) Transform(ec *egdm.EntityCollection) (*egdm.EntityCollection, ct.TransformError) {
	// Your transform logic goes here
	return ec, nil
}

// NewSampleTransform is a factory function that creates a new instance of the sample transform
func NewSampleTransform(conf *ct.Config, logger ct.Logger, metrics ct.Metrics) (ct.TransformService, error) {
	sampleTransform := &SampleTransform{config: conf, logger: logger, metrics: metrics}
	return sampleTransform, nil
}

// The hosting service monitors the config file for change and will notify with new config
func (dl *SampleTransform) UpdateConfiguration(config *ct.Config) ct.TransformError {
	return nil
}

For reference the TransformService interface is defined as follows:

type Stoppable interface {
	Stop(ctx context.Context) error
}

type TransformService interface {
	Stoppable
	Transform(entityCollection *egdm.EntityCollection) (*egdm.EntityCollection, TransformError)
	UpdateConfiguration(config *Config) TransformError
}

Some Guidance. The NewSampleTransform function is the place to grab any config values and store then in your TransformService struct. Things such as connection strings, or urls, or credentials.

To start up the service the following example main.go illustrates how to start a service. Note that your NewSampleTransform function is passed as the parameter when starting the service runner.

package main

import (
	ct "github.com/mimiro-io/common-http-transform"
	"os"
)

// main function
func main() {
	args := os.Args[1:]
	configFile := args[0]
	serviceRunner := ct.NewServiceRunner(NewSampleTransform)
	serviceRunner.WithConfigLocation(configFile)
	serviceRunner.StartAndWait()
}

A complete sample can be found in the ./sample folder. A template project that uses this common library can be found at ...

Documentation

Index

• func BuildNativeSystemEnvOverrides(envOverrides ...EnvOverride) func(config *Config) error
• type Config
• type EnvOverride
  • func Env(key string, specs ...any) EnvOverride
• type ExternalSystemConfig
• type LayerErrorType
• type LayerServiceConfig
• type Logger
  • func NewLogger(serviceName string, format string, level string) Logger
• type Metrics
• type ServiceRunner
  • func NewServiceRunner(...) *ServiceRunner
  • func (serviceRunner *ServiceRunner) Start() error
  • func (serviceRunner *ServiceRunner) StartAndWait()
  • func (serviceRunner *ServiceRunner) Stop() error
  • func (serviceRunner *ServiceRunner) TransformService() TransformService
  • func (serviceRunner *ServiceRunner) WithConfigLocation(configLocation string) *ServiceRunner
  • func (serviceRunner *ServiceRunner) WithEnrichConfig(enrichConfig func(config *Config) error) *ServiceRunner
• type StatsdMetrics
  • func (sm StatsdMetrics) Gauge(name string, value float64, tags []string, rate int) TransformError
  • func (sm StatsdMetrics) Incr(name string, tags []string, rate int) TransformError
  • func (sm StatsdMetrics) Timing(name string, value time.Duration, tags []string, rate int) TransformError
• type Stoppable
• type TransformError
  • func Err(err error, errType LayerErrorType) TransformError
  • func Errorf(errType LayerErrorType, format string, args ...any) TransformError
• type TransformService
• type TransformServiceFactory

Functions

func BuildNativeSystemEnvOverrides

func BuildNativeSystemEnvOverrides(envOverrides ...EnvOverride) func(config *Config) error

BuildNativeSystemEnvOverrides can be plugged into `WithEnrichConfig`

it takes a variadic parameter list, each of which declares an environment variable
that the layer will try to look up at start, and add to system_config.

Types

type Config

type Config struct {
	ConfigFile           string               // set by service runner
	ExternalSystemConfig ExternalSystemConfig `json:"external_config"`
	LayerServiceConfig   *LayerServiceConfig  `json:"layer_config"`
}

type EnvOverride

type EnvOverride struct {
	EnvVar   string
	ConfKey  string
	Required bool
}

****************************************************************************

func Env

func Env(key string, specs ...any) EnvOverride

Env function to conveniently construct EnvOverride instances

type ExternalSystemConfig

type ExternalSystemConfig map[string]any

type LayerErrorType

type LayerErrorType int

const (
	LayerErrorBadParameter LayerErrorType = iota
	LayerErrorInternal
	LayerNotSupported
)

type LayerServiceConfig

type LayerServiceConfig struct {
	Custom                map[string]any `json:"custom"`
	ServiceName           string         `json:"service_name"`
	Port                  json.Number    `json:"port"`
	ConfigRefreshInterval string         `json:"config_refresh_interval"`
	LogLevel              string         `json:"log_level"`
	LogFormat             string         `json:"log_format"`
	StatsdAgentAddress    string         `json:"statsd_agent_address"`
	StatsdEnabled         bool           `json:"statsd_enabled"`
}

type Logger

type Logger interface {
	Error(message string, args ...any)
	Info(message string, args ...any)
	Debug(message string, args ...any)
	Warn(message string, args ...any)
	With(name string, value string) Logger
}

func NewLogger

func NewLogger(serviceName string, format string, level string) Logger

type Metrics

type Metrics interface {
	Incr(s string, tags []string, i int) TransformError
	Timing(s string, timed time.Duration, tags []string, i int) TransformError
	Gauge(s string, f float64, tags []string, i int) TransformError
}

type ServiceRunner

type ServiceRunner struct {
	// contains filtered or unexported fields
}

func NewServiceRunner

func NewServiceRunner(newTransformService func(config *Config, logger Logger, metrics Metrics) (TransformService, error)) *ServiceRunner

func (*ServiceRunner) Start

func (serviceRunner *ServiceRunner) Start() error

func (*ServiceRunner) StartAndWait

func (serviceRunner *ServiceRunner) StartAndWait()

func (*ServiceRunner) Stop

func (serviceRunner *ServiceRunner) Stop() error

func (*ServiceRunner) TransformService

func (serviceRunner *ServiceRunner) TransformService() TransformService

func (*ServiceRunner) WithConfigLocation

func (serviceRunner *ServiceRunner) WithConfigLocation(configLocation string) *ServiceRunner

func (*ServiceRunner) WithEnrichConfig

func (serviceRunner *ServiceRunner) WithEnrichConfig(enrichConfig func(config *Config) error) *ServiceRunner

type StatsdMetrics

type StatsdMetrics struct {
	// contains filtered or unexported fields
}

func (StatsdMetrics) Gauge

func (sm StatsdMetrics) Gauge(name string, value float64, tags []string, rate int) TransformError

func (StatsdMetrics) Incr

func (sm StatsdMetrics) Incr(name string, tags []string, rate int) TransformError

func (StatsdMetrics) Timing

func (sm StatsdMetrics) Timing(name string, value time.Duration, tags []string, rate int) TransformError

type Stoppable

type Stoppable interface {
	Stop(ctx context.Context) error
}

type TransformError

type TransformError interface {
	error

	Underlying() error
	// contains filtered or unexported methods
}

func Err

func Err(err error, errType LayerErrorType) TransformError

func Errorf

func Errorf(errType LayerErrorType, format string, args ...any) TransformError

type TransformService

type TransformService interface {
	Stoppable
	Transform(entityCollection *egdm.EntityCollection) (*egdm.EntityCollection, TransformError)
	UpdateConfiguration(config *Config) TransformError
}

type TransformServiceFactory

type TransformServiceFactory interface {
	Build(config *Config, logger Logger, metrics Metrics) (TransformService, error)
}