batch

README

Concepts

• resource: a wrapper for data (in the form of bytes or file) which resides in local machine or remotely
• reader: a reader reads data from resource and unmarshalls it according to certain format
• writer: a writer marshalls data into certain format and writes to resource
• transformer: there are a few types of transformers
  • record extractor: a record extractor extracts records from input, e.g. extracts Transactions from the response of Get Transactions API
  • field transformer: a field transformer transforms a field (a cell in CSV) from one format to another

graph LR
   Resource --> Reader --> Transformer --> Processor("(Processor)") --> Writer --> Transformer --> Resource

• Processors are business specific.

Resource

A resource may reside locally or remotely and can be accessed via certain protocol, such as HTTP or FTP.

Supported resources:

• memory resource: data in memory
• local resource: a file on local file system
• http resource: resource can be accessed via HTTP (GET)
• sftp resource: resource can be accessed via Sftp

Readers

Supported readers:

• csv reader
• json reader

Writers

Supported writer:

• csv writer

Usage

Example

• csv reader + local resource: read a CSV file from local file system
• csv reader + sftp resource: read a CSV file from remote file system via Sftp
• json reader + http resource: read JSON data from remote system via HTTP
• csv writer + memory resource: format a report as CSV and send it via email, without storing data in file system

Documentation

Index

• Constants
• type ConnectionError
  • func (e *ConnectionError) Error() string
  • func (e *ConnectionError) Unwrap() error
• type ConnectionOperationType
• type IllegalArgumentError
  • func (e *IllegalArgumentError) Error() string
• type InvalidStatusError
  • func (e *InvalidStatusError) Error() string
• type IoError
  • func (e *IoError) Error() string
  • func (e *IoError) Unwrap() error
• type IoOperationType
• type OpenCloser
• type Reader
• type Resource
• type ResourceReader
• type ResourceWriter
• type Writer

Constants

const (
	ConnOpen  ConnectionOperationType = "open"
	ConnClose ConnectionOperationType = "close"

	IoOpen  IoOperationType = "open"
	IoClose IoOperationType = "close"
	IoRead  IoOperationType = "read"
	IoWrite IoOperationType = "write"
)

Types

type ConnectionError

type ConnectionError struct {
	Operation ConnectionOperationType
	Address   string
	Err       error
}

func (*ConnectionError) Error

func (e *ConnectionError) Error() string

func (*ConnectionError) Unwrap

func (e *ConnectionError) Unwrap() error

type ConnectionOperationType

type ConnectionOperationType string

type IllegalArgumentError

type IllegalArgumentError struct {
	Name  string
	Value any
}

func (*IllegalArgumentError) Error

func (e *IllegalArgumentError) Error() string

type InvalidStatusError

type InvalidStatusError struct {
	StatusCode int
}

func (*InvalidStatusError) Error

func (e *InvalidStatusError) Error() string

type IoError

type IoError struct {
	Operation IoOperationType
	Resource  string
	Err       error
}

func (*IoError) Error

func (e *IoError) Error() string

func (*IoError) Unwrap

func (e *IoError) Unwrap() error

type IoOperationType

type IoOperationType string

type OpenCloser

type OpenCloser interface {
	Open(ctx context.Context) error
	Close(ctx context.Context) error
}

type Reader

type Reader interface {
	OpenCloser

	Read(ctx context.Context, record any) error
}

type Resource

type Resource interface {
	OpenCloser
	GetID() string
}

type ResourceReader

type ResourceReader interface {
	Resource
	io.Reader
}

type ResourceWriter

type ResourceWriter interface {
	Resource
	io.Writer
}

type Writer

type Writer interface {
	OpenCloser

	Write(ctx context.Context, record any) error
}