chiv

README

---

Archive relational data to Amazon S3.

---

This project provides the chiv package and a simple CLI wrapper of the same name. It requires Go 1.13.

Use chiv to download relational data from a database and format it before uploading to Amazon S3. Built-in formats include CSV, YAML and JSON. Custom formats are also supported through the Formatter interface.

Getting Started

import "gavincabbage.com/chiv"

Provide a database and upload manager to upload a table to an S3 bucket in the default CSV format.

db, _ := sql.Open(config.driver, config.url)

client := s3.New(session.NewSessionWithOptions(session.Options{}))
uploader := s3manager.NewUploaderWithClient(client)

chiv.Archive(db, uploader, "table", "bucket")

Use options to configure the archival format, upload key, null placeholder, etc.

chiv.Archive(db, uploader, "table", "bucket"
    chiv.WithFormat(chiv.JSON),
    chiv.WithKey("2019/september/monthly_archive.json"),
    chiv.WithNull("empty"),
)

For multiple uploads using the same database and S3 clients, construct an Archiver. Options provided during construction of an Archiver can be overridden in individual archival calls.

a := chiv.NewArchiver(db, uploader, chiv.WithFormat(chiv.YAML))
a.Archive("first_table", "bucket")
a.Archive("second_table", "bucket")
a.Archive("second_table", "bucket", chiv.WithFormat(chiv.JSON), chiv.WithKey("second_table.json"))

Custom queries can be archived using the ArchiveRows family of functions.

rows, _ := db.Exec("SELECT * FROM table and JOIN all the things...")

chiv.ArchiveRows(rows, uploader, "bucket")

Context-aware versions are also provided, e.g. ArchiveWithContext, ArchiveRowsWithContext, etc.

See the unit and integration tests for additional examples.

Custom Formats

Custom formats can be used by implementing the FormatterFunc and Formatter interfaces. The optional Extensioner interface can be implemented to allow a Formatter to provide a default extension.

See the three built-in formats for examples.

CLI

A simple CLI wrapping the package is also included.

NAME:
   chiv - Archive relational data to Amazon S3

USAGE:
   chiv [flags...]

VERSION:
   vX.Y.Z

GLOBAL OPTIONS:
   --database value, -d value   database connection string [$DATABASE_URL]
   --table value, -t value      database table to archive
   --bucket value, -b value     upload S3 bucket name
   --driver value, -r value     database driver type: postgres or mysql (default: "postgres")
   --columns value, -c value    database columns to archive, comma-separated
   --format value, -f value     upload format: csv, yaml or json (default: "csv")
   --key value, -k value        upload key
   --extension value, -e value  upload extension
   --null value, -n value       upload null value
   --help, -h                   show usage details
   --version, -v                print the version

Design

This package ties together three components of what is essentially an ETL operation:

• Extract database rows with a Database or Rows
  • Database may be a *sql.DB, *sql.Conn, *sql.Tx, etc.
• Transform data into an upload format with a Formatter
• Load the formatted data into S3 with a Uploader
  • Uploader is typically an *s3manager.Uploader

An io.Pipe is used to guarantee a cap on total memory usage by chiv itself. A benchmark is provided to profile the package's memory usage. Total memory usage of the archival process can be further controlled by configuring the S3 upload manager.

Testing & Development

Tests are segregated by build tags.

Run unit tests:

go test -tags=unit ./...

Use docker-compose to run the full suite of unit and integration tests:

docker-compose up --exit-code-from test --build

Run benchmarks:

go test -v -p 1 -tags=benchmark,unit -run=Benchmark -benchmem -bench=.

This project uses GitHub Actions for CI. Linting, unit tests, integration tests and benchmarks are run for each push or pull request.

Tagged releases are built automatically by GoReleaser.

Contributing

Contributions in the form of comments, issues or pull requests are very welcome.

If you use this package I would love to hear from you!

Documentation

Overview

Package chiv archives relational data to Amazon S3.

Index

• func Archive(db Database, s3 Uploader, table, bucket string, options ...Option) error
• func ArchiveRows(rows Rows, s3 Uploader, bucket string, options ...Option) error
• func ArchiveRowsWithContext(ctx context.Context, rows Rows, s3 Uploader, bucket string, options ...Option) error
• func ArchiveWithContext(ctx context.Context, db Database, s3 Uploader, table, bucket string, ...) error
• type Archiver
  • func NewArchiver(db Database, s3 Uploader, options ...Option) *Archiver
  • func (a *Archiver) Archive(table, bucket string, options ...Option) error
  • func (a *Archiver) ArchiveRows(rows Rows, bucket string, options ...Option) (err error)
  • func (a *Archiver) ArchiveRowsWithContext(ctx context.Context, rows Rows, bucket string, options ...Option) (err error)
  • func (a *Archiver) ArchiveWithContext(ctx context.Context, table, bucket string, options ...Option) (err error)
• type Column
• type Database
• type Extensioner
• type Formatter
  • func CSV(w io.Writer, columns []Column) Formatter
  • func JSON(w io.Writer, columns []Column) Formatter
  • func YAML(w io.Writer, columns []Column) Formatter
• type FormatterFunc
• type Option
  • func WithColumns(c ...string) Option
  • func WithExtension(s string) Option
  • func WithFormat(f FormatterFunc) Option
  • func WithKey(s string) Option
  • func WithNull(s string) Option
• type Rows
• type Uploader

Functions

func Archive

func Archive(db Database, s3 Uploader, table, bucket string, options ...Option) error

Archive a database table to S3.

func ArchiveRows

func ArchiveRows(rows Rows, s3 Uploader, bucket string, options ...Option) error

ArchiveRows to S3.

func ArchiveRowsWithContext

func ArchiveRowsWithContext(ctx context.Context, rows Rows, s3 Uploader, bucket string, options ...Option) error

ArchiveRowsWithContext is like ArchiveRows, with context.

func ArchiveWithContext

func ArchiveWithContext(ctx context.Context, db Database, s3 Uploader, table, bucket string, options ...Option) error

ArchiveWithContext is like Archive, with context.

Types

type Archiver

type Archiver struct {
	// contains filtered or unexported fields
}

Archiver archives database tables to Amazon S3.

func NewArchiver

func NewArchiver(db Database, s3 Uploader, options ...Option) *Archiver

NewArchiver constructs an archiver with the given Database, S3 uploader and options. Options set on creation apply to all calls to Archive unless overridden.

func (*Archiver) Archive

func (a *Archiver) Archive(table, bucket string, options ...Option) error

Archive a Database table to S3. Any options provided override those set on creation.

func (*Archiver) ArchiveRows

func (a *Archiver) ArchiveRows(rows Rows, bucket string, options ...Option) (err error)

ArchiveRows to S3.

func (*Archiver) ArchiveRowsWithContext

func (a *Archiver) ArchiveRowsWithContext(ctx context.Context, rows Rows, bucket string, options ...Option) (err error)

ArchiveRowsWithContext is like ArchiveRows, with context.

func (*Archiver) ArchiveWithContext

func (a *Archiver) ArchiveWithContext(ctx context.Context, table, bucket string, options ...Option) (err error)

ArchiveWithContext is like Archive, with context. Any options provided override those set on creation.

type Column

type Column interface {
	Name() string
	DatabaseTypeName() string
	ScanType() reflect.Type
}

Column reports its name, database type name and scan type.

type Database

type Database interface {
	QueryContext(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error)
}

Database queries tables for rows.

type Extensioner

type Extensioner interface {
	Extension() string
}

Extensioner is a Formatter that provides a default extension.

type Formatter

type Formatter interface {
	// Open the Formatter and perform any format-specific initialization.
	Open() error
	// Format and write a single record.
	Format([][]byte) error
	// Close the Formatter and perform any format-specific cleanup.
	Close() error
}

Formatter formats and writes records. A custom Formatter may implement Extensioner to provide chiv with a default file extension.

func CSV

func CSV(w io.Writer, columns []Column) Formatter

CSV writes column headers and returns an initialized CSV formatter.

func JSON

func JSON(w io.Writer, columns []Column) Formatter

JSON opens a JSON array and returns an initialized JSON formatter.

func YAML

func YAML(w io.Writer, columns []Column) Formatter

YAML returns an initialized YAML formatter.

type FormatterFunc

type FormatterFunc func(io.Writer, []Column) Formatter

FormatterFunc returns an initialized Formatter.

type Option

type Option func(*Archiver)

Option configures the Archiver. Options can be provided when creating an Archiver or on each call to Archive.

func WithColumns

func WithColumns(c ...string) Option

WithColumns configures a list of column names to archive.

func WithExtension

func WithExtension(s string) Option

WithExtension configures an extension for object keys uploaded to S3.

func WithFormat

func WithFormat(f FormatterFunc) Option

WithFormat configures the upload format.

func WithKey

func WithKey(s string) Option

WithKey configures the object key uploaded to S3.

func WithNull

func WithNull(s string) Option

WithNull configures a custom null string.

type Rows

type Rows interface {
	ColumnTypes() ([]*sql.ColumnType, error)
	Next() bool
	Scan(...interface{}) error
	Err() error
}

Rows reports its column types and is iterable.

type Uploader

type Uploader interface {
	UploadWithContext(ctx aws.Context, input *s3manager.UploadInput, opts ...func(*s3manager.Uploader)) (*s3manager.UploadOutput, error)
}

Uploader uploads input to S3.