README
¶
gtl
Extendable and minimalistic ETL toolkit in Go, built on generic io (std lib) pipelines
Index:
Errors
Expand/collapse section.
GTL tries to get out of your way and so only two errors are used in the core pkg, both inherited from io in the std lib:
io.EOF // Stop reading/pulling/consuming.
io.ErrClosedPipe // Stop writing/pushing/producing.
Core interfaces
There are two core interfaces and they are shown below. They are simply generic variants of io.Readerand io.Writer combined with a context.Context.
type Reader[T any] interface {
Read(context.Context) (T, error)
}
type Writer[T any] interface {
Write(context.Context, T) error
}
As with the io package, there are varying combinations of basic interfaces, e.g io.ReadCloser. These groupings are for the most part mirrored here and can be viewed by clicking on this section.
type ReadCloser[T any] interface {
io.Closer
Reader[T]
}
type WriteCloser[T any] interface {
io.Closer
Writer[T]
}
type ReadWriter[T, U any] interface {
Reader[T]
Writer[U]
}
type ReadWriteCloser[T, U any] interface {
io.Closer
Reader[T]
Writer[U]
}
There are also "impl structs" which lets you implement most core interfaces with a function, allowing you to dodge boilerplate-y code. These can be viewed by clicking on this section.
Signatures are links to the Go playground (examples).
Core constructors
Core constructors for the most part facilitates interoperability between core interfaces and the io package. I.e conversion of io.Reader (bytes) into core.Reader[T](generic vals) and back, and io.Writer (bytes) into core.Writer[T] (vals) and back.
func NewReaderFrom[T any](vs ...T) Reader[T]]func NewReaderFromBytes[T any](r io.Reader) func(f func(io.Reader) Decoder) Reader[T]func NewReaderFromValues[T any](r Reader[T]) func(f func(io.Writer) Encoder) io.Reader
func NewWriterFromValues[T any](w io.Writer) func(f func(io.Writer) Encoder) Writer[T]func NewWriterFromBytes[T any](w Writer[T]) func(f func(io.Reader) Decoder) io.Writer
Also, there are additional constructors for manipulating streams.
func NewReaderWithBatching[T any](r Reader[T], size int) Reader[[]T]func NewReaderWithUnbatching[T any](r Reader[[]T]) Reader[T]func NewWriterWithBatching[T any](w Writer[[]T], size int) Writer[T]func NewWriterWithUnbatching[T any](w Writer[T]) Writer[[]T]
ETL Components
This section covers ETL components, they wrap core interfaces in order to provide some useful functionality. All links here go to the Go playground.
Logging
Stats
- stats.NewStreamedTeeReader
- stats.NewBatchedTeeReader
- stats.NewStreamedTeeWriter
- stats.NewBatchedTeeWriter
Eventloop
Sleep
Pagination
Directories