etl

Documentation

Overview

Package etl provides a simple iterator interface.

Index

• Variables
• func Collect[T any](it Iter) ([]T, error)
• func CollectContext[T any](ctx context.Context, it Iter) ([]T, error)
• func Consume[T any](it Iter, fn func(T) error) error
• func ConsumeBatch[T any](it Iter, n int, fn func([]T) error) error
• func ConsumeContext[T any](ctx context.Context, it Iter, fn func(T) error) error
• func Count(it Iter) (int, error)
• func Print(it Iter) error
• func Take[T any](it Iter, n int) ([]T, error)
• func WorkersConsume[Ti any](it Iter, workers int, fn func(context.Context, Ti) error) error
• func WorkersConsumeContext[Ti any](ctx context.Context, it Iter, workers int, fn func(context.Context, Ti) error) error
• type Custom
• type FilterFunc
• type Gen
• type Iter
  • func Cat(its ...Iter) Iter
  • func Chan[T any](ch <-chan T) Iter
  • func ErrIter(err error) Iter
  • func Filter[T any](it Iter, fn FilterFunc[T]) Iter
  • func FlatMap[Ti, To any](it Iter, fn func(Ti) []To) Iter
  • func Limit(it Iter, n int) Iter
  • func MakeGen[T any](g Gen[T]) Iter
  • func MakeIter[T any](c Custom[T]) Iter
  • func Map[Ti, To any](it Iter, fn func(Ti) To) Iter
  • func MapE[Ti, To any](it Iter, fn func(Ti) (To, error)) Iter
  • func NewIter[IT, DT any](start func() (IT, error), next func(context.Context, IT) (DT, error), ...) Iter
  • func Peek[T any](it Iter, fn func(v T)) Iter
  • func Seq(start, end, step int) Iter
  • func Values[T any](vs ...T) Iter
  • func Workers[To any](it Iter, workers int, fn func(context.Context, W[To]) error) Iter
  • func WorkersValue[Ti, To any](it Iter, workers int, fn func(context.Context, Ti, Y[To]) error) Iter
  • func Yield[Ti, To any](it Iter, fn func(Ti, Y[To]) error) Iter
• type W
  • func (w W[To]) Yield(v To) error
• type Y

Variables

var EOI = io.EOF

EOI is returned when iterator doesn't have more data.

var ErrCancelled = errors.New("iterator cancelled")

Functions

func Collect

func Collect[T any](it Iter) ([]T, error)

func CollectContext

func CollectContext[T any](ctx context.Context, it Iter) ([]T, error)

Collect collects all iterator values into a slice.

func Consume

func Consume[T any](it Iter, fn func(T) error) error

Consume iterates over the given iterator and calls fn for each value.

func ConsumeBatch

func ConsumeBatch[T any](it Iter, n int, fn func([]T) error) error

ConsumeBatch consumes the given iterator in batches of n values.

func ConsumeContext

func ConsumeContext[T any](ctx context.Context, it Iter, fn func(T) error) error

func Count

func Count(it Iter) (int, error)

Count consumes the iterator and return the number of iterations.

func Print

func Print(it Iter) error

Print prints the values of the given iterator to stdout.

func Take

func Take[T any](it Iter, n int) ([]T, error)

Take consumes the given iterator and return the first n values as a slice.

func WorkersConsume

func WorkersConsume[Ti any](it Iter, workers int, fn func(context.Context, Ti) error) error

WorkersConsume creates a pool of workers that call fn for every iteration value. this will close the consumed Iter upon finish.

func WorkersConsumeContext

func WorkersConsumeContext[Ti any](ctx context.Context, it Iter, workers int, fn func(context.Context, Ti) error) error

Types

type Custom

type Custom[T any] struct {
	Next  func(context.Context) (T, error)
	Close func() error
}

type FilterFunc

type FilterFunc[T any] func(T) bool

type Gen

type Gen[T any] struct {
	Run   func(context.Context, Y[T]) error
	Close func() error
}

type Iter

type Iter interface {
	Next(context.Context) (any, error)
	Close() error
}

Iter iterator interface.

func Cat

func Cat(its ...Iter) Iter

func Chan

func Chan[T any](ch <-chan T) Iter

func ErrIter

func ErrIter(err error) Iter

ErrIter is an iterator that always returns an error.

func Filter

func Filter[T any](it Iter, fn FilterFunc[T]) Iter

Filter returns an iterator that filters the values of the source given the func fn if fn returns true the value is passed through

func FlatMap

func FlatMap[Ti, To any](it Iter, fn func(Ti) []To) Iter

func Limit

func Limit(it Iter, n int) Iter

Limit returns an iterator that returns at most n values from the given iterator Closing the returned iterator will close the given iterator.

func MakeGen

func MakeGen[T any](g Gen[T]) Iter

func MakeIter

func MakeIter[T any](c Custom[T]) Iter

func Map

func Map[Ti, To any](it Iter, fn func(Ti) To) Iter

Map returns an interator that transforms the values of the source iterator using the func fn.

func MapE

func MapE[Ti, To any](it Iter, fn func(Ti) (To, error)) Iter

Map returns an interator that transforms the values of the source iterator using the func fn.

func NewIter

func NewIter[IT, DT any](
	start func() (IT, error),
	next func(context.Context, IT) (DT, error),
	end func(IT) error,
) Iter

func Peek

func Peek[T any](it Iter, fn func(v T)) Iter

Peek calls the func fn each time the Next from the iterator is called

func Seq

func Seq(start, end, step int) Iter

Seq iterates over the sequence of integers from start to end.

func Values

func Values[T any](vs ...T) Iter

Values returns an iterator that iterates over the variadic arguments.

func Workers

func Workers[To any](it Iter, workers int, fn func(context.Context, W[To]) error) Iter

Workers iterates over a slice of values and calls a worker func for each value.

func WorkersValue

func WorkersValue[Ti, To any](it Iter, workers int, fn func(context.Context, Ti, Y[To]) error) Iter

WorkersValue is a convinitent func that calls fn for every consumed value, it will yield any value by calling the yield func.

func Yield

func Yield[Ti, To any](it Iter, fn func(Ti, Y[To]) error) Iter

Yield returns an iterator that for each consumed value calls a fn passing an yielder,

type W

type W[To any] struct {
	Iter
	ID int
	// contains filtered or unexported fields
}

func (W[To]) Yield

func (w W[To]) Yield(v To) error

type Y

type Y[T any] func(T) error

Y yield parameter.