batch

Documentation

Index

• func Min(x, y int) int
• func ProcessInBatches(items []string, processBatch func([]string) error, batchSize int) (processedBatches int, err error)
• func ProcessInConcurrentBatches(getBatch GenericBatchGetter, processBatch GenericBatchProcessor, ...) (err error)
• type GenericBatchGetter
• type GenericBatchProcessor

Functions

func Min

func Min(x, y int) int

Min returns the lowest value

func ProcessInBatches

func ProcessInBatches(items []string, processBatch func([]string) error, batchSize int) (processedBatches int, err error)

ProcessInBatches is a generic method that splits the provided items in batches and calls processBatch for each batch

func ProcessInConcurrentBatches

func ProcessInConcurrentBatches(getBatch GenericBatchGetter, processBatch GenericBatchProcessor, batchSize, maxWorkers int) (err error)

ProcessInConcurrentBatches is a generic method to concurrently obtain some resource in batches and then process each batch

Types

type GenericBatchGetter

type GenericBatchGetter func(offset int) (batch interface{}, totalCount int, eTag string, err error)

GenericBatchGetter defines the method signature for a batch getter to obtain a batch of some generic resource

type GenericBatchProcessor

type GenericBatchProcessor func(batch interface{}, batchETag string) (abort bool, err error)

GenericBatchProcessor defines the method signature for a batch processor to process a batch of some generic resource