memthrottle

Documentation

Index

• func MemoryThrottler(cfg Config, lgr log.Logger) abstract.AsyncMiddleware
• type Config
  • func DefaultConfig(totalBytes uint64) Config

Functions

func MemoryThrottler

func MemoryThrottler(cfg Config, lgr log.Logger) abstract.AsyncMiddleware

MemoryThrottler is middleware that triggers buffer flush (and GC) if memory consumption is close to limit

MemoryThrottler is coupled with async.Bufferer and relies on immediate buffer flush if non-row item has been encountered. It doesn't work without Bufferer and should not work well with sources pushing big chunks of data in single AsyncPush call. It's believed that all sources should produce reasonably small batches in future

It use simple adaptive algorithm to determine and control flush threshold depending on current memory usage

Types

type Config

type Config struct {
	// contains filtered or unexported fields
}

func DefaultConfig

func DefaultConfig(totalBytes uint64) Config

DefaultConfig makes default config for memory throttler algo

totalBytes parameter should define total memory amount available to the process, not VM size or smth