config

README

Unbounded configurations and behavior

MBS = MaxBatchSize
BW  = BatchWait
MEC = MaxEvaluatorConcurrency
MQB = MaxQueuedBatches

N is N as in number N.

MBS BW  MEC MQB - unbounded behavior
0   0   0   0   = invalid configuration - use defaults (add bound BatchWait)
N   0   0   0   = infinite running evaluations with batch size MaxBatchSize
0   N   0   0   = infinite running evaluations with variable batch size 
N   N   0   0   = infinite running evaluations with batch size MaxBatchSize or less
0   0   N   0   = invalid configuration - use defaults (bound BatchWait)
N   0   N   0   = infinite queued batches of size MaxBatchSize
0   N   N   0   = infinite queued batches with variable batch size
N   N   N   0   = infinite queued batches with batch size MaxBatchSize or less
0   0   0   N   = invalid configuration - use defaults (add bound BatchWait)
N   0   0   N   = infinite running evaluations with batch size MaxBatchSize
0   N   0   N   = infinite running evaluations with variable batch size 
N   N   0   N   = infinite running evaluations with batch size MaxBatchSize or less
0   0   N   N   = invalid configuration - use defaults (add bound BatchWait)
N   0   N   N   = shed with N queued batches of size MaxBatchSize
0   N   N   N   = infinite size batch with N queued batches of variable batch size
N   N   N   N   = shed with N queued batches of size MaxBatchSize or less

Documentation

Index

• Variables
• type BatcherConfig
  • func (b *BatcherConfig) Init()
• type V
  • func (v *V) Debug(tag string, msg string)
  • func (v *V) DebugFn(tag string, mfn func() string, second func() bool)
  • func (v *V) InputEnabled() func() bool

Variables

var SysMaxQueuedBatches int

SysMaxQueuedBatches will be used as the channel size for unbounded batch queues. Can be overridden by environment variable DISPATCHER_MAX_QUEUED_BATCHES.

Types

type BatcherConfig

type BatcherConfig struct {
	// MaxBatchSize is a hard limit to the number of elements in a batch
	// that we attempt to send to the model for prediction.
	// Currently will indicate a minimum batch size, i.e. a batch will be forced
	// if the current batch size is >= MaxBatchSize.
	//
	// TODO make this an exact constraint.
	//
	// If this is <= 0, then the maximum size of a batch will be unbounded.
	// This constraint will always force a batch to be queued.
	MaxBatchSize int `json:",omitempty" yaml:",omitempty"`

	// BatchWait indicates the maximum amount of time to wait to collect
	// requests to make a batch.
	// This is not a rolling window.
	//
	// Set to < 0 to turn off batching.
	// If this is 0, then a default of 500 * time.Microsecond will be used.
	// This constraint may sometimes cause a batch to be queued.
	// If there TimeoutAdjustments are provided, then this value
	// will be ignored.
	//
	// If MaxEvaluatorConcurrency is NOT set then the batch will be queued
	// (and run) as soon MaxBatchSize is hit (if set) or BatchWait elapsed.
	BatchWait time.Duration `json:",omitempty" yaml:",omitempty"`

	// MaxEvaluatorConcurrency determines the maximum number of evaluators to
	// have running.
	//
	// Set to 0 to have it unbounded - it will use SysMaxQueuedBatches
	// as channel size.
	// If set to > 0, but MaxBatchSize is not set, then requests will queue into
	// a single "mega-batch," and MaxQueuedBatches is ignored.
	// If set to > 0, and MaxBatchSize is set, then batches will queue until
	// MaxQueuedBatches is hit.
	MaxEvaluatorConcurrency int `json:",omitempty" yaml:",omitempty"`

	// MaxQueuedBatches works IFF both MaxEvaluatorConcurrency and
	// MaxBatchSize are set.
	//
	// Requests will be rejected if the number of batches queued to be evaluated
	// exceeds MaxQueuedBatches.
	//
	// In the case that there are too many queued batches, the server will start to reject
	// requests with HTTP code 429.
	MaxQueuedBatches int `json:",omitempty" yaml:",omitempty"`

	TimeoutAdjustments *adjust.AdjustConfig `json:",omitempty" yaml:",omitempty"`

	// Verbose enables a lot of debugging messages.
	Verbose *V `json:",omitempty" yaml:",omitempty"`
}

BatcherConfig provides a fixed set of parameters for the batching system. By default, if no properties are explicitly declared, then a only the BatchWait will be set to 500 microseconds.

func (*BatcherConfig) Init

func (b *BatcherConfig) Init()

type V

type V struct {
	ID     string
	Output bool
	Input  bool
}

func (*V) Debug

func (v *V) Debug(tag string, msg string)

func (*V) DebugFn

func (v *V) DebugFn(tag string, mfn func() string, second func() bool)

func (*V) InputEnabled

func (v *V) InputEnabled() func() bool