batching

README

Dynamic Batching for Deep Learning Serving

Ventu already implement this protocol, so it can be used as the worker for deep learning inference.

Features

• dynamic batching with batch size and latency
• invalid request won't affects others in the same batch
• communicate with workers through Unix domain socket or TCP
• load balancing

If you are interested in the design, check my blog Deep Learning Serving Framework.

Configs

go run service/app.go --help

Usage app:
  -address string
        socket file or host:port (default "batch.socket")
  -batch int
        max batch size (default 32)
  -capacity int
        max jobs in the queue (default 1024)
  -host string
        host address (default "0.0.0.0")
  -latency int
        max latency (millisecond) (default 10)
  -port int
        service port (default 8080)
  -protocol string
        unix or tcp (default "unix")
  -timeout int
        timeout for a job (millisecond) (default 5000)

Demo

go run service/app.go
python examples/app.py
python examples/client.py

Documentation

Index

• Constants
• type Batching
  • func NewBatching(address, protocol string, batchSize, capacity int, ...) *Batching
  • func (b *Batching) HandleHTTP(ctx *fasthttp.RequestCtx)
  • func (b *Batching) Run()
  • func (b *Batching) Stop() error
• type Job
• type String2Bytes

Constants

const (
	// IntByteLength defines the byte length of `length` for data
	IntByteLength = 4
	// UUIDLength defines the string bits length
	UUIDLength = 36
	// ErrorIDsKey defines the key for error IDs
	ErrorIDsKey = "error_ids"
)

Types

type Batching

type Batching struct {
	Address string // socket file or "{host}:{port}"
	// contains filtered or unexported fields
}

Batching provides HTTP handler and socket communication. It generate batch jobs when workers request and send the inference results (or error) to the right client.

func NewBatching

func NewBatching(address, protocol string, batchSize, capacity int, maxLatency, timeout time.Duration) *Batching

NewBatching creates a Batching instance

func (*Batching) HandleHTTP

func (b *Batching) HandleHTTP(ctx *fasthttp.RequestCtx)

HandleHTTP is the handler for fasthttp

func (*Batching) Run

func (b *Batching) Run()

Run the socket communication

func (*Batching) Stop

func (b *Batching) Stop() error

Stop the batching service and socket, close the queue channel

type Job

type Job struct {
	// contains filtered or unexported fields
}

Job wrap the new request as a job waiting to be done by workers

type String2Bytes

type String2Bytes map[string][]byte

String2Bytes structure used in socket communication protocol