circuit

Documentation

Overview

Package circuit implements circuit breaker functionality for the proxy.

It provides two types of circuit breakers: consecutive and failure rate based. The circuit breakers can be configured either globally, based on hosts or individual routes. The registry ensures synchronized access to the active breakers and the recycling of the idle ones.

The circuit breakers are always assigned to backend hosts, so that the outcome of requests to one host never affects the circuit breaker behavior of another host. Besides hosts, individual routes can have separate circuit breakers, too.

Breaker Type - Consecutive Failures

This breaker opens when the proxy couldn't connect to a backend or received a >=500 status code at least N times in a row. When open, the proxy returns 503 - Service Unavailable response during the breaker timeout. After this timeout, the breaker goes into half-open state, in which it expects that M number of requests succeed. The requests in the half-open state are accepted concurrently. If any of the requests during the half-open state fails, the breaker goes back to open state. If all succeed, it goes to closed state again.

Breaker Type - Failure Rate

The "rate breaker" works similar to the "consecutive breaker", but instead of considering N consecutive failures for going open, it maintains a sliding window of the last M events, both successes and failures, and opens only when the number of failures reaches N within the window. This way the sliding window is not time based and allows the same breaker characteristics for low and high rate traffic.

Usage

When imported as a package, the Registry can be used to hold the circuit breakers and their settings. On a higher level, the circuit breaker settings can be simply passed to skipper as part of the skipper.Options object, or, equivalently, defined as command line flags.

The following command starts skipper with a global consecutive breaker that opens after 5 failures for any backend host:

skipper -breaker type=consecutive,failures=5

To set only the type of the breaker globally, and the rates individually for the hosts:

skipper -breaker type=rate,timeout=3m,idleTTL=30m \
	-breaker host=foo.example.org,window=300,failures=30 \
	-breaker host=bar.example.org,window=120,failures=45

To enable circuit breakers only for specific hosts:

skipper -breaker type=disabled \
	-breaker type=rate,host=foo.example.org,window=300,failures=30

To change (or set) the breaker configurations for an individual route and disable for another, in eskip:

updates: Method("POST") && Host("foo.example.org")
  -> consecutiveBreaker(9)
  -> "https://foo.backend.net";

backendHealthcheck: Path("/healthcheck")
  -> disableBreaker()
  -> "https://foo.backend.net";

The breaker settings can be defined in the following levels: global, based on the backend host, based on individual route settings. The values are merged in the same order, so the global settings serve as defaults for the host settings, and the result of the global and host settings serve as defaults for the route settings. Setting global values happens the same way as setting host values, but leaving the Host field empty. Setting route based values happens with filters in the route definitions. (https://godoc.org/github.com/zalando/skipper/filters/circuit)

Settings - Type

It can be ConsecutiveFailures, FailureRate or Disabled, where the first two values select which breaker to use, while the Disabled value can override a global or host configuration disabling the circuit breaker for the specific host or route.

Command line name: type. Possible command line values: consecutive, rate, disabled.

Settings - Host

The Host field indicates to which backend host should the current set of settings be applied. Leaving it empty indicates global settings.

Command line name: host.

Settings - Window

The window value sets the size of the sliding counter window of the failure rate breaker.

Command line name: window. Possible command line values: any positive integer.

Settings - Failures

The failures value sets the max failure count for both the "consecutive" and "rate" breakers.

Command line name: failures. Possible command line values: any positive integer.

Settings - Timeout

With the timeout we can set how long the breaker should stay open, before becoming half-open.

Command line name: timeout. Possible command line values: any positive integer as milliseconds or a duration string, e.g. 15m30s.

Settings - Half-Open Requests

Defines the number of requests expected to succeed while the circuit breaker is in the half-open state.

Command line name: half-open-requests. Possible command line values: any positive integer.

Settings - Idle TTL

Defines the idle timeout after which a circuit breaker gets recycled, if it hasn't been used.

Command line name: idle-ttl. Possible command line values: any positive integer as milliseconds or a duration string, e.g. 15m30s.

Filters

The following circuit breaker filters are supported: consecutiveBreaker(), rateBreaker() and disableBreaker().

The consecutiveBreaker filter expects one mandatory parameter: the number of consecutive failures to open. It accepts the following optional arguments: timeout, half-open requests, idle-ttl.

consecutiveBreaker(5, "1m", 12, "30m")

The rateBreaker filter expects two mandatory parameters: the number of consecutive failures to open and the size of the sliding window. It accepts the following optional arguments: timeout, half-open requests, idle-ttl.

rateBreaker(30, 300, "1m", 12, "30m")

The disableBreaker filter doesn't expect any arguments, and it disables the circuit breaker, if any, for the route that it appears in.

disableBreaker()

Proxy Usage

The proxy, when circuit breakers are configured, uses them for backend connections. It checks the breaker for the current backend host if it's closed before making backend requests. It reports the outcome of the request to the breaker, considering connection failures and backend responses with status code >=500 as failures. When the breaker is open, the proxy doesn't try to make backend requests, and returns a response with a status code of 503 and appending a header to the response:

X-Circuit-Open: true

Registry

The active circuit breakers are stored in a registry. They are created on-demand, for the requested settings. The registry synchronizes access to the shared circuit breakers. When the registry detects that a circuit breaker is idle, it resets it, this way avoiding that an old series of failures would cause the circuit breaker go open after an unreasonably low number of recent failures. The registry also makes sure to cleanup idle circuit breakers that are not requested anymore by the proxy. This happens in a passive way, whenever a new circuit breaker is created. The cleanup prevents storing circuit breakers for inaccessible backend hosts infinitely in those scenarios where the route configuration is continuously changing.

Index

• Constants
• type Breaker
  • func (b *Breaker) Allow() (func(bool), bool)
• type BreakerSettings
  • func (s BreakerSettings) String() string
• type BreakerType
  • func (b *BreakerType) UnmarshalYAML(unmarshal func(interface{}) error) error
• type Registry
  • func NewRegistry(settings ...BreakerSettings) *Registry
  • func (r *Registry) Get(s BreakerSettings) *Breaker

Constants

const DefaultIdleTTL = time.Hour

Types

type Breaker

type Breaker struct {
	// contains filtered or unexported fields
}

Breaker represents a single circuit breaker for a particular set of settings.

Use the Get() method of the Registry to request fully initialized breakers.

func (*Breaker) Allow

func (b *Breaker) Allow() (func(bool), bool)

Allow returns true if the breaker is in the closed state and a callback function for reporting the outcome of the operation. The callback expects true values if the outcome of the request was successful. Allow may not return a callback function when the state is open.

type BreakerSettings

type BreakerSettings struct {
	Type             BreakerType   `yaml:"type"`
	Host             string        `yaml:"host"`
	Window           int           `yaml:"window"`
	Failures         int           `yaml:"failures"`
	Timeout          time.Duration `yaml:"timeout"`
	HalfOpenRequests int           `yaml:"half-open-requests"`
	IdleTTL          time.Duration `yaml:"idle-ttl"`
}

BreakerSettings contains the settings for individual circuit breakers.

See the package overview for the detailed merging/overriding rules of the settings and for the meaning of the individual fields.

func (BreakerSettings) String

func (s BreakerSettings) String() string

String returns the string representation of a particular set of settings.

type BreakerType

type BreakerType int

BreakerType defines the type of the used breaker: consecutive, rate or disabled.

const (
	BreakerNone BreakerType = iota
	ConsecutiveFailures
	FailureRate
	BreakerDisabled
)

func (*BreakerType) UnmarshalYAML

func (b *BreakerType) UnmarshalYAML(unmarshal func(interface{}) error) error

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry objects hold the active circuit breakers, ensure synchronized access to them, apply default settings and recycle the idle breakers.

func NewRegistry

func NewRegistry(settings ...BreakerSettings) *Registry

NewRegistry initializes a registry with the provided default settings. Settings with an empty Host field are considered as defaults. Settings with the same Host field are merged together.

func (*Registry) Get

func (r *Registry) Get(s BreakerSettings) *Breaker

Get returns a circuit breaker for the provided settings. The BreakerSettings object is used here as a key, but typically it is enough to just set its Host field:

r.Get(BreakerSettings{Host: backendHost})

The key will be filled up with the defaults and the matching circuit breaker will be returned if it exists, or a new one will be created if not.