overcurrent

package module
v0.0.0-...-ab99255 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 28, 2018 License: MIT Imports: 8 Imported by: 2

README

Overcurrent

GoDoc Build Status Maintainability Test Coverage

Go library for protecting function calls via circuit breaker pattern.

The circuit breaker pattern can prevent an application from repeatedly trying to execute an operation that is likely to fail. If the problem appears to have been rectified, the application can attempt to invoke the operation. This is useful in a distributed environment where an application accesses remote resources and services. It is possible (and likely at scale) for these operations to fail due to transient faults such as:

  • timeouts
  • slow network connections
  • resource or service being overcommitted
  • resource or service being temporarily unavailable

These faults typically correct themselves after a short period of time. You can read more about the pattern here and here.

This library depends on the backoff library, which defines structures for creating backoff interval generator.

Example

First, create a circuit breaker instance with any of the following configuration functions supplied. A circuit breaker can be created with all default parameters.

The InvocationTimeout specifies how long a protected function can run for before returning an error (zero allows for unbounded runtime).

The ResetBackoff specifies how long the circuit breaker stays in the open state until transitioning to the half-closed state. If a failure occurs while in the half-closed state, the circuit breaker will transition back to the open state, and the time spent before transitioning to the half-closed state may increase, depending on the implementation of the backoff interface.

The HalfClosedRetryProbability specifies the probability that a request in the half-closed state will attempt to retry instead of immediately returning a CircuitOpenError.

breaker := NewCircuitBreaker(
	WithInvocationTimeout(50 * time.Millisecond),
	WithResetBackoff(backoff.NewConstantBackoff(1 * time.Second)),
	WithHalfClosedRetryProbability(0.1),
	WithFailureInterpreter(NewAnyErrorFailureInterpreter()),
	WithTripCondition(NewConsecutiveFailureTripCondition(5)),
)

The FailureInterpreter determines which errors returned from the protected function should count as a failure with respect to the circuit breaker. As an example, the following failure interpreter will only count HTTP 500 errors as a circuit error (given an omitted definition of the type ProtocolError).

type ServerErrorFailureInterpreter struct{}

func (fi *ServerErrorFailureInterpreter) ShouldTrip(err error) bool {
	if perr, ok := err.(ProtocolError); ok {
		return perr.StatusCode >= 500
	}

	return false
}

The TripCondition determines, based on recent failure history, when the breaker should trip. This interface can be customized to trip after a number of failures in a row, number of failures in a given time span, fail rate, etc.

The breaker can be explicitly tripped and reset via the Trip and Reset methods. If a breaker is manually tripped, then it will remain in open state until it is manually reset (it will never transition to the half-closed state).

Function API

To use the breaker, simply pass the function that attempts to access a resource to the Call method of the breaker. This method may attempt to call the passed function. The breaker will return a custom error if the circuit is closed or if the function is invoked but takes too long to complete. If the function is called and completes (successfully or unsuccessfully), the method returns the error that the function returns.

err := breaker.Call(func(ctx context.Context) error {
	req, err := http.NewRequest("GET", "http://example.com", nil)
	if err != nil {
		return err
	}

	// Wrap in context passed to function so that the request
	// is canceled if the function exceeds the invocation max.
	resp, err := http.DefaultClient.Do(r.WithContext(ctx))
	if err != nil {
		return err
	}

	// Process HTTP response
	return nil
})

if err == nil {
	// Success
} else if err == ErrInvocationTimeout {
	// Took too long
} else if err == ErrCircuitOpen {
	// Not attempted, in failure mode
} else {
	// Unsuccessful, error is HTTP error
}

Design Choice: The protected function is given to the breaker as a parameter to each invocation of Call, as opposed to begin registered with the circuit breaker at initialization. This is to increase the flexibility of the API so the input to the function can easily change on each invocation. It is not advised that several disparate functions are passed to the same breaker - failures from one function will influence the other in ways that are not intuitive.

A breaker also has a CallAsync function that does the work of Call in a separate goroutine and returns a channnel that receives an error value then immediately closes.

Registry

A registry is a collection of breakers which can be invoked by a unique name. In order to use a breaker in the registry, it must first be configured. Any of the options used to configure a breaker can be used along with two new options:

  • max concurrency: the maximum number of goroutines which may execute the breaker function concurrently
  • max concurrency timeout: how long you are willing to wait while the breaker function is at max concurrency
registry := NewRegistry()

registry.Configure(
	"redis-cache",
	WithFailureInterpreter(NewAnyErrorFailureInterpreter()),
	WithTripCondition(NewConsecutiveFailureTripCondition(5)),
	// ...
	WithMaxConcurrency(50),
	WithMaxConcurrencyTimeout(time.Second),
)

To use a breaker, invoke the Call method of the registry with the name of the breaker to use. You can also pass a second nillable fallback function which is invoked when the breaker function fails (or fails to be called due to the circuit state or the concurrency state of the breaker).

registry.Call("redis-cache", func(ctx context.Context) error {
	// get value from redis
}, func(err error) error {
	// do some canned action
	return nil
})

Symmetrically to the breaker, a CallAsync method is also available with the same semantics as Call.

Non-Function API

Sometimes a chunk of code which should be protected by the circuit breaker is not easily contained in a single function. In these cases, the methods used by the Call method are exposed for manual use. First, the ShouldTry method can be queried to get the current state of the circuit breaker.

if breaker.ShouldTry() {
	// Try to execute the protected section
}

Then, in the location where an attempt has been made, the result can be applied to the circuit breaker manually. Special effort must be made to ensure that all code paths mark a result with the breaker; otherwise, the trip condition may set the circuit state erroneously.

if success {
	// A nil error is successful
	breaker.MarkResult(nil)
} else {
	// Mark whatever error is accepted by the registered failure interpreter
	breaker.MarkResult(ErrBadAttempt)
}

Metric Collectors

Hystrix

Overcurrent comes with a metric collector compatible with the Netflix Hystrix Dashboard. For testing purposes, a stand-alone dashboard can be used via Docker (see docker-compose.yml for details). Below is a minimal example setting up the Hystrix collector.

import (
	"github.com/efritz/overcurrent"
	"github.com/efritz/overcurrent/hystrix"
)

func main() {
	hystrixCollector := hystrix.NewCollector()
	hystrixCollector.Start()
	defer hystrixCollector.Stop()

	registry := overcurrent.NewRegistry()
	registry.Configure(
		"name",
		overcurrent.WithCollector(overcurrent.NamedCollector("name", hystrixCollector)),
	)

	// ...

	http.ListenAndServe("0.0.0.0:8080", hystrixCollector.Handler())
}

License

Copyright (c) 2016 Eric Fritz

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrCircuitOpen occurs when the Call method fails immediately.
	ErrCircuitOpen = fmt.Errorf("circuit is open")

	// ErrInvocationTimeout occurs when the method takes too long to execute.
	ErrInvocationTimeout = fmt.Errorf("invocation has timed out")
)
View Source 
var (
	ErrAlreadyConfigured   = errors.New("breaker is already configured")
	ErrBreakerUnconfigured = errors.New("breaker not configured")
	ErrMaxConcurrency      = errors.New("breaker is at max concurrency")
)

Functions

This section is empty.

Types

type BreakerConfig 

type BreakerConfig struct {
	MaxConcurrency int
}

BreakerConfig is a struct that contains a copy of some of a breaker's initialization values. This struct may grow as metric collectors track additional breaker state.

type BreakerConfigFunc 

type BreakerConfigFunc func(*circuitBreaker)

func WithCollector 

func WithCollector(collector MetricCollector) BreakerConfigFunc

func WithFailureInterpreter 

func WithFailureInterpreter(failureInterpreter FailureInterpreter) BreakerConfigFunc

func WithHalfClosedRetryProbability 

func WithHalfClosedRetryProbability(probability float64) BreakerConfigFunc

func WithInvocationTimeout 

func WithInvocationTimeout(timeout time.Duration) BreakerConfigFunc

func WithMaxConcurrency 

func WithMaxConcurrency(maxConcurrency int) BreakerConfigFunc

func WithMaxConcurrencyTimeout 

func WithMaxConcurrencyTimeout(timeout time.Duration) BreakerConfigFunc

func WithResetBackoff 

func WithResetBackoff(resetBackoff backoff.Backoff) BreakerConfigFunc

func WithTripCondition 

func WithTripCondition(tripCondition TripCondition) BreakerConfigFunc

type BreakerFunc 

type BreakerFunc func(ctx context.Context) error

type CircuitBreaker 

type CircuitBreaker interface {
	// Trip manually trips the circuit breaker. The circuit breaker will remain open
	// until it is manually reset.
	Trip()

	// Reset the circuit breaker.
	Reset()

	// ShouldTry returns true if the circuit breaker is closed or half-closed with
	// some probability. Successive calls to this method may yield different results
	// depending on the registered trip condition.
	ShouldTry() bool

	// MarkResult takes the result of the protected section and marks it as a success if
	// the error is nil or if the failure interpreter decides not to trip on this error.
	MarkResult(err error) bool

	// Call attempts to call the given function if the circuit breaker is closed, or if
	// the circuit breaker is half-closed (with some probability). Otherwise, return an
	// ErrCircuitOpen. If the function times out, the circuit breaker will fail with an
	// ErrInvocationTimeout. If the function is invoked and yields a value before the
	// timeout elapses, that value is returned.
	Call(f BreakerFunc) error

	// CallAsync invokes the given function in a goroutine, returning a channel which
	// may receive one non-nil error value and then close. The channel will close without
	// writing a value on success.
	CallAsync(f BreakerFunc) <-chan error
}

CircuitBreaker protects the invocation of a function and monitors failures. After a certain failure threshold is reached, future invocations will instead return an ErrErrCircuitOpen instead of attempting to invoke the function again.

func NewCircuitBreaker 

func NewCircuitBreaker(configs ...BreakerConfigFunc) CircuitBreaker

NewCircuitBreaker creates a new CircuitBreaker.

func NewNoopBreaker 

func NewNoopBreaker() CircuitBreaker

NewNoopBreaker creates a new non-tripping breaker.

type CircuitState 

type CircuitState int
const (
	StateOpen       CircuitState // Failure state
	StateClosed                  // Success state
	StateHalfClosed              // Cautious, probabilistic retry state
	StateHardOpen                // Forced failure state
)

type ConsecutiveFailureTripCondition 

type ConsecutiveFailureTripCondition struct {
	// contains filtered or unexported fields
}

ConsecutiveFailureTripCondition is a trip condition that trips the circuit breaker after a configurable number of failures occur in a row. A single successful call will break the failure chain.

func (*ConsecutiveFailureTripCondition) Failure 

func (tc *ConsecutiveFailureTripCondition) Failure()

func (*ConsecutiveFailureTripCondition) ShouldTrip 

func (tc *ConsecutiveFailureTripCondition) ShouldTrip() bool

func (*ConsecutiveFailureTripCondition) Success 

func (tc *ConsecutiveFailureTripCondition) Success()

type EventType 

type EventType int

EventType distinguishes interesting occurrences. 

const (
	// EventTypeAttempt occurs when Call or CallAsync is called.
	EventTypeAttempt EventType = iota

	// EventTypeSuccess occurs when a breaker func returns a nil error.
	EventTypeSuccess

	// EventTypeFailure occurs when a breaker func returns a non-nil error
	// or cannot be called due to breaker status or semaphore contention.
	EventTypeFailure

	// EventTypeError occurs when a breaker func returns a non-nil error
	// which is interpreted as an error against the breaker.
	EventTypeError

	// EventTypeBadRequest occurs when a breaker func returns a non-nil
	// error which is not interpreted as an error against the breaker.
	EventTypeBadRequest

	// EventTypeShortCircuit occurs when a circuit is open and no breaker
	// func is invoked.
	EventTypeShortCircuit

	// EventTypeTimeout occurs when execution of a breaker func times out.
	EventTypeTimeout

	// EventTypeRejection occurs when a breaker func cannot be invoked due
	// to semaphore contention.
	EventTypeRejection

	// EventTypeFallbackSuccess occurs when a fallback func returns a nil
	// error.
	EventTypeFallbackSuccess

	// EventTypeFallbackFailure occurs when a fallback func returns a non-nil
	// error.
	EventTypeFallbackFailure

	// EventTypeRunDuration marks the duration of a breaker func invocation.
	EventTypeRunDuration

	// EventTypeTotalDuration marks the duration of a Call or CallAsync method.
	EventTypeTotalDuration

	// EventTypeSemaphoreQueued occurs once a routine begins waiting for a
	// semaphore token. This event does not occur if a token is immediately
	// available.
	EventTypeSemaphoreQueued

	// EventTypeSemaphoreDequeued occurs once a routine stops waiting for a
	// semaphore token. This could be because the routine got a successful token,
	// or because the max timeout has elapsed.
	EventTypeSemaphoreDequeued

	// EventTypeSemaphoreAcquired occurs once a semaphore token is acquired and
	// the breaker func can be invoked.
	EventTypeSemaphoreAcquired

	// EventTypeSemaphoreReleased occurs after the breaker func is invoked.
	EventTypeSemaphoreReleased
)

type FailureInterpreter 

type FailureInterpreter interface {
	// ShouldTrip determines if this error should cause a CircuitBreaker
	// to transition from an open to a closed state.
	ShouldTrip(error) bool
}

FailureInterpreter is the interface that determines if an error should affect whether or not the circuit breaker will trip. This is useful if some errors are indictive of a system failure and others are not (e.g. HTTP 500 vs HTTP 400 responses).

func NewAnyErrorFailureInterpreter 

func NewAnyErrorFailureInterpreter() FailureInterpreter

NewAnyErrorFailureInterpreter creates a failure interpreter that trips on every error.

type FailureInterpreterFunc 

type FailureInterpreterFunc func(error) bool

func (FailureInterpreterFunc) ShouldTrip 

func (f FailureInterpreterFunc) ShouldTrip(err error) bool

type FallbackFunc 

type FallbackFunc func(error) error

type MetricCollector 

type MetricCollector interface {
	// ReportNew fires when a breaker is first initialized so the collector
	// can track the immutable config of circuit breakers.
	ReportNew(BreakerConfig)

	// ReportCount fires each time a non-latency event is emitted.
	ReportCount(EventType)

	// ReportDuration fires on latency events with the time spent inside a
	// code path of-interest (inside a breaker func or inside a Call func).
	ReportDuration(EventType, time.Duration)

	// ReportState fires when a breaker changes state.
	ReportState(CircuitState)
}

func NamedCollector 

func NamedCollector(name string, collector NamedMetricCollector) MetricCollector

NamedCollector converts a named metric collector into a metric collector. The name given to this constructor will be sent as the first argument to all of the named metric collector methods.

func NewMultiCollector 

func NewMultiCollector(collectors ...MetricCollector) MetricCollector

NewMultiCollector creates a new MultiCollector.

func NewNoopCollector 

func NewNoopCollector() MetricCollector

NewNoopCollector creates a new do-nothing collector.

type MultiCollector 

type MultiCollector struct {
	// contains filtered or unexported fields
}

MultiCollector is a metric collector that wraps several other collector instances. Each event will be passed to every backend in the registered order.

func (*MultiCollector) ReportCount 

func (c *MultiCollector) ReportCount(eventType EventType)

func (*MultiCollector) ReportDuration 

func (c *MultiCollector) ReportDuration(eventType EventType, duration time.Duration)

func (*MultiCollector) ReportNew 

func (c *MultiCollector) ReportNew(config BreakerConfig)

func (*MultiCollector) ReportState 

func (c *MultiCollector) ReportState(state CircuitState)

type NamedMetricCollector 

type NamedMetricCollector interface {
	// ReportNew is MetricCollector.ReportNew with the name of the breaker
	// passed in as a first argument.
	ReportNew(string, BreakerConfig)

	// ReportCount is MetricCollector.ReportCount with the name of the breaker
	// passed in as a first argument.
	ReportCount(string, EventType)

	// ReportDuration is MetricCollector.ReportDuration with the name of the breaker
	// passed in as a first argument.
	ReportDuration(string, EventType, time.Duration)

	// ReportState is MetricCollector.ReportState with the name of the breaker
	// passed in as a first argument.
	ReportState(string, CircuitState)
}

type NoopBreaker 

type NoopBreaker struct{}

NoopBreaker is a circuit breaker that never trips.

func (*NoopBreaker) Call 

func (b *NoopBreaker) Call(f BreakerFunc) error

func (*NoopBreaker) CallAsync 

func (b *NoopBreaker) CallAsync(f BreakerFunc) <-chan error

func (*NoopBreaker) MarkResult 

func (b *NoopBreaker) MarkResult(err error) bool

func (*NoopBreaker) Reset 

func (b *NoopBreaker) Reset()

func (*NoopBreaker) ShouldTry 

func (b *NoopBreaker) ShouldTry() bool

func (*NoopBreaker) Trip 

func (b *NoopBreaker) Trip()

type NoopCollector 

type NoopCollector struct{}

NoopCollector is a metric collector that does nothing.

func (*NoopCollector) ReportCount 

func (c *NoopCollector) ReportCount(EventType)

func (*NoopCollector) ReportDuration 

func (c *NoopCollector) ReportDuration(EventType, time.Duration)

func (*NoopCollector) ReportNew 

func (c *NoopCollector) ReportNew(BreakerConfig)

func (*NoopCollector) ReportState 

func (c *NoopCollector) ReportState(CircuitState)

type PercentageFailureTripCondition 

type PercentageFailureTripCondition struct {
	// contains filtered or unexported fields
}

PercentageFailureTripCondition is a trip condition that trips the circuit breaker after the number of failures for the last fixed number of attempts exceeds a percentage threshold.

func (*PercentageFailureTripCondition) Failure 

func (tc *PercentageFailureTripCondition) Failure()

func (*PercentageFailureTripCondition) ShouldTrip 

func (tc *PercentageFailureTripCondition) ShouldTrip() bool

func (*PercentageFailureTripCondition) Success 

func (tc *PercentageFailureTripCondition) Success()

type Registry 

type Registry interface {
	// Configure will register a new breaker instance under the given name using
	// the given configuration. A breaker's configuration may not be changed after
	// being initialized. It is an error to register the same breaker twice, or try
	// to invoke Call or CallAsync with an unregistered breaker.
	Configure(name string, configs ...BreakerConfigFunc) error

	// Call will invoke `Call` on the breaker configured with the given name. If
	// the breaker returns a non-nil error, the fallback function is invoked with
	// the error as the value. It may be the case that the fallback function is
	// invoked without the breaker function failing (e.g. circuit open).
	Call(name string, f BreakerFunc, fallback FallbackFunc) error

	// CallAsync will create a channel that receives the error value from an similar
	// invocation of Call. See the Breaker docs for more details.
	CallAsync(name string, f BreakerFunc, fallback FallbackFunc) <-chan error
}

func NewRegistry 

func NewRegistry() Registry

type TripCondition 

type TripCondition interface {
	// Invoked when the a call to a service completes successfully (note - all
	// successful calls, not just ones immediately after a failure period), or
	// when the breaker is hard reset. Stats should be reset at this point.
	Success()

	// Invoked when the circuit breaker was closed or half-closed and failure
	// occurs, or when the breaker is hard-tripped. Stats should be collected
	// at this point.
	Failure()

	// Determines the state of the breaker - returns true or false for open and
	// closed state, respectively.
	ShouldTrip() bool
}

TripCondition is the interface that controls the open/closed state of the circuit breaker based on failure history.

func NewConsecutiveFailureTripCondition 

func NewConsecutiveFailureTripCondition(threshold int) TripCondition

NewConsecutiveFailureTripCondition creates a ConsecutiveFailureTripCondition.

func NewPercentageFailureTripCondition 

func NewPercentageFailureTripCondition(window int, threshold float64) TripCondition

NewPercentageFailureFailureTripCondition creates a PercentageFailureTripCondition.

func NewWindowFailureTripCondition 

func NewWindowFailureTripCondition(window time.Duration, threshold int) TripCondition

NewWindowFailureTripCondition creates a WindowFailureTripCondition.

type WindowFailureTripCondition 

type WindowFailureTripCondition struct {
	// contains filtered or unexported fields
}

WindowFailureTripCondition is a trip condition that trips the circuit breaker after a configurable number of failures occur within a configurable rolling window. The circuit breaker will remain in the open state until the time has elapsed for a failure to fall out of recent memory.

func (*WindowFailureTripCondition) Failure 

func (tc *WindowFailureTripCondition) Failure()

func (*WindowFailureTripCondition) ShouldTrip 

func (tc *WindowFailureTripCondition) ShouldTrip() bool

func (*WindowFailureTripCondition) Success 

func (tc *WindowFailureTripCondition) Success()

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.