yarpcconfig

package
v1.42.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 2, 2019 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package yarpcconfig implements a generic configuration system that may be used to build YARPC Dispatchers from configurations specified in different markup formats.

Usage

To build a Dispatcher, first create a new Configurator. This object will be responsible for loading your configuration. It does not yet know about the different transports, peer lists, etc. that you want to use. You can inform the Configurator about the different transports, peer lists, etc. by registering them using RegisterTransport, RegisterPeerChooser, RegisterPeerList, and RegisterPeerListUpdater. 

cfg := config.New()
cfg.MustRegisterTransport(http.TransportSpec())
cfg.MustRegisterPeerList(roundrobin.Spec())

This object is re-usable and may be stored as a singleton in your application. Custom transports, peer lists, and peer list updaters may be integrated with the configuration system by registering more TransportSpecs, PeerChooserSpecs, PeerListSpecs, and PeerListUpdaterSpecs with it.

Use LoadConfigFromYAML to load a yarpc.Config from YAML and pass that to yarpc.NewDispatcher. 

c, err := cfg.LoadConfigFromYAML("myservice", yamlConfig)
if err != nil {
	log.Fatal(err)
}
dispatcher := yarpc.NewDispatcher(c)

If you have already parsed your configuration from a different format, pass the parsed data to LoadConfig instead. 

var m map[string]interface{} = ...
c, err := cfg.LoadConfig("myservice", m)
if err != nil {
	log.Fatal(err)
}
dispatcher := yarpc.NewDispatcher(c)

NewDispatcher or NewDispatcherFromYAML may be used to get a yarpc.Dispatcher directly instead of a yarpc.Config. 

dispatcher, err := cfg.NewDispatcherFromYAML("myservice", yamlConfig)

Configuration parameters for the different transports, inbounds, and outbounds are defined in the TransportSpecs that were registered against the Configurator. A TransportSpec uses this information to build the corresponding Transport, Inbound and Outbound objects.

Configuration

The configuration may be specified in YAML or as any Go-level map[string]interface{}. The examples below use YAML for illustration purposes but other markup formats may be parsed into map[string]interface{} as long as the information provided is the same.

The configuration accepts the following top-level attributes: transports, inbounds, and outbounds. 

inbounds:
  # ...
outbounds:
  # ...
transports:
  # ...
logging:
  # ...

See the following sections for details on the logging, transports, inbounds, and outbounds keys in the configuration.

Inbound Configuration

The 'inbounds' attribute configures the different ways in which the service receives requests. It is represented as a mapping between inbound transport type and its configuration. For example, the following states that we want to receive requests over HTTP. 

inbounds:
  http:
    address: :8080

(For details on the configuration parameters of individual transport types, check the documentation for the corresponding transport package.)

If you want multiple inbounds of the same type, specify a different name for it and add a 'type' attribute to its configuration: 

inbounds:
  http:
    address: :8081
  http-deprecated:
    type: http
    address: :8080

Any inbound can be disabled by adding a 'disabled' attribute. 

inbounds:
  http:
    address: :8080
  http-deprecated:
    type: http
    disabled: true
    address: :8081

Outbound Configuration

The 'outbounds' attribute configures how this service makes requests to other YARPC-compatible services. It is represented as mapping between service name and outbound configuration. 

outbounds:
  keyvalue:
    # ..
  anotherservice:
    # ..

(For details on the configuration parameters of individual transport types, check the documentation for the corresponding transport package.)

The outbound configuration for a service has at least one of the following keys: unary, oneway. These specify the configurations for the corresponding RPC types for that service. For example, the following specifies that we make Unary requests to keyvalue service over TChannel and Oneway requests over HTTP. 

	keyvalue:
	  unary:
	    tchannel:
       peer: 127.0.0.1:4040
	  oneway:
	    http:
       url: http://127.0.0.1:8080/

For convenience, if there is only one outbound configuration for a service, it may be specified one level higher (without the 'unary' or 'oneway' attributes). In this case, that transport will be used to send requests for all compatible RPC types. For example, the HTTP transport supports both, Unary and Oneway RPC types so the following states that requests for both RPC types must be made over HTTP. 

keyvalue:
  http:
    url: http://127.0.0.1:8080/

Similarly, the following states that we only make Oneway requests to the "email" service and those are always made over HTTP. 

email:
  http:
    url: http://127.0.0.1:8080/

When the name of the target service differs from the outbound name, it may be overridden with the 'service' key. 

keyvalue:
  unary:
    # ...
  oneway:
    # ...
keyvalue-staging:
  service: keyvalue
  unary:
    # ...
  oneway:
    # ...

Peer Configuration

Transports that support peer management and selection through YARPC accept some additional keys in their outbound configuration.

An explicit peer may be specified for a supported transport by using the `peer` option. 

keyvalue:
  tchannel:
    peer: 127.0.0.1:4040

All requests made to this outbound will be made through this peer.

If a peer list was registered with the system, the name of the peer list may be used to specify a more complex peer selection and load balancing strategy. 

keyvalue:
  http:
    url: https://host/yarpc
    round-robin:
      peers:
        - 127.0.0.1:8080
        - 127.0.0.1:8081
        - 127.0.0.1:8082

In the example above, the system will round-robin the requests between the different addresses. In case of the HTTP transport, the URL will be used as a template for the HTTP requests made to these hosts.

Finally, the TransportSpec for a Transport may include named presets for peer lists in its definition. These may be referenced by name in the config using the `with` key. 

# Given a preset "dev-proxy" that was included in the TransportSpec, the
# following is valid.
keyvalue:
  http:
    url: https://host/yarpc
    with: dev-proxy

Transport Configuration

The 'transports' attribute configures the Transport objects that are shared between all inbounds and outbounds of that transport type. It is represented as a mapping between the transport name and its configuration. 

transports:
  http:
    keepAlive: 30s

(For details on the configuration parameters of individual transport types, check the documentation for the corresponding transport package.)

Logging Configuration

The 'logging' attribute configures how YARPC's observability middleware logs output. 

logging:
  levels:
    # ...

The following keys are supported under the 'levels' key, 

success
  Configures the level at which successful requests are logged.
  Defaults to "debug".
applicationError
  Configures the level at which application errors are logged.
  All Thrift exceptions are considered application errors.
  Defaults to "error".
failure
  Configures the level at which all other failures are logged.
  Default is "error".

For example, the following configuration will have the effect of logging Thrift exceptions for inbound and outbound calls ("Error handling inbound request" and "Error making outbound call") at info level instead of error. 

logging:
  levels:
    applicationError: info

The 'logging' attribute also has 'inbound' and 'outbound' sections to specify log levels that depend on the traffic direction. For example, the following configuration will only override the log level for successful outbound requests. 

logging:
  levels:
    inbound:
      success: debug

The log levels are: 

debug
info
warn
error
dpanic
panic
fatal

Customizing Configuration

When building your own TransportSpec, PeerListSpec, or PeerListUpdaterSpec, you will define functions accepting structs or pointers to structs which define the different configuration parameters needed to build that entity. These configuration parameters will be decoded from the user-specified configuration using a case-insensitive match on the field names.

Given the struct, 

type MyConfig struct {
	URL string
}

An object containing a `url`, `URL` or `Url` key with a string value will be accepted in place of MyConfig.

Configuration structs can use standard Go primitive types, time.Duration, maps, slices, and other similar structs. For example only, an outbound might accept a config containing an array of host:port structs (in practice, an outbound would use a config.PeerList to build a peer.Chooser). 

type Peer struct {
	Host string
	Port int
}

type MyOutboundConfig struct{ Peers []Peer }

The above will accept the following YAML: 

myoutbound:
  peers:
    - host: localhost
      port: 8080
    - host: anotherhost
      port: 8080

Field names can be changed by adding a `config` tag to fields in the configuration struct. 

type MyInboundConfig struct {
	Address string `config:"addr"`
}

This struct will accept the `addr` key, not `address`.

In addition to specifying the field name, the `config` tag may also include an `interpolate` option to request interpolation of variables in the form ${NAME} or ${NAME:default} at the time the value is decoded. By default, environment variables are used to fill these variables; this may be changed with the InterpolationResolver option.

Interpolation may be requested only for primitive fields and time.Duration. 

type MyConfig struct {
	Address string `config:"addr,interpolate"`
	Timeout time.Duration `config:",interpolate"`
}

Note that for the second field, we don't change the name with the tag; we only indicate that we want interpolation for that variable.

In the example above, values for both, Address and Timeout may contain strings in the form ${NAME} or ${NAME:default} anywhere in the value. These will be replaced with the value of the environment variable or the default (if specified) if the environment variable was unset. 

addr: localhost:${PORT}
timeout: ${TIMEOUT_SECONDS:5}s

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Backoff 

type Backoff struct {
	Exponential ExponentialBackoff `config:"exponential"`
}

Backoff specifies a backoff strategy, particularly for retries. The only supported strategy at time of writing is "exponential" with full jitter. This structure may be extended in the future to support registering alternate backoff strategies. 

exponential:
  first: 100ms
  max: 30s

func (Backoff) Strategy 

func (c Backoff) Strategy() (backoffapi.Strategy, error)

Strategy returns a backoff strategy constructor (in terms of the number of attempts already made) and the given configuration, or an error.

type Configurator 

type Configurator struct {
	// contains filtered or unexported fields
}

Configurator helps build Dispatchers using runtime configuration.

A new Configurator does not know about any transports, peer lists, or peer list updaters. Inform it about them by using the RegisterTransport, RegisterPeerList, and RegisterPeerListUpdater functions, or their Must* variants.

func New 

func New(opts ...Option) *Configurator

New sets up a new empty Configurator. The returned Configurator does not know about any Transports, peer lists, or peer list updaters.

func (*Configurator) LoadConfig 

func (c *Configurator) LoadConfig(serviceName string, data interface{}) (yarpc.Config, error)

LoadConfig loads a yarpc.Config from a map[string]interface{} or map[interface{}]interface{}.

See the module documentation for the shape the map[string]interface{} is expected to conform to.

func (*Configurator) LoadConfigFromYAML 

func (c *Configurator) LoadConfigFromYAML(serviceName string, r io.Reader) (yarpc.Config, error)

LoadConfigFromYAML loads a yarpc.Config from YAML data. Use LoadConfig if you have already parsed a map[string]interface{} or map[interface{}]interface{}.

func (*Configurator) MustRegisterPeerChooser added in v1.27.0 

func (c *Configurator) MustRegisterPeerChooser(s PeerChooserSpec)

MustRegisterPeerChooser registers the given PeerChooserSpec with the Configurator. This function panics if the PeerChooserSpec is invalid.

func (*Configurator) MustRegisterPeerList 

func (c *Configurator) MustRegisterPeerList(s PeerListSpec)

MustRegisterPeerList registers the given PeerListSpec with the Configurator. This function panics if the PeerListSpec is invalid.

func (*Configurator) MustRegisterPeerListUpdater 

func (c *Configurator) MustRegisterPeerListUpdater(s PeerListUpdaterSpec)

MustRegisterPeerListUpdater registers the given PeerListUpdaterSpec with the Configurator. This function panics if the PeerListUpdaterSpec is invalid.

func (*Configurator) MustRegisterTransport 

func (c *Configurator) MustRegisterTransport(t TransportSpec)

MustRegisterTransport registers the given TransportSpec with the Configurator. This function panics if the TransportSpec is invalid.

func (*Configurator) NewDispatcher 

func (c *Configurator) NewDispatcher(serviceName string, data interface{}) (*yarpc.Dispatcher, error)

NewDispatcher builds a new Dispatcher from the given configuration data.

func (*Configurator) NewDispatcherFromYAML 

func (c *Configurator) NewDispatcherFromYAML(serviceName string, r io.Reader) (*yarpc.Dispatcher, error)

NewDispatcherFromYAML builds a Dispatcher from the given YAML configuration.

func (*Configurator) RegisterPeerChooser added in v1.27.0 

func (c *Configurator) RegisterPeerChooser(s PeerChooserSpec) error

RegisterPeerChooser registers a PeerChooserSpec with the given Configurator, teaching it how to build peer choosers of this kind from configuration.

An error is returned if the PeerChooserSpec is invalid. Use MustRegisterPeerChooser to panic in the case of registration failure.

If a peer chooser with the same name already exists, it will be replaced.

If a peer list is registered with the same name, it will be ignored.

See PeerChooserSpec for details on how to integrate your own peer chooser with the system.

func (*Configurator) RegisterPeerList 

func (c *Configurator) RegisterPeerList(s PeerListSpec) error

RegisterPeerList registers a PeerListSpec with the given Configurator, teaching it how to build peer lists of this kind from configuration.

An error is returned if the PeerListSpec is invalid. Use MustRegisterPeerList to panic in the case of registration failure.

If a peer list with the same name already exists, it will be replaced.

If a peer chooser is registered with the same name, this list will be ignored.

See PeerListSpec for details on how to integrate your own peer list with the system.

func (*Configurator) RegisterPeerListUpdater 

func (c *Configurator) RegisterPeerListUpdater(s PeerListUpdaterSpec) error

RegisterPeerListUpdater registers a PeerListUpdaterSpec with the given Configurator, teaching it how to build peer list updaters of this kind from configuration.

Returns an error if the PeerListUpdaterSpec is invalid. Use MustRegisterPeerListUpdater to panic if the registration fails.

If a peer list updater with the same name already exists, it will be replaced.

See PeerListUpdaterSpec for details on how to integrate your own peer list updater with the system.

func (*Configurator) RegisterTransport 

func (c *Configurator) RegisterTransport(t TransportSpec) error

RegisterTransport registers a TransportSpec with the Configurator, teaching it how to load configuration and build inbounds and outbounds for that transport.

An error is returned if the TransportSpec is invalid. Use MustRegisterTransport if you want to panic in case of registration failure.

If a transport with the same name already exists, it will be replaced.

See TransportSpec for details on how to integrate your own transport with the system.

type ExponentialBackoff 

type ExponentialBackoff struct {
	First time.Duration `config:"first"`
	Max   time.Duration `config:"max"`
}

ExponentialBackoff details the exponential with full jitter backoff strategy. "first" defines the range of possible durations for the first attempt. Each subsequent attempt has twice the range of possible jittered delay duration. The range of possible values will not exceed "max", inclusive. 

first: 100ms
max: 30s

func (ExponentialBackoff) Strategy 

func (c ExponentialBackoff) Strategy() (backoffapi.Strategy, error)

Strategy returns an exponential backoff strategy (in terms of the number of attempts already made) and the given configuration.

type Kit 

type Kit struct {
	// contains filtered or unexported fields
}

Kit is an opaque object that carries context for the Configurator. Build functions that receive this object MUST NOT modify it.

func (*Kit) ServiceName 

func (k *Kit) ServiceName() string

ServiceName returns the name of the service for which components are being built.

type Option 

type Option func(*Configurator)

Option customizes a Configurator.

func InterpolationResolver 

func InterpolationResolver(f func(k string) (v string, ok bool)) Option

InterpolationResolver changes how interpolated variables in the configuration are resolved. By default, environment variables are used.

Variables will be interpolated using this function if a parsed field inside a configuration structure was annotated with config:",interpolate" or config:"$name,interpolate" where $name is encoded name of that field.

Only primitive types (numbers, strings, and time.Duration) are interpolated. 

type myConfig struct {
	Host string `config:"host,interpolate"`
	Port int `config:",interpolate"`
	Timeout time.Duration `config:",interpolate"`
}

type PeerChooser 

type PeerChooser struct {
	// contains filtered or unexported fields
}

PeerChooser facilitates decoding and building peer choosers. A peer chooser combines a peer list (which implements the peer selection strategy) and a peer list updater (which informs the peer list about different peers), allowing transports to rely on these two pieces for peer selection and load balancing.

Format

Peer chooser configuration may define only one of the following keys: `peer`, `with`, or the name of any registered PeerListSpec.

`peer` indicates that requests must be sent to a single peer. 

http:
  peer: 127.0.0.1:8080

Note that how this string is interpreted is transport-dependent.

`with` specifies that a named peer chooser preset defined by the transport should be used rather than defining one by hand in the configuration. 

# Given a dev-proxy preset on your TransportSpec,
http:
  with: dev-proxy

If the name of a registered PeerListSpec is the key, an object specifying the configuration parameters for the PeerListSpec is expected along with the name of a known peer list updater and its configuration. 

# cfg.RegisterPeerList(roundrobin.Spec())
round-robin:
  peers:
    - 127.0.0.1:8080
    - 127.0.0.1:8081

In the example above, there are no configuration parameters for the round robin peer list. The only remaining key is the name of the peer list updater: `peers` which is just a static list of peers.

Integration

To integrate peer choosers with your transport, embed this struct into your outbound configuration. 

type myOutboundConfig struct {
	config.PeerChooser

	Address string
}

Then in your Build*Outbound function, use the PeerChooser.BuildPeerChooser method to retrieve a peer chooser for your outbound. The following example only works if your transport implements the peer.Transport interface. 

func buildOutbound(cfg *myOutboundConfig, t transport.Transport, k *config.Kit) (transport.UnaryOutbound, error) {
	myTransport := t.(*MyTransport)
	peerChooser, err := cfg.BuildPeerChooser(myTransport, hostport.Identify, k)
	if err != nil {
		return nil, err
	}
	return myTransport.NewOutbound(peerChooser), nil
}

The *config.Kit received by the Build*Outbound function MUST be passed to the BuildPeerChooser function as-is.

Note that the keys for the PeerChooser: peer, with, and any peer list name, share the namespace with the attributes of your outbound configuration.

func (PeerChooser) BuildPeerChooser 

func (pc PeerChooser) BuildPeerChooser(transport peer.Transport, identify func(string) peer.Identifier, kit *Kit) (peer.Chooser, error)

BuildPeerChooser translates the decoded configuration into a peer.Chooser.

The identify function informs us how to convert string-based peer names into peer identifiers for the transport.

The Kit received by the Build*Outbound function MUST be passed to BuildPeerChooser as-is.

func (PeerChooser) Empty 

func (pc PeerChooser) Empty() bool

Empty returns true if the PeerChooser is empty, i.e., it does not have any keys defined.

This allows Build*Outbound functions to handle the case where the peer configuration is specified in a different way than the standard peer configuration.

type PeerChooserPreset 

type PeerChooserPreset struct {
	Name string

	// A function in the shape,
	//
	//  func(peer.Transport, *config.Kit) (peer.Chooser, error)
	//
	// Where the first argument is the transport object for which this preset
	// is being built.
	BuildPeerChooser interface{}
}

PeerChooserPreset defines a named preset for a peer chooser. Peer chooser presets may be used by specifying a `with` key in the outbound configuration. 

http:
  with: mypreset

type PeerChooserSpec added in v1.27.0 

type PeerChooserSpec struct {
	Name string

	// A function in the shape,
	//
	//  func(C, p peer.Transport, *config.Kit) (peer.Chooser, error)
	//
	// Where C is a struct or pointer to a struct defining the configuration
	// parameters needed to build this peer chooser.
	//
	// BuildPeerChooser is required.
	BuildPeerChooser interface{}
}

PeerChooserSpec specifies the configuration parameters for an outbound peer chooser. Peer choosers dictate how peers are selected for an outbound. These specifications are registered against a Configurator to teach it how to parse the configuration for that peer chooser and build instances of it.

For example, we could implement and register a peer chooser spec that selects peers based on advanced configuration or sharding information. 

	myoutbound:
	  tchannel:
	    mysharder:
       shard1: 1.1.1.1:1234
       ...

type PeerListSpec 

type PeerListSpec struct {
	Name string

	// A function in the shape,
	//
	//  func(C, peer.Transport, *config.Kit) (peer.ChooserList, error)
	//
	// Where C is a struct or pointer to a struct defining the configuration
	// parameters needed to build this peer list. Parameters on the struct
	// should not conflict with peer list updater names as they share the
	// namespace with these fields.
	//
	// BuildPeerList is required.
	BuildPeerList interface{}
}

PeerListSpec specifies the configuration parameters for an outbound peer list. Peer lists dictate the peer selection strategy and receive updates of new and removed peers from peer updaters. These specifications are registered against a Configurator to teach it how to parse the configuration for that peer list and build instances of it.

For example, we could implement and register a peer list spec that selects peers at random and a peer list updater which pushes updates to it by polling a specific DNS A record. 

myoutbound:
  random:
    dns:
      name: myservice.example.com

type PeerListUpdaterSpec 

type PeerListUpdaterSpec struct {
	// Name of the peer selection strategy.
	Name string

	// A function in the shape,
	//
	//  func(C, *config.Kit) (peer.Binder, error)
	//
	// Where C is a struct or pointer to a struct defining the configuration
	// parameters accepted by this peer chooser.
	//
	// The returned peer binder will receive the peer list specified alongside
	// the peer updater; it should return a peer updater that feeds updates to
	// that peer list once started.
	//
	// BuildPeerListUpdater is required.
	BuildPeerListUpdater interface{}
}

PeerListUpdaterSpec specifies the configuration parameters for an outbound peer list updater. Peer list updaters inform peer lists about peers as they are added or removed. These specifications are registered against a Configurator to teach it how to parse the configuration for that peer list updater and build instances of it.

For example, we could implement a peer list updater which monitors a specific file on the system for a list of peers and pushes updates to any peer list. 

myoutbound:
  round-robin:
    peers-file:
      format: json
      path: /etc/hosts.json

type TransportSpec 

type TransportSpec struct {
	// Name of the transport
	Name string

	// A function in the shape,
	//
	// 	func(C, *config.Kit) (transport.Transport, error)
	//
	// Where C is a struct or pointer to a struct defining the configuration
	// parameters accepted by this transport.
	//
	// This function will be called with the parsed configuration to build
	// Transport defined by this spec.
	BuildTransport interface{}

	// A function in the shape,
	//
	// 	func(C, transport.Transport, *config.Kit) (transport.Inbound, error)
	//
	// Where C is a struct or pointer to a struct defining the configuration
	// parameters for the inbound.
	//
	// This may be nil if this transport does not support inbounds.
	//
	// This function will be called with the parsed configuration and the
	// transport built by BuildTransport to build the inbound for this
	// transport.
	BuildInbound interface{}

	// The following two are functions in the shapes,
	//
	// 	func(C, transport.Transport, *config.Kit) (transport.UnaryOutbound, error)
	// 	func(C, transport.Transport, *config.Kit) (transport.OnewayOutbound, error)
	//
	// Where C is a struct or pointer to a struct defining the configuration
	// parameters for outbounds of that RPC type.
	//
	// Either value may be nil to indicate that the transport does not support
	// unary or oneway outbounds.
	//
	// These functions will be called with the parsed configurations and the
	// transport built by BuildTransport to build the unary and oneway
	// outbounds for this transport.
	BuildUnaryOutbound  interface{}
	BuildOnewayOutbound interface{}
	BuildStreamOutbound interface{}

	// Named presets.
	//
	// These may be used by specifying a `with` key in the outbound
	// configuration.
	PeerChooserPresets []PeerChooserPreset
}

TransportSpec specifies the configuration parameters for a transport. These specifications are registered against a Configurator to teach it how to parse the configuration for that transport and build instances of it.

Every TransportSpec MUST have a BuildTransport function. The spec may provide BuildInbound, BuildUnaryOutbound, and BuildOnewayOutbound functions if the Transport supports that functionality. For example, if a transport only supports incoming and outgoing Oneway requests, its spec will provide a BuildTransport, BuildInbound, and BuildOnewayOutbound function.

The signature of BuildTransport must have the shape: 

func(C, *config.Kit) (transport.Transport, error)

Where C is a struct defining the configuration parameters for the transport, the kit carries information and tools from the configurator to this and other builders.

The remaining Build* functions must have a similar signature, but also receive the transport instance. 

func(C, transport.Transport, *config.Kit) (X, error)

Where X is one of, transport.Inbound, transport.UnaryOutbound, or transport.OnewayOutbound.

For example, 

func(*OutboundConfig, transport.Transport) (transport.UnaryOutbound, error)

Is a function to build a unary outbound from its outbound configuration and the corresponding transport.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.