forwarder

package
v0.0.2-0...-4ce78c8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 17, 2021 License: Apache-2.0 Imports: 23 Imported by: 0

README

package forwarder

This package is responsible for sending payloads to the backend. Payloads can come from different sources in different formats, the forwarder will not inspect them.

The forwarder can receive multiple domains with a list of API keys for each of them. Every payload will be sent to every domain/API keys couple, this became a Transaction. Transactions will be retried on error. The newest transactions will be retried first. Transactions are consumed by Workers asynchronously.

Usage


KeysPerDomains := map[string][]string{
	"http://api.datadog.com": {"my_secret_key_1", "my_secret_key_2"},
	"http://debug.api.com":   {"secret_api"},
}

forwarder := forwarder.NewForwarder(KeysPerDomains)
forwarder.NumberOfWorkers = 1 // default: config.Datadog.GetInt("forwarder_num_workers")
forwarder.Start()

// ...

payload1 := []byte("some payload")
payload2 := []byte("another payload")
forwarder.SubmitSeries(Payloads{&payload1, &payload2}
)

// ...

forwarder.Stop()

Configuration

There are several settings that influence the behavior of the forwarder.

Exponential backoff and circuit breaker settings
  • forwarder_backoff_factor - This controls the overlap between consecutive retry interval ranges. When set to 2, there is a guarantee that there will be no overlap. The overlap will asymptotically approach 50% the higher the value is set. Values less then 2 are verboten as there will be range gaps. Default: 2
  • forwarder_backoff_base - This controls the rate of exponential growth. Also, you can calculate the start of the very first retry interval range by evaluating the following expression: forwarder_backoff_base / forwarder_backoff_factor * 2. Default: 2
  • forwarder_backoff_max - This is the maximum number of seconds to wait for a retry. Default: 64
  • forwarder_recovery_interval - This controls how many retry interval ranges to step down for an endpoint upon success. Default: 2
  • forwarder_recovery_reset - Whether or not a successful request should completely clear an endpoint's error count. Default: false

Internal

The forwarder is composed of multiple parts:

DefaultForwarder

DefaultForwarder it the default implementation of the Forwarder interface (and the only one for now). This class is in charge of receiving payloads, creating the HTTP transactions and distributing them among every domainForwarder.

domainForwarder

The agent can be configured to send the same payload to multiple destinations. Each destination (or domain) can be configured with 1 or more API keys. Every payload will be sent to each domain/API key pair.

A domainForwarder is in charge of sending payloads to one domain. This avoids slowing down every domain when one is down/slow. Each domainForwarder will have a number of dedicated Worker to process Transaction. We process new transactions first and then (when the workers have time) we retry the erroneous ones (newest transactions are retried first).

We start dropping transactions (oldest first) when the sum of all the payload sizes is bigger than forwarder_retry_queue_payloads_max_size (see the agent configuration).

Disclaimer: using multiple API keys with the Datadog backend will multiply your billing ! Most customers will only use one API key.

Worker

A Worker processes transactions coming from 2 queues: HighPrio and LowPrio. New transactions are sent to the HighPrio queue and the ones to retry are sent to LowPrio. A Worker is dedicated to on domain (ie: domainForwarder).

blockedEndpoints (or exponential backoff)

When a transaction fails to be sent to a backend we blacklist that particular endpoints for some time to avoid flooding an unavailable endpoint (the transactions will be retried later). A blacklist is specific to one endpoint on one domain (ie: "http(s):///"). The blacklist time will grow, up to a maximum, has more and more errors are encountered for that endpoint and is gradually cleared when a transaction is successful. The blacklist is shared by all workers.

Transaction

A HTTPTransaction contains every information about a payload and how/where to send it. On failure a transaction will be retried later (see blockedEndpoints).

Documentation

Index

Constants

View Source 
const (
	// PayloadTypePod is the name of the pod payload type
	PayloadTypePod = "pod"
	// PayloadTypeDeployment is the name of the deployment payload type
	PayloadTypeDeployment = "deployment"
	// PayloadTypeReplicaSet is the name of the replica set payload type
	PayloadTypeReplicaSet = "replicaset"
	// PayloadTypeService is the name of the service payload type
	PayloadTypeService = "service"
	// PayloadTypeNode is the name of the node payload type
	PayloadTypeNode = "node"
	// PayloadTypeCluster is the name of the cluster payload type
	PayloadTypeCluster = "cluster"
)
View Source 
const (
	// Stopped represent the internal state of an unstarted Forwarder.
	Stopped uint32 = iota
	// Started represent the internal state of an started Forwarder.
	Started
)

Variables

This section is empty.

Functions

func HasFeature 

func HasFeature(features, flag Features) bool

HasFeature lets you know if a specific feature flag is set in a feature set

Types

type DefaultForwarder 

type DefaultForwarder struct {
	// NumberOfWorkers Number of concurrent HTTP request made by the DefaultForwarder (default 4).
	NumberOfWorkers int
	// contains filtered or unexported fields
}

DefaultForwarder is the default implementation of the Forwarder.

func NewDefaultForwarder 

func NewDefaultForwarder(options *Options) *DefaultForwarder

NewDefaultForwarder returns a new DefaultForwarder.

func (*DefaultForwarder) Start 

func (f *DefaultForwarder) Start() error

Start initialize and runs the forwarder.

func (*DefaultForwarder) State 

func (f *DefaultForwarder) State() uint32

State returns the internal state of the forwarder (Started or Stopped)

func (*DefaultForwarder) Stop 

func (f *DefaultForwarder) Stop()

Stop all the component of a forwarder and free resources

func (*DefaultForwarder) SubmitAgentChecksMetadata 

func (f *DefaultForwarder) SubmitAgentChecksMetadata(payload Payloads, extra http.Header) error

SubmitAgentChecksMetadata will send a agentchecks_metadata tag type payload to Datadog backend.

func (*DefaultForwarder) SubmitConnectionChecks 

func (f *DefaultForwarder) SubmitConnectionChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitConnectionChecks sends connection checks

func (*DefaultForwarder) SubmitContainerChecks 

func (f *DefaultForwarder) SubmitContainerChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitContainerChecks sends container checks

func (*DefaultForwarder) SubmitEvents 

func (f *DefaultForwarder) SubmitEvents(payload Payloads, extra http.Header) error

SubmitEvents will send an event type payload to Datadog backend.

func (*DefaultForwarder) SubmitHostMetadata 

func (f *DefaultForwarder) SubmitHostMetadata(payload Payloads, extra http.Header) error

SubmitHostMetadata will send a host_metadata tag type payload to Datadog backend.

func (*DefaultForwarder) SubmitMetadata 

func (f *DefaultForwarder) SubmitMetadata(payload Payloads, extra http.Header) error

SubmitMetadata will send a metadata type payload to Datadog backend.

func (*DefaultForwarder) SubmitOrchestratorChecks 

func (f *DefaultForwarder) SubmitOrchestratorChecks(payload Payloads, extra http.Header, payloadType string) (chan Response, error)

SubmitOrchestratorChecks sends orchestrator checks

func (*DefaultForwarder) SubmitProcessChecks 

func (f *DefaultForwarder) SubmitProcessChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitProcessChecks sends process checks

func (*DefaultForwarder) SubmitRTContainerChecks 

func (f *DefaultForwarder) SubmitRTContainerChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitRTContainerChecks sends real time container checks

func (*DefaultForwarder) SubmitRTProcessChecks 

func (f *DefaultForwarder) SubmitRTProcessChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitRTProcessChecks sends real time process checks

func (*DefaultForwarder) SubmitSeries 

func (f *DefaultForwarder) SubmitSeries(payload Payloads, extra http.Header) error

SubmitSeries will send a series type payload to Datadog backend.

func (*DefaultForwarder) SubmitServiceChecks 

func (f *DefaultForwarder) SubmitServiceChecks(payload Payloads, extra http.Header) error

SubmitServiceChecks will send a service check type payload to Datadog backend.

func (*DefaultForwarder) SubmitSketchSeries 

func (f *DefaultForwarder) SubmitSketchSeries(payload Payloads, extra http.Header) error

SubmitSketchSeries will send payloads to Datadog backend - PROTOTYPE FOR PERCENTILE

func (*DefaultForwarder) SubmitV1CheckRuns 

func (f *DefaultForwarder) SubmitV1CheckRuns(payload Payloads, extra http.Header) error

SubmitV1CheckRuns will send service checks to v1 endpoint (this will be removed once the backend handles v2 endpoints).

func (*DefaultForwarder) SubmitV1Intake 

func (f *DefaultForwarder) SubmitV1Intake(payload Payloads, extra http.Header) error

SubmitV1Intake will send payloads to the universal `/intake/` endpoint used by Agent v.5

type Features 

type Features uint8

Features is a bitmask to enable specific forwarder features 

const (
	// CoreFeatures bitmask to enable specific core features
	CoreFeatures Features = 1 << iota
	// TraceFeatures bitmask to enable specific trace features
	TraceFeatures
	// ProcessFeatures bitmask to enable specific process features
	ProcessFeatures
	// SysProbeFeatures bitmask to enable specific system-probe features
	SysProbeFeatures
)

func ClearFeature 

func ClearFeature(features, flag Features) Features

ClearFeature clears forwarder features from a feature set

func SetFeature 

func SetFeature(features, flag Features) Features

SetFeature sets forwarder features in a feature set

func ToggleFeature 

func ToggleFeature(features, flag Features) Features

ToggleFeature toggles forwarder features in a feature set

type Forwarder 

type Forwarder interface {
	Start() error
	Stop()
	SubmitV1Intake(payload Payloads, extra http.Header) error
	SubmitV1CheckRuns(payload Payloads, extra http.Header) error
	SubmitSeries(payload Payloads, extra http.Header) error
	SubmitEvents(payload Payloads, extra http.Header) error
	SubmitServiceChecks(payload Payloads, extra http.Header) error
	SubmitSketchSeries(payload Payloads, extra http.Header) error
	SubmitHostMetadata(payload Payloads, extra http.Header) error
	SubmitAgentChecksMetadata(payload Payloads, extra http.Header) error
	SubmitMetadata(payload Payloads, extra http.Header) error
	SubmitProcessChecks(payload Payloads, extra http.Header) (chan Response, error)
	SubmitRTProcessChecks(payload Payloads, extra http.Header) (chan Response, error)
	SubmitContainerChecks(payload Payloads, extra http.Header) (chan Response, error)
	SubmitRTContainerChecks(payload Payloads, extra http.Header) (chan Response, error)
	SubmitConnectionChecks(payload Payloads, extra http.Header) (chan Response, error)
	SubmitOrchestratorChecks(payload Payloads, extra http.Header, payloadType string) (chan Response, error)
}

Forwarder interface allows packages to send payload to the backend

type Options 

type Options struct {
	NumberOfWorkers                int
	RetryQueuePayloadsTotalMaxSize int
	DisableAPIKeyChecking          bool
	EnabledFeatures                Features
	APIKeyValidationInterval       time.Duration
	KeysPerDomain                  map[string][]string
	ConnectionResetInterval        time.Duration
	CompletionHandler              transaction.HTTPCompletionHandler
}

Options contain the configuration options for the DefaultForwarder

func NewOptions 

func NewOptions(keysPerDomain map[string][]string) *Options

NewOptions creates new Options with default values

type Payloads 

type Payloads []*[]byte

Payloads is a slice of pointers to byte arrays, an alias for the slices of payloads we pass into the forwarder

type Response 

type Response struct {
	Domain     string
	Body       []byte
	StatusCode int
	Err        error
}

Response contains the response details of a successfully posted transaction

type SyncForwarder 

type SyncForwarder struct {
	// contains filtered or unexported fields
}

SyncForwarder is a very simple Forwarder synchronously sending the data to the intake. It doesn't ship any retry mechanism for now.

func NewSyncForwarder 

func NewSyncForwarder(keysPerDomains map[string][]string, timeout time.Duration) *SyncForwarder

NewSyncForwarder returns a new synchronous forwarder.

func (*SyncForwarder) Start 

func (f *SyncForwarder) Start() error

Start starts the sync forwarder: nothing to do.

func (*SyncForwarder) Stop 

func (f *SyncForwarder) Stop()

Stop stops the sync forwarder: nothing to do.

func (*SyncForwarder) SubmitAgentChecksMetadata 

func (f *SyncForwarder) SubmitAgentChecksMetadata(payload Payloads, extra http.Header) error

SubmitAgentChecksMetadata will send a agentchecks_metadata tag type payload to Datadog backend.

func (*SyncForwarder) SubmitConnectionChecks 

func (f *SyncForwarder) SubmitConnectionChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitConnectionChecks sends connection checks

func (*SyncForwarder) SubmitContainerChecks 

func (f *SyncForwarder) SubmitContainerChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitContainerChecks sends container checks

func (*SyncForwarder) SubmitEvents 

func (f *SyncForwarder) SubmitEvents(payload Payloads, extra http.Header) error

SubmitEvents will send an event type payload to Datadog backend.

func (*SyncForwarder) SubmitHostMetadata 

func (f *SyncForwarder) SubmitHostMetadata(payload Payloads, extra http.Header) error

SubmitHostMetadata will send a host_metadata tag type payload to Datadog backend.

func (*SyncForwarder) SubmitMetadata 

func (f *SyncForwarder) SubmitMetadata(payload Payloads, extra http.Header) error

SubmitMetadata will send a metadata type payload to Datadog backend.

func (*SyncForwarder) SubmitOrchestratorChecks 

func (f *SyncForwarder) SubmitOrchestratorChecks(payload Payloads, extra http.Header, payloadType string) (chan Response, error)

SubmitOrchestratorChecks sends orchestrator checks

func (*SyncForwarder) SubmitProcessChecks 

func (f *SyncForwarder) SubmitProcessChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitProcessChecks sends process checks

func (*SyncForwarder) SubmitRTContainerChecks 

func (f *SyncForwarder) SubmitRTContainerChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitRTContainerChecks sends real time container checks

func (*SyncForwarder) SubmitRTProcessChecks 

func (f *SyncForwarder) SubmitRTProcessChecks(payload Payloads, extra http.Header) (chan Response, error)

SubmitRTProcessChecks sends real time process checks

func (*SyncForwarder) SubmitSeries 

func (f *SyncForwarder) SubmitSeries(payload Payloads, extra http.Header) error

SubmitSeries will send a series type payload to Datadog backend.

func (*SyncForwarder) SubmitServiceChecks 

func (f *SyncForwarder) SubmitServiceChecks(payload Payloads, extra http.Header) error

SubmitServiceChecks will send a service check type payload to Datadog backend.

func (*SyncForwarder) SubmitSketchSeries 

func (f *SyncForwarder) SubmitSketchSeries(payload Payloads, extra http.Header) error

SubmitSketchSeries will send payloads to Datadog backend - PROTOTYPE FOR PERCENTILE

func (*SyncForwarder) SubmitV1CheckRuns 

func (f *SyncForwarder) SubmitV1CheckRuns(payload Payloads, extra http.Header) error

SubmitV1CheckRuns will send service checks to v1 endpoint (this will be removed once the backend handles v2 endpoints).

func (*SyncForwarder) SubmitV1Intake 

func (f *SyncForwarder) SubmitV1Intake(payload Payloads, extra http.Header) error

SubmitV1Intake will send payloads to the universal `/intake/` endpoint used by Agent v.5

type Worker 

type Worker struct {
	// Client the http client used to processed transactions.
	Client *http.Client
	// HighPrio is the channel used to receive high priority transaction from the Forwarder.
	HighPrio <-chan transaction.Transaction
	// LowPrio is the channel used to receive low priority transaction from the Forwarder.
	LowPrio <-chan transaction.Transaction
	// RequeueChan is the channel used to send failed transaction back to the Forwarder.
	RequeueChan chan<- transaction.Transaction
	// contains filtered or unexported fields
}

Worker consumes Transaction (aka transactions) from the Forwarder and processes them. If the transaction fails to be processed the Worker will send it back to the Forwarder to be retried later.

func NewWorker 

func NewWorker(
	highPrioChan <-chan transaction.Transaction,
	lowPrioChan <-chan transaction.Transaction,
	requeueChan chan<- transaction.Transaction,
	blocked *blockedEndpoints) *Worker

NewWorker returns a new worker to consume Transaction from inputChan and push back erroneous ones into requeueChan.

func (*Worker) ScheduleConnectionReset 

func (w *Worker) ScheduleConnectionReset()

ScheduleConnectionReset allows signaling the worker that all connections should be recreated before sending the next transaction. Returns immediately.

func (*Worker) Start 

func (w *Worker) Start()

Start starts a Worker.

func (*Worker) Stop 

func (w *Worker) Stop(purgeHighPrio bool)

Stop stops the worker.

Source Files

View all Source files

Directories

Path Synopsis
internal
retry

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.