doubleteam

README

Double-Team

Synopsis

A HTTP Kafka producer that handles outages.

Usage

Double-Team can be used in two different modes: server and restore

Server

Server mode accepts HTTP post requests and publishes them to Kafka.

Restore

Restore mode sends messages from S3 to Kafka.

Configuration

Server

The Double-Team server ./double-team server can be configured with the following options:

Flag	Description	Environment Variable
--port	The address to bind to for the http server.	PORT
--log-level	The log level to use (options: debug, info, warn, error).	LOG_LEVEL
--log-format	Log format to use (eg.: json, terminal).	LOG_FORMAT
--log-tags	Additional tags for logs.	LOG_TAGS
--stats-dsn	The statistics service to send metrics to (e.g.: l2met://, prometheus://0.0.0.0:8082).	STATS_DSN
--stats-prefix	Prefix for statistics.	STATS_PREFIX
--stats-tags	Additional tags for stats.	STATS_TAGS
--kafka.brokers	The kafka seed brokers connect to. Format: 'ip:port' (multiple allowed).	DOUBLE_TEAM_KAFKA_BROKERS
--kafka.version	Version of Kafka for producing messages: '2.3.0'.	DOUBLE_TEAM_KAFKA_VERSION
--kafka.retry	The number of times to retry sending to Kafka.	DOUBLE_TEAM_KAFKA_RETRY
--s3.endpoint	The S3 endpoint to use. This is mainly for debugging.	DOUBLE_TEAM_S3_ENDPOINT
--s3.region	The S3 region the bucket exists in.	DOUBLE_TEAM_S3_REGION
--s3.bucket	The S3 bucket to write messages to.	DOUBLE_TEAM_S3_BUCKET

Restore

The Double-Team server ./double-team restore can be configured with the following options:

Flag	Description	Environment Variable
--log-level	The log level to use (options: debug, info, warn, error).	LOG_LEVEL
--log-format	Log format to use (eg.: json, terminal).	LOG_FORMAT
--log-tags	Additional tags for logs.	LOG_TAGS
--stats-dsn	The statistics service to send metrics to (e.g.: l2met://, prometheus://0.0.0.0:8082).	STATS_DSN
--stats-prefix	Prefix for statistics.	STATS_PREFIX
--stats-tags	Additional tags for stats.	STATS_TAGS
--kafka.brokers	The kafka seed brokers connect to. Format: 'ip:port' (multiple allowed).	DOUBLE_TEAM_KAFKA_BROKERS
--kafka.version	Version of Kafka for producing messages: '2.3.0'.	DOUBLE_TEAM_KAFKA_VERSION
--kafka.retry	The number of times to retry sending to Kafka.	DOUBLE_TEAM_KAFKA_RETRY
--s3.endpoint	The S3 endpoint to use. This is mainly for debugging.	DOUBLE_TEAM_S3_ENDPOINT
--s3.region	The S3 region the bucket exists in.	DOUBLE_TEAM_S3_REGION
--s3.bucket	The S3 bucket to read messages from.	DOUBLE_TEAM_S3_BUCKET

Server HTTP Endpoints

POST /

Accepts a JSON payload with the message topic and data.

Payload:

{
	"topic": "test",
	"key": "key",
	"data": "test data"
}

GET /health

Gets the current health status of the server. Returns a 200 status code if the server is healthy, otherwise a 503 status code

License

MIT-License. As is. No warranties whatsoever. Mileage may vary. Batteries not included.

Documentation

Index

• type Application
  • func NewApplication(ctx context.Context, producers []streaming.Producer, queueSize int) *Application
  • func (a *Application) Close() error
  • func (a *Application) IsHealthy() error
  • func (a *Application) Send(topic string, key, data []byte)

Types

type Application

type Application struct {
	// contains filtered or unexported fields
}

Application represents the application.

func NewApplication

func NewApplication(ctx context.Context, producers []streaming.Producer, queueSize int) *Application

NewApplication creates an instance of Application.

func (*Application) Close

func (a *Application) Close() error

Close closes the application and cleans up.

func (*Application) IsHealthy

func (a *Application) IsHealthy() error

IsHealthy checks the health of the Application.

func (*Application) Send

func (a *Application) Send(topic string, key, data []byte)

Send sends a message to the producer chain.