teacup

README

teacup

Teacup provides tooling for quickly building microservices that operate in a consistent, managed environment.

Workflow

To create a new Teacup microservice the normal work flow involves:

1. Create a new git repo.
2. Run go mod init in project root
3. Run go get github.com/betafish-inc/teacup to add the teacup dependency.
4. Create go launcher in _apps/service/main.go - using a standard naming allows tools to automatically build.
5. If using GitHub, add a GitHub Actions Workflow in .github/workflows/teacup.yaml using the template from _apps/tools/teacup/teacup.yaml.
6. Implement your service in service specific go packages in the project.

TODO it would be nice to add a tool as part of the Teacup project to automate the lion's share of these tasks. e.g. teacup init.

Building

Install Go and in the project root run go get -v -t -d ./... to fetch all go dependencies.

Typical Go development process:

• Unit tests: go test
• Run: go run _apps/subscriber/main.go
• Build: go build -o dist/subscriber _apps/subscriber/main.go

Runtime Environment

A Teacup based microservice operates in a standardized environment. Typically, this environment will run within a Docker Swarm or Kubernetes cluster. However, the same environment can also be run on bare OS's, Nomad clusters, etc.

In particular, Teacup assumes the following services are available:

• NATS.io for control plane and messaging
• Redis for data
• PostgreSQL for SQL data

Configuration for accessing these services will be searched in the following order:

1. Environmental variables.
2. DNS SRV records.
3. Default ports on localhost.

DNS SRV: This is most common in Consul managed environments. DNS SRV records must resolve using the default operating system DNS resolution system for Teacup microservices to lookup the information required.

For local development, run all the services your microservice relies on using default ports. The easiest is to install Docker and create a file named docker-compose.yml containing the following:

version: '3'
services:
  nats:
    image: nats:latest
    ports:
      - "6222:6222"
  redis:
    image: redislabs/redistimeseries
    ports:
      - "6379:6379"

Then run docker-compose up to start the services. Once the docker containers are running go run _apps/subscriber/main.go to run the subscriber microservice example. You'll need to send data to the same subject using go run _apps/publisher/main.go.

NATS.io

Teacup supports NATS.io "core" servers running in an optional cluster and will search for the following configuration options:

1. NATS_ADDR environmental variable. Should contain a comma separated list of host and port pairs e.g. 100.0.1.1:6222,100.0.1.2:6222.
2. nats.service.consul & nats.local DNS SRV entry.
3. localhost:6222 fall back to the most likely default configuration.

Redis

Teacup will attempt to connect to Redis using a connection URL:

• REDIS_URL environmental variable. Should be in the following format: redis://[:password@]host:port/db.

If the connection URL environmental variable doesn't exist, Teacup will search the following locations for Redis connection configuration information:

1. REDIS_ADDR environmental variable. Should contain the host and port like localhost:6379.
2. redis.service.consul & redis.local DNS SRV entry.
3. localhost:6379 fall back to the most likely default configuration.

Optionally, set a password and database using:

• REDIS_PASSWORD and REDIS_DATABASE environmental variables.

The default is no password, and database 0.

PostgreSQL

TBD

Documentation

Overview

Package teacup provides simple tooling for rapidly building microservices that will operate in a standardized environment. That environment currently assumes:

• service discovery using Hashicorp Consul https://hashicorp.com/consul
• secrets using Hashicorp Vault https://hashicorp.com/vault
• message queues using nsq.io https://nsq.io
• data storage using Redis https://redis.io

Index

• type Producer
  • func (p *Producer) Publish(_ context.Context, topic string, body []byte) error
  • func (p *Producer) Request(ctx context.Context, topic string, body []byte) ([]byte, error)
• type Queue
  • func (q *Queue) Producer(ctx context.Context) *Producer
  • func (q *Queue) Sub(ctx context.Context, topic, channel string, sub Subscriber)
• type Subscriber
• type SubscriberFunc
  • func (f SubscriberFunc) Message(ctx context.Context, t *Teacup, topic, channel string, msg []byte) error
• type Teacup
  • func NewTeacup() *Teacup
  • func (t *Teacup) Context() context.Context
  • func (t *Teacup) Option(_ context.Context, key string) (string, error)
  • func (t *Teacup) Queue() *Queue
  • func (t *Teacup) Redis(ctx context.Context) (*redis.Client, error)
  • func (t *Teacup) Register(worker Worker)
  • func (t *Teacup) Secret(_ context.Context, key string) (string, error)
  • func (t *Teacup) ServiceAddr(ctx context.Context, name string, port int) string
  • func (t *Teacup) ServiceAddrs(ctx context.Context, name string, port int) []string
  • func (t *Teacup) Start()
  • func (t *Teacup) Stop()
• type Worker
• type WorkerFunc
  • func (f WorkerFunc) Start(ctx context.Context, t *Teacup)
• type WorkerReg

Types

type Producer

type Producer struct {
	// contains filtered or unexported fields
}

Producer provides an abstract way to publish messages to queues. It follows the nsq implementation closely and on other platforms, many operations may be NOOPs.

func (*Producer) Publish

func (p *Producer) Publish(_ context.Context, topic string, body []byte) error

Publish synchronously publishes a message body to the specified topic, returning an error if publish failed

func (*Producer) Request

func (p *Producer) Request(ctx context.Context, topic string, body []byte) ([]byte, error)

Request synchronously publishes a message request to the specified topic and waits for a response.

type Queue

type Queue struct {
	Client *nats.Conn
	// contains filtered or unexported fields
}

Queue abstracts the underlying message queue from microservices. We will use nsq for "native" deployments but want to have the freedom of easily using a platform provided queue service if available.

func (*Queue) Producer

func (q *Queue) Producer(ctx context.Context) *Producer

Producer provides a Producer ready for sending messages to queues.

func (*Queue) Sub

func (q *Queue) Sub(ctx context.Context, topic, channel string, sub Subscriber)

Sub to an event topic and channel. The returned CancelFunc can be used to cancel the subscription.

type Subscriber

type Subscriber interface {
	// Message is called for each received message from the given topic/channel combination.
	// The "reply" channel subject is added to the context for message handlers that need to respond to
	// a message queue request.
	//
	// TODO make the following an actual go doc example
	//
	// Example:
	//
	// Message(ctx context.Context, t *Teacup, topic, channel string, msg []byte) error {
	//   t.Publisher(ctx).Publish(ctx, ctx.Value("reply"), []byte("{\"foo\":\"bar\"}"))
	// }
	//
	//
	// Errors can be used to force a requeue of the message.
	// TODO we need better documentation on exactly how errors are used to force requeues (and any other actions)
	Message(ctx context.Context, t *Teacup, topic, channel string, msg []byte) error
}

Subscriber allows microservices to process queue messages on a topic/channel without dealing directly with the underlying message queue (including all it's various settings and configuration options). Instead, microservices implement Subscriber and Sub to a topic/channel and receive messages as they are ready for processing.

type SubscriberFunc

type SubscriberFunc func(ctx context.Context, t *Teacup, topic, channel string, msg []byte) error

The SubscriberFunc type is an adapter to allow ordinary functions to act as Subscribers. If f is a function with the appropriate signature, SubscriberFunc(f) is a Subscriber that calls f.

func (SubscriberFunc) Message

func (f SubscriberFunc) Message(ctx context.Context, t *Teacup, topic, channel string, msg []byte) error

Message calls f(cxt, topic, channel, msg).

type Teacup

type Teacup struct {
	// contains filtered or unexported fields
}

Teacup provides essential tools for writing microservices with minimum boilerplate. Create an empty Teacup and then register at least one Worker or Sub then call Teacup.Start() to begin running.

func NewTeacup

func NewTeacup() *Teacup

NewTeacup creates a new Teacup instance ready for use.

func (*Teacup) Context

func (t *Teacup) Context() context.Context

Context returns the context associated with the main Teacup goroutine.

func (*Teacup) Option

func (t *Teacup) Option(_ context.Context, key string) (string, error)

Option returns the configuration setting associated with a key name. The function searches the environment first and if not found, tries to obtain the value from Consul.

func (*Teacup) Queue

func (t *Teacup) Queue() *Queue

Queue returns a queue helper for interacting with the message bus.

func (*Teacup) Redis

func (t *Teacup) Redis(ctx context.Context) (*redis.Client, error)

Redis returns a redis client ready to use. The first call to Redis() will dial the Redis server and the provided context is used to control things like timeouts.

func (*Teacup) Register

func (t *Teacup) Register(worker Worker)

Register a worker for running. THe Worker's Start() method will be called some time after the Teacup.Start() method is called and should not return unless the provided context is done or a fatal error occurs.

func (*Teacup) Secret

func (t *Teacup) Secret(_ context.Context, key string) (string, error)

Secret returns a secret associated with a key name. Secret searches the environment first and if not found, tries to obtain the value from Vault.

func (*Teacup) ServiceAddr

func (t *Teacup) ServiceAddr(ctx context.Context, name string, port int) string

ServiceAddr searches for a service address `name` by checking for:

* `NAME_ADDR` - an environmental variable containing a host:port pair * `name.service.consul` & `name.local` - a SRV record * `localhost:<port>` - a fallback assuming the service is on the default port

func (*Teacup) ServiceAddrs

func (t *Teacup) ServiceAddrs(ctx context.Context, name string, port int) []string

ServiceAddr searches for a service address `name` by checking for:

* `NAME_ADDR` - an environmental variable containing comma separated host:port pairs. * `name.service.consul` & `name.local` - one or more SRV records * `localhost:<port>` - a fallback assuming the service is on the default port

func (*Teacup) Start

func (t *Teacup) Start()

Start teacup running as long as one or more Worker or Sub are registered. Start does not return until the microservice is ready to terminate.

func (*Teacup) Stop

func (t *Teacup) Stop()

Stop teacup from running.

type Worker

type Worker interface {
	// Start the Worker and run until the provided context's done channel is closed. The provided Teacup
	// reference can be used to access other Teacup managed services.
	Start(ctx context.Context, t *Teacup)
}

Worker allows microservices to run "forever" without having to worry about initializing an environment or properly responding to system inputs. The worker should watch the context done channel to know when to terminate.

type WorkerFunc

type WorkerFunc func(ctx context.Context, t *Teacup)

The WorkerFunc type is an adapter to allow ordinary functions to act as Workers. If f is a function with the appropriate signature, WorkerFunc(f) is a Worker that calls f.

func (WorkerFunc) Start

func (f WorkerFunc) Start(ctx context.Context, t *Teacup)

Start calls f(ctx, t).

type WorkerReg

type WorkerReg struct {
	Worker Worker
	// contains filtered or unexported fields
}

WorkerReg contains information about a Worker registration with Teacup.