kit

README

Go kit

Go kit is a distributed programming toolkit for building microservices in large organizations. We solve common problems in distributed systems, so you can focus on your business logic.

• Mailing list: go-kit
• Slack: gophers.slack.com #go-kit (invite)

Documentation

Examples

Perhaps the best way to understand Go kit is to follow along as we build an example service from first principles. This can serve as a blueprint for your own new service, or demonstrate how to adapt your existing service to use Go kit components.

Endpoint

Go kit primarily deals in the RPC messaging pattern. We use an abstraction called an endpoint to model individual RPCs. An endpoint can be implemented by a server, and called by a client. It's the fundamental building block of many Go kit components.

Circuit breaker

The circuitbreaker package provides endpoint adapters to several popular circuit breaker libraries. Circuit breakers prevent thundering herds, and improve resiliency against intermittent errors. Every client-side endpoint should be wrapped in a circuit breaker.

Rate limiter

The ratelimit package provides endpoint adapters to rate limiting packages. Rate limiters are equally applicable to both server- and client-side endpoints. Use rate limiters to enforce upper thresholds on incoming or outgoing request throughput.

Transport

The transport package provides helpers to bind endpoints to specific serialization mechanisms. At the moment, Go kit just provides helpers for simple JSON over HTTP. If your organization uses a fully-featured transport, bindings are typically provided by the Go library for the transport, and there's not much for Go kit to do. In those cases, see the examples to understand how to write adapters for your endpoints. For now, see the addsvc to understand how transport bindings work. We'll soon have specific examples for Thrift, gRPC, net/rpc, and JSON over HTTP. Avro and JSON/RPC support is planned.

Logging

Services produce logs to be consumed later, either by humans or machines. Humans might be interested in debugging errors, or tracing specific requests. Machines might be interested in counting interesting events, or aggregating information for offline processing. In both cases, it's important that the log messages be structured and actionable. Go kit's log package is designed to encourage both of these best practices.

Metrics (Instrumentation)

Services can't be considered production-ready until they're thoroughly instrumented with metrics that track counts, latency, health, and other periodic or per-request information. Go kit's metrics package provides a robust common set of interfaces for instrumenting your service. Bindings exist for common backends, from expvar to statsd to Prometheus.

Request tracing

As your infrastructure grows, it becomes important to be able to trace a request, as it travels through multiple services and back to the user. Go kit's tracing package provides enhancements for your endpoints and transport bindings to capture information about requests and emit them to request tracing systems. (Currently, Zipkin is supported; Appdash support is planned.)

Service discovery and load balancing

If your service calls another service, it needs to know how to find it, and should intelligently spread its load among those discovered instances. Go kit's loadbalancer package provides client-side endpoint middleware to solve that problem, whether your organization uses static hosts or IPs, DNS SRV records, Consul, etcd, or Zookeeper. And if you use a custom system, it's very easy to write your own Publisher and use Go kit's load balancing strategies. (Currently, static hosts, DNS SRV, and etcd are supported; Consul support is coming, and Zookeeper support is planned.)

Motivation

Go has emerged as the language of the server, but it remains underrepresented in large, consumer-focused tech companies like Facebook, Twitter, Netflix, and SoundCloud. These organizations have largely adopted JVM-based stacks for their business logic, owing in large part to libraries and ecosystems that directly support their microservice architectures.

To reach its next level of success, Go needs more than simple primitives and idioms. It needs a comprehensive toolkit, for coherent distributed programming in the large. Go kit is a set of packages and best practices, leveraging years of production experience, and providing a comprehensive, robust, and trustable platform for organizations of any size.

In short, Go kit brings Go to the modern enterprise.

For more details, see the motivating blog post and the video of the talk. See also the Go kit talk at GopherCon 2015.

Goals

• Operate in a heterogeneous SOA — expect to interact with mostly non-Go-kit services
• RPC as the primary messaging pattern
• Pluggable serialization and transport — not just JSON over HTTP
• Operate within existing infrastructures — no mandates for specific tools or technologies

Non-goals

• Supporting messaging patterns other than RPC (in the initial release) — MPI, pub/sub, CQRS, etc.
• Re-implementing functionality that can be provided by wrapping existing packages
• Having opinions on deployment, orchestration, process supervision, etc.
• Having opinions on configuration passing — flags, env vars, files, etc.

Contributing

Please see CONTRIBUTING.md. Thank you, contributors!

Dependency management

Go kit is a library, designed to be imported into a binary package. Vendoring is currently the best way for binary package authors to ensure reliable, reproducible builds. Therefore, we strongly recommend our users use vendoring for all of their dependencies, including Go kit. To avoid compatibility and availability issues, Go kit doesn't vendor its own dependencies, and doesn't recommend use of third-party import proxies.

There are several tools which make vendoring easier, including gb, govendor, and Godep. And Go kit uses a variety of continuous integration providers to find and fix compatibility problems as soon as they occur.

API stability policy

The Go kit project depends on code maintained by others. This includes the Go standard library and sub-repositories and other external libraries. The Go language and standard library provide stability guarantees, but the other external libraries typically do not. The API Stability RFC proposes a standard policy for package authors to advertise API stability. The Go kit project prefers to depend on code that abides the API stability policy.

Related projects

Projects with a ★ have had particular influence on Go kit's design (or vice-versa).

Service frameworks

• go-micro, a microservices client/server library ★
• gocircuit, dynamic cloud orchestration
• gotalk, async peer communication protocol & library
• h2, a microservices framework ★
• Kite, a micro-service framework

Individual components

• afex/hystrix-go, client-side latency and fault tolerance library
• armon/go-metrics, library for exporting performance and runtime metrics to external metrics systems
• codahale/lunk, structured logging in the style of Google's Dapper or Twitter's Zipkin
• eapache/go-resiliency, resiliency patterns
• FogCreek/logging, a tagged style of logging
• grpc/grpc-go, HTTP/2 based RPC
• inconshreveable/log15, simple, powerful logging for Go ★
• mailgun/vulcand, programmatic load balancer backed by etcd
• mattheath/phosphor, distributed system tracing
• pivotal-golang/lager, an opinionated logging library
• rubyist/circuitbreaker, circuit breaker library
• Sirupsen/logrus, structured, pluggable logging for Go ★
• sourcegraph/appdash, application tracing system based on Google's Dapper
• spacemonkeygo/monitor, data collection, monitoring, instrumentation, and Zipkin client library
• streadway/handy, net/http handler filters
• vitess/rpcplus, package rpc + context.Context
• gdamore/mangos, nanomsg implementation in pure Go

Web frameworks

• Beego
• Gin
• Goji
• Gorilla
• Martini
• Negroni
• Revel (considered harmful)

Additional reading

• Architecting for the Cloud — Netflix
• Dapper, a Large-Scale Distributed Systems Tracing Infrastructure — Google
• Your Server as a Function (PDF) — Twitter