README
¶
Go kit 
Go kit is a distributed programming toolkit for building microservices in large organizations. We solve common problems in distributed systems, so you can focus on your business logic.
- Mailing list: go-kit
- Slack: gophers.slack.com #go-kit (invite)
Documentation
Examples
Perhaps the best way to understand Go kit is to follow along as we build an example service from first principles. This can serve as a blueprint for your own new service, or demonstrate how to adapt your existing service to use Go kit components.
Endpoint
Go kit primarily deals in the RPC messaging pattern. We use an abstraction called an endpoint to model individual RPCs. An endpoint can be implemented by a server, and called by a client. It's the fundamental building block of many Go kit components.
Circuit breaker
The circuitbreaker package provides endpoint adapters to several popular circuit breaker libraries. Circuit breakers prevent thundering herds, and improve resiliency against intermittent errors. Every client-side endpoint should be wrapped in a circuit breaker.
Rate limiter
The ratelimit package provides endpoint adapters to rate limiting packages. Rate limiters are equally applicable to both server- and client-side endpoints. Use rate limiters to enforce upper thresholds on incoming or outgoing request throughput.
Transport
The transport package provides helpers to bind endpoints to specific serialization mechanisms. At the moment, Go kit just provides helpers for simple JSON over HTTP. If your organization uses a fully-featured transport, bindings are typically provided by the Go library for the transport, and there's not much for Go kit to do. In those cases, see the examples to understand how to write adapters for your endpoints. For now, see the addsvc to understand how transport bindings work. We have specific examples for Thrift, gRPC, net/rpc, and JSON over HTTP. JSON/RPC and Swagger support is planned.
Logging
Services produce logs to be consumed later, either by humans or machines. Humans might be interested in debugging errors, or tracing specific requests. Machines might be interested in counting interesting events, or aggregating information for offline processing. In both cases, it's important that the log messages be structured and actionable. Go kit's log package is designed to encourage both of these best practices.
Metrics (Instrumentation)
Services can't be considered production-ready until they're thoroughly instrumented with metrics that track counts, latency, health, and other periodic or per-request information. Go kit's metrics package provides a robust common set of interfaces for instrumenting your service. Bindings exist for common backends, from expvar to statsd to Prometheus.
Request tracing
As your infrastructure grows, it becomes important to be able to trace a request, as it travels through multiple services and back to the user. Go kit's tracing package provides enhancements for your endpoints and transport bindings to capture information about requests and emit them to request tracing systems. (Currently, Zipkin is supported; Appdash support is planned.)
Service discovery and load balancing
If your service calls another service, it needs to know how to find it, and should intelligently spread its load among those discovered instances. Go kit's loadbalancer package provides client-side endpoint middleware to solve that problem, whether your organization uses static hosts or IPs, DNS SRV records, Consul, etcd, or Zookeeper. And if you use a custom system, it's very easy to write your own Publisher and use Go kit's load balancing strategies. (Currently, static hosts, DNS SRV, etcd, Consul and ZooKeeper are supported)
Motivation
Go has emerged as the language of the server, but it remains underrepresented in large, consumer-focused tech companies like Facebook, Twitter, Netflix, and SoundCloud. These organizations have largely adopted JVM-based stacks for their business logic, owing in large part to libraries and ecosystems that directly support their microservice architectures.
To reach its next level of success, Go needs more than simple primitives and idioms. It needs a comprehensive toolkit, for coherent distributed programming in the large. Go kit is a set of packages and best practices, leveraging years of production experience, and providing a comprehensive, robust, and trustable platform for organizations of any size.
In short, Go kit makes Go a viable choice for business-domain microservices.
For more details, see the motivating blog post and the video of the talk. See also the Go kit talk at GopherCon 2015.
Goals
- Operate in a heterogeneous SOA — expect to interact with mostly non-Go-kit services
- RPC as the primary messaging pattern
- Pluggable serialization and transport — not just JSON over HTTP
- Operate within existing infrastructures — no mandates for specific tools or technologies
Non-goals
- Supporting messaging patterns other than RPC (for now) — e.g. MPI, pub/sub, CQRS, etc.
- Re-implementing functionality that can be provided by adapting existing software
- Having opinions on operational concerns: deployment, configuration, process supervision, orchestration, etc.
Contributing
Please see CONTRIBUTING.md. Thank you, contributors!
Dependency management
Go kit is a library, designed to be imported into a binary package.
Vendoring is currently the best way for binary package authors to ensure reliable, reproducible builds.
Therefore, we strongly recommend our users use vendoring for all of their dependencies, including Go kit.
To avoid compatibility and availability issues, Go kit doesn't vendor its own dependencies, and doesn't recommend use of third-party import proxies.
There are several tools which make vendoring easier, including gb, glide, gvt, govendor, and vendetta. In addition, Go kit uses a variety of continuous integration providers to find and fix compatibility problems as soon as they occur.
Related projects
Projects with a ★ have had particular influence on Go kit's design (or vice-versa).
Service frameworks
- gizmo, a microservice toolkit from The New York Times ★
- go-micro, a microservices client/server library ★
- gocircuit, dynamic cloud orchestration
- gotalk, async peer communication protocol & library
- h2, a microservices framework ★
- Kite, a micro-service framework
Individual components
- afex/hystrix-go, client-side latency and fault tolerance library
- armon/go-metrics, library for exporting performance and runtime metrics to external metrics systems
- codahale/lunk, structured logging in the style of Google's Dapper or Twitter's Zipkin
- eapache/go-resiliency, resiliency patterns
- sasbury/logging, a tagged style of logging
- grpc/grpc-go, HTTP/2 based RPC
- inconshreveable/log15, simple, powerful logging for Go ★
- mailgun/vulcand, programmatic load balancer backed by etcd
- mattheath/phosphor, distributed system tracing
- pivotal-golang/lager, an opinionated logging library
- rubyist/circuitbreaker, circuit breaker library
- Sirupsen/logrus, structured, pluggable logging for Go ★
- sourcegraph/appdash, application tracing system based on Google's Dapper
- spacemonkeygo/monitor, data collection, monitoring, instrumentation, and Zipkin client library
- streadway/handy, net/http handler filters
- vitess/rpcplus, package rpc + context.Context
- gdamore/mangos, nanomsg implementation in pure Go
Web frameworks
Additional reading
- Architecting for the Cloud — Netflix
- Dapper, a Large-Scale Distributed Systems Tracing Infrastructure — Google
- Your Server as a Function (PDF) — Twitter
Directories
¶
| Path | Synopsis |
|---|---|
|
examples
|
|
|
addsvc/pb
Package pb is a generated protocol buffer package.
|
Package pb is a generated protocol buffer package. |
|
shipping/booking
Package booking provides the use-case of booking a cargo.
|
Package booking provides the use-case of booking a cargo. |
|
shipping/cargo
Package cargo contains the heart of the domain model.
|
Package cargo contains the heart of the domain model. |
|
shipping/handling
Package handling provides the use-case for registering incidents.
|
Package handling provides the use-case for registering incidents. |
|
shipping/inspection
Package inspection provides means to inspect cargos.
|
Package inspection provides means to inspect cargos. |
|
shipping/location
Package location provides the Location aggregate.
|
Package location provides the Location aggregate. |
|
shipping/repository
Package repository provides implementations of all the domain repositories.
|
Package repository provides implementations of all the domain repositories. |
|
shipping/routing
Package routing provides the routing domain service.
|
Package routing provides the routing domain service. |
|
shipping/tracking
Package tracking provides the use-case of tracking a cargo.
|
Package tracking provides the use-case of tracking a cargo. |
|
shipping/voyage
Package voyage provides the Voyage aggregate.
|
Package voyage provides the Voyage aggregate. |
|
Package log provides basic interfaces for structured logging.
|
Package log provides basic interfaces for structured logging. |
|
term
Package term provides tools for logging to a terminal.
|
Package term provides tools for logging to a terminal. |
|
Package metrics provides an extensible framework to instrument your application.
|
Package metrics provides an extensible framework to instrument your application. |
|
circonus
Package circonus provides a Circonus backend for package metrics.
|
Package circonus provides a Circonus backend for package metrics. |
|
discard
Package discard implements a backend for package metrics that succeeds without doing anything.
|
Package discard implements a backend for package metrics that succeeds without doing anything. |
|
dogstatsd
Package dogstatsd implements a DogStatsD backend for package metrics.
|
Package dogstatsd implements a DogStatsD backend for package metrics. |
|
expvar
Package expvar implements an expvar backend for package metrics.
|
Package expvar implements an expvar backend for package metrics. |
|
graphite
Package graphite implements a Graphite backend for package metrics.
|
Package graphite implements a Graphite backend for package metrics. |
|
influxdb
Package influxdb implements a InfluxDB backend for package metrics.
|
Package influxdb implements a InfluxDB backend for package metrics. |
|
prometheus
Package prometheus implements a Prometheus backend for package metrics.
|
Package prometheus implements a Prometheus backend for package metrics. |
|
statsd
Package statsd implements a statsd backend for package metrics.
|
Package statsd implements a statsd backend for package metrics. |
|
teststat
Package teststat contains helper functions for statistical testing of metrics implementations.
|
Package teststat contains helper functions for statistical testing of metrics implementations. |
|
tracing
|
|
|
transport
|
|
|
util
|
|
|
conn
Package conn provides utilities related to connections.
|
Package conn provides utilities related to connections. |