twirp

README

---

Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API definition files and lets you focus on your application's logic instead of thinking about folderol like HTTP methods and paths and JSON.

Define your service in a Protobuf file and then Twirp autogenerates Go code with a server interface and fully functional clients. It's similar to gRPC, but without the custom HTTP server and transport implementations: it runs on the standard library's extremely-well-tested-and-high-performance net/http Server. It can run on HTTP 1.1, not just http/2, and supports JSON clients for easy integrations across languages

Twirp handles routing and serialization for you in a well-tested, standardized, thoughtful way so you don't have to. Serialization and deserialization code is error-prone and tricky, and you shouldn't be wasting your time deciding whether it should be "POST /friends/:id/new" or "POST /:id/friend" or whatever. Just get to the real work of building services!

Along the way, you get an autogenerated client and a simple, smart framework for passing error messages. Nice!

For more on the motivation behind Twirp (and a comparison to REST APIs and gRPC), the announcement blog post is a good read.

Documentation

Documentation: https://twitchtv.github.io/twirp/docs/intro.html

Source for this documentation is in the docs subdirectory. The website is generated from that folder using Docusaurus.

Implementations in other languages

This repo only has the official generators, which write out Go and Python (partial, see #185 and #220) code.

For other languages, there are third-party generators available:

Language	Clients	Servers	Repository
Python3	✓	✓	github.com/verloop/twirpy
Java	✓	✓	github.com/fajran/protoc-gen-twirp_java_jaxrs
Java		✓	https://github.com/devork/flit
JavaScript	✓		github.com/thechriswalker/protoc-gen-twirp_js
JavaScript	✓		github.com/Xe/twirp-codegens/cmd/protoc-gen-twirp_jsbrowser
Typescript	✓		github.com/larrymyers/protoc-gen-twirp_typescript
Ruby	✓	✓	github.com/twitchtv/twirp-ruby
Rust	✓	✓	github.com/cretz/prost-twirp
Scala	✓	✓	github.com/soundcloud/twinagle
Swagger	✓	✓	github.com/elliots/protoc-gen-twirp_swagger
PHP	✓	✓	github.com/twirphp/twirp
Dart	✓		github.com/apptreesoftware/protoc-gen-twirp_dart
Elixir	✓	✓	github.com/keathley/twirp-elixir

This list isn't an endorsement, it's just a convenience to help you find stuff for your language.

Support and Community

We have a channel on the Gophers slack, #twirp, which is the best place to get quick answers to your questions. You can join the Gopher slack here.

Releases

Twirp follows semantic versioning through git tags, and uses Github Releases for release notes and upgrade guides: Twirp Releases

Contributing

Check out CONTRIBUTING.md for notes on making contributions.

License

This library is licensed under the Apache 2.0 License.

Documentation

Overview

Copyright 2018 Twitch Interactive, Inc. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

http://www.apache.org/licenses/LICENSE-2.0

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Package twirp provides core types used in generated Twirp servers and client.

Twirp services handle errors using the `twirp.Error` interface.

For example, a server method may return an InvalidArgumentError:

if req.Order != "DESC" && req.Order != "ASC" {
    return nil, twirp.InvalidArgumentError("Order", "must be DESC or ASC")
}

And the same twirp.Error is returned by the client, for example:

resp, err := twirpClient.RPCMethod(ctx, req)
if err != nil {
    if twerr, ok := err.(twirp.Error); ok {
        switch twerr.Code() {
        case twirp.InvalidArgument:
            log.Error("invalid argument "+twirp.Meta("argument"))
        default:
            log.Error(twerr.Error())
        }
    }
}

Clients may also return Internal errors if something failed on the system: the server, the network, or the client itself (i.e. failure parsing response).

Copyright 2018 Twitch Interactive, Inc. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

http://www.apache.org/licenses/LICENSE-2.0

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2018 Twitch Interactive, Inc. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

http://www.apache.org/licenses/LICENSE-2.0

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• Constants
• func AddHTTPResponseHeader(ctx context.Context, key, value string) error
• func HTTPRequestHeaders(ctx context.Context) (http.Header, bool)
• func IsValidErrorCode(code ErrorCode) bool
• func MethodName(ctx context.Context) (string, bool)
• func PackageName(ctx context.Context) (string, bool)
• func ServerHTTPStatusFromErrorCode(code ErrorCode) int
• func ServiceName(ctx context.Context) (string, bool)
• func SetHTTPResponseHeader(ctx context.Context, key, value string) error
• func StatusCode(ctx context.Context) (string, bool)
• func WithHTTPRequestHeaders(ctx context.Context, h http.Header) (context.Context, error)
• func WriteError(resp http.ResponseWriter, err error) error
• type ClientHooks
  • func ChainClientHooks(hooks ...*ClientHooks) *ClientHooks
• type ClientOption
  • func WithClientHooks(hooks *ClientHooks) ClientOption
  • func WithClientInterceptors(interceptors ...Interceptor) ClientOption
  • func WithClientLiteralURLs(b bool) ClientOption
  • func WithClientPathPrefix(prefix string) ClientOption
• type ClientOptions
  • func (opts *ClientOptions) PathPrefix() string
• type Error
  • func InternalError(msg string) Error
  • func InternalErrorWith(err error) Error
  • func InvalidArgumentError(argument string, validationMsg string) Error
  • func NewError(code ErrorCode, msg string) Error
  • func NotFoundError(msg string) Error
  • func RequiredArgumentError(argument string) Error
  • func WrapError(twerr Error, err error) Error
• type ErrorCode
• type Interceptor
  • func ChainInterceptors(interceptors ...Interceptor) Interceptor
• type Method
• type ServerHooks
  • func ChainHooks(hooks ...*ServerHooks) *ServerHooks
• type ServerOption
  • func WithServerHooks(hooks *ServerHooks) ServerOption
  • func WithServerInterceptors(interceptors ...Interceptor) ServerOption
  • func WithServerJSONSkipDefaults(skipDefaults bool) ServerOption
  • func WithServerPathPrefix(prefix string) ServerOption
• type ServerOptions
  • func (opts *ServerOptions) PathPrefix() string

Constants

const TwirpPackageIsVersion7 = true

TwirpPackageIsVersion7 is a constant referenced from generated code that requires features on this version of the twirp package. Older versions don't have this constant defined, causing a compile-time assetion error.

Functions

func AddHTTPResponseHeader

func AddHTTPResponseHeader(ctx context.Context, key, value string) error

AddHTTPResponseHeader behaves like SetHTTPResponseHeader, but it appends the key-value pair to the header (instead of replacing it).

AddHTTPResponseHeader returns an error if the key is "Content-Type".

func HTTPRequestHeaders

func HTTPRequestHeaders(ctx context.Context) (http.Header, bool)

func IsValidErrorCode

func IsValidErrorCode(code ErrorCode) bool

IsValidErrorCode returns true if is one of the valid predefined constants.

func MethodName

func MethodName(ctx context.Context) (string, bool)

MethodName extracts the name of the method being handled in the given context. If it is not known, it returns ("", false).

func PackageName

func PackageName(ctx context.Context) (string, bool)

PackageName extracts the fully-qualified protobuf package name of the service handling the given context. If it is not known, it returns ("", false). If the service comes from a proto file that does not declare a package name, it returns ("", true).

Note that the protobuf package name can be very different than the go package name; the two are unrelated.

func ServerHTTPStatusFromErrorCode

func ServerHTTPStatusFromErrorCode(code ErrorCode) int

ServerHTTPStatusFromErrorCode maps a Twirp error type into a similar HTTP response status. It is used by the Twirp server handler to set the HTTP response status code. Returns 0 if the ErrorCode is invalid.

func ServiceName

func ServiceName(ctx context.Context) (string, bool)

ServiceName extracts the name of the service handling the given context. If it is not known, it returns ("", false).

func SetHTTPResponseHeader

func SetHTTPResponseHeader(ctx context.Context, key, value string) error

SetHTTPResponseHeader sets an HTTP header key-value pair using a context provided by a twirp-generated server, or a child of that context. The server will include the header in its response for that request context.

This can be used to respond with custom HTTP headers like "Cache-Control". But note that HTTP headers are a Twirp implementation detail, only visible by middleware, not by the clients or their responses.

The header will be ignored (noop) if the context is invalid (i.e. using a new context.Background() instead of passing the context from the handler).

If called multiple times with the same key, it replaces any existing values associated with that key.

SetHTTPResponseHeader returns an error if the provided header key would overwrite a header that is needed by Twirp, like "Content-Type".

func StatusCode

func StatusCode(ctx context.Context) (string, bool)

StatusCode retrieves the status code of the response (as string like "200"). If it is known returns (status, true). If it is not known, it returns ("", false).

func WithHTTPRequestHeaders

func WithHTTPRequestHeaders(ctx context.Context, h http.Header) (context.Context, error)

WithHTTPRequestHeaders stores an http.Header in a context.Context. When using a Twirp-generated client, you can pass the returned context into any of the request methods, and the stored header will be included in outbound HTTP requests.

This can be used to set custom HTTP headers like authorization tokens or client IDs. But note that HTTP headers are a Twirp implementation detail, only visible by middleware, not by the server implementation.

WithHTTPRequestHeaders returns an error if the provided http.Header would overwrite a header that is needed by Twirp, like "Content-Type".

func WriteError

func WriteError(resp http.ResponseWriter, err error) error

WriteError writes an HTTP response with a valid Twirp error format (code, msg, meta). Useful outside of the Twirp server (e.g. http middleware). If err is not a twirp.Error, it will get wrapped with twirp.InternalErrorWith(err)

Types

type ClientHooks

type ClientHooks struct {
	// RequestPrepared is called as soon as a request has been created and before
	// it has been sent to the Twirp server.
	RequestPrepared func(context.Context, *http.Request) (context.Context, error)

	// ResponseReceived is called after a request has finished sending. Since this
	// is terminal, the context is not returned. ResponseReceived will not be
	// called in the case of an error being returned from the request.
	ResponseReceived func(context.Context)

	// Error hook is called whenever an error occurs during the sending of a
	// request. The Error is passed as an argument to the hook.
	Error func(context.Context, Error)
}

ClientHooks is a container for callbacks that can instrument a Twirp-generated client. These callbacks all accept a context and some return a context. They can use this to add to the context, appending values or deadlines to it.

The RequestPrepared hook is special because it can return errors. If it returns non-nil error, handling for that request will be stopped at that point. The Error hook will then be triggered.

The RequestPrepared hook will always be called first and will be called for each outgoing request from the Twirp client. The last hook to be called will either be Error or ResponseReceived, so be sure to handle both cases in your hooks.

func ChainClientHooks

func ChainClientHooks(hooks ...*ClientHooks) *ClientHooks

ChainClientHooks creates a new *ClientHooks which chains the callbacks in each of the constituent hooks passed in. Each hook function will be called in the order of the ClientHooks values passed in.

For the erroring hook, RequestPrepared, any returned errors prevent processing by later hooks.

type ClientOption

type ClientOption func(*ClientOptions)

ClientOption is a functional option for extending a Twirp client.

func WithClientHooks

func WithClientHooks(hooks *ClientHooks) ClientOption

WithClientHooks defines the hooks for a Twirp client.

func WithClientInterceptors

func WithClientInterceptors(interceptors ...Interceptor) ClientOption

WithClientInterceptors defines the interceptors for a Twirp client.

func WithClientLiteralURLs

func WithClientLiteralURLs(b bool) ClientOption

WithClientLiteralURLs configures the Twirp client to use the exact values as defined in the proto file for Service and Method names, fixing the issue https://github.com/twitchtv/twirp/issues/244, which is manifested when working with Twirp services implemented other languages (e.g. Python) and the proto file definitions are not properly following the [Protobuf Style Guide](https://developers.google.com/protocol-buffers/docs/style#services). By default (false), Go clients modify the routes by CamelCasing the values. For example, with Service: `haberdasher`, Method: `make_hat`, the URLs generated by Go clients are `Haberdasher/MakeHat`, but with this option enabled (true) the client will properly use `haberdasher/make_hat` instead.

func WithClientPathPrefix

func WithClientPathPrefix(prefix string) ClientOption

WithClientPathPrefix specifies a different prefix to use for routing. If not specified, the "/twirp" prefix is used by default. The service must be configured to serve on the same prefix. An empty value "" can be speficied to use no prefix. URL format: "<baseURL>[<prefix>]/<package>.<Service>/<Method>" More info on Twirp docs: https://twitchtv.github.io/twirp/docs/routing.html

type ClientOptions

type ClientOptions struct {
	Interceptors []Interceptor
	Hooks        *ClientHooks

	LiteralURLs bool
	// contains filtered or unexported fields
}

ClientOptions encapsulate the configurable parameters on a Twirp client.

func (*ClientOptions) PathPrefix

func (opts *ClientOptions) PathPrefix() string

type Error

type Error interface {
	// Code is of the valid error codes.
	Code() ErrorCode

	// Msg returns a human-readable, unstructured messages describing the error.
	Msg() string

	// WithMeta returns a copy of the Error with the given key-value pair attached
	// as metadata. If the key is already set, it is overwritten.
	WithMeta(key string, val string) Error

	// Meta returns the stored value for the given key. If the key has no set
	// value, Meta returns an empty string. There is no way to distinguish between
	// an unset value and an explicit empty string.
	Meta(key string) string

	// MetaMap returns the complete key-value metadata map stored on the error.
	MetaMap() map[string]string

	// Error returns a string of the form "twirp error <Type>: <Msg>"
	Error() string
}

Error represents an error in a Twirp service call.

func InternalError

func InternalError(msg string) Error

InternalError constructor for the common Internal error. Should be used to specify that something bad or unexpected happened.

func InternalErrorWith

func InternalErrorWith(err error) Error

InternalErrorWith makes an internal error, wrapping the original error and using it for the error message, and with metadata "cause" with the original error type. This function is used by Twirp services to wrap non-Twirp errors as internal errors. The wrapped error can be extracted later with (github.com/pkg/errors).Unwrap or errors.Is from the standard errors package on Go 1.13+.

func InvalidArgumentError

func InvalidArgumentError(argument string, validationMsg string) Error

InvalidArgumentError constructor for the common InvalidArgument error. Can be used when an argument has invalid format, is a number out of range, is a bad option, etc).

func NewError

func NewError(code ErrorCode, msg string) Error

NewError is the generic constructor for a twirp.Error. The ErrorCode must be one of the valid predefined constants, otherwise it will be converted to an error {type: Internal, msg: "invalid error type {{code}}"}. If you need to add metadata, use .WithMeta(key, value) method after building the error.

func NotFoundError

func NotFoundError(msg string) Error

NotFoundError constructor for the common NotFound error.

func RequiredArgumentError

func RequiredArgumentError(argument string) Error

RequiredArgumentError is a more specific constructor for InvalidArgument error. Should be used when the argument is required (expected to have a non-zero value).

func WrapError

func WrapError(twerr Error, err error) Error

WrapError allows Twirp errors to wrap other errors. The wrapped error can be extracted later with (github.com/pkg/errors).Unwrap or errors.Is from the standard errors package on Go 1.13+.

type ErrorCode

type ErrorCode string

ErrorCode represents a Twirp error type.

const (
	// Canceled indicates the operation was cancelled (typically by the caller).
	Canceled ErrorCode = "canceled"

	// Unknown error. For example when handling errors raised by APIs that do not
	// return enough error information.
	Unknown ErrorCode = "unknown"

	// InvalidArgument indicates client specified an invalid argument. It
	// indicates arguments that are problematic regardless of the state of the
	// system (i.e. a malformed file name, required argument, number out of range,
	// etc.).
	InvalidArgument ErrorCode = "invalid_argument"

	// Malformed indicates an error occurred while decoding the client's request.
	// This may mean that the message was encoded improperly, or that there is a
	// disagreement in message format between the client and server.
	Malformed ErrorCode = "malformed"

	// DeadlineExceeded means operation expired before completion. For operations
	// that change the state of the system, this error may be returned even if the
	// operation has completed successfully (timeout).
	DeadlineExceeded ErrorCode = "deadline_exceeded"

	// NotFound means some requested entity was not found.
	NotFound ErrorCode = "not_found"

	// BadRoute means that the requested URL path wasn't routable to a Twirp
	// service and method. This is returned by the generated server, and usually
	// shouldn't be returned by applications. Instead, applications should use
	// NotFound or Unimplemented.
	BadRoute ErrorCode = "bad_route"

	// AlreadyExists means an attempt to create an entity failed because one
	// already exists.
	AlreadyExists ErrorCode = "already_exists"

	// PermissionDenied indicates the caller does not have permission to execute
	// the specified operation. It must not be used if the caller cannot be
	// identified (Unauthenticated).
	PermissionDenied ErrorCode = "permission_denied"

	// Unauthenticated indicates the request does not have valid authentication
	// credentials for the operation.
	Unauthenticated ErrorCode = "unauthenticated"

	// ResourceExhausted indicates some resource has been exhausted or rate-limited,
	// perhaps a per-user quota, or perhaps the entire file system is out of space.
	ResourceExhausted ErrorCode = "resource_exhausted"

	// FailedPrecondition indicates operation was rejected because the system is
	// not in a state required for the operation's execution. For example, doing
	// an rmdir operation on a directory that is non-empty, or on a non-directory
	// object, or when having conflicting read-modify-write on the same resource.
	FailedPrecondition ErrorCode = "failed_precondition"

	// Aborted indicates the operation was aborted, typically due to a concurrency
	// issue like sequencer check failures, transaction aborts, etc.
	Aborted ErrorCode = "aborted"

	// OutOfRange means operation was attempted past the valid range. For example,
	// seeking or reading past end of a paginated collection.
	//
	// Unlike InvalidArgument, this error indicates a problem that may be fixed if
	// the system state changes (i.e. adding more items to the collection).
	//
	// There is a fair bit of overlap between FailedPrecondition and OutOfRange.
	// We recommend using OutOfRange (the more specific error) when it applies so
	// that callers who are iterating through a space can easily look for an
	// OutOfRange error to detect when they are done.
	OutOfRange ErrorCode = "out_of_range"

	// Unimplemented indicates operation is not implemented or not
	// supported/enabled in this service.
	Unimplemented ErrorCode = "unimplemented"

	// Internal errors. When some invariants expected by the underlying system
	// have been broken. In other words, something bad happened in the library or
	// backend service. Do not confuse with HTTP Internal Server Error; an
	// Internal error could also happen on the client code, i.e. when parsing a
	// server response.
	Internal ErrorCode = "internal"

	// Unavailable indicates the service is currently unavailable. This is a most
	// likely a transient condition and may be corrected by retrying with a
	// backoff.
	Unavailable ErrorCode = "unavailable"

	// DataLoss indicates unrecoverable data loss or corruption.
	DataLoss ErrorCode = "data_loss"

	// NoError is the zero-value, is considered an empty error and should not be
	// used.
	NoError ErrorCode = ""
)

Valid Twirp error types. Most error types are equivalent to gRPC status codes and follow the same semantics.

type Interceptor

type Interceptor func(Method) Method

Interceptor is a form of middleware for Twirp requests, that can be installed on both clients and servers. To intercept RPC calls in the client, use the option `twirp.WithClientInterceptors` on the client constructor. To intercept RPC calls in the server, use the option `twirp.WithServerInterceptors` on the server constructor.

Just like http middleware, interceptors can mutate requests and responses. This can enable some powerful integrations, but it should be used with much care because it may result in code that is very hard to debug.

Example of an interceptor that logs every request and response:

func LogInterceptor(l *log.Logger) twirp.Interceptor {
  return func(next twirp.Method) twirp.Method {
    return func(ctx context.Context, req interface{}) (interface{}, error) {
      l.Printf("Service: %s, Method: %s, Request: %v",
          twirp.ServiceName(ctx), twirp.MethodName(ctx), req)
      resp, err := next(ctx, req)
      l.Printf("Response: %v, Error: %v", resp)
      return resp, err
    }
  }
}

func ChainInterceptors

func ChainInterceptors(interceptors ...Interceptor) Interceptor

ChainInterceptors chains multiple Interceptors into a single Interceptor. The first interceptor wraps the second one, and so on. Returns nil if interceptors is empty. Nil interceptors are ignored.

type Method

type Method func(ctx context.Context, request interface{}) (interface{}, error)

Method is a generic representation of a Twirp-generated RPC method. It is used to define Interceptors.

type ServerHooks

type ServerHooks struct {
	// RequestReceived is called as soon as a request enters the Twirp
	// server at the earliest available moment.
	RequestReceived func(context.Context) (context.Context, error)

	// RequestRouted is called when a request has been routed to a
	// particular method of the Twirp server.
	RequestRouted func(context.Context) (context.Context, error)

	// ResponsePrepared is called when a request has been handled and a
	// response is ready to be sent to the client.
	ResponsePrepared func(context.Context) context.Context

	// ResponseSent is called when all bytes of a response (including an error
	// response) have been written. Because the ResponseSent hook is terminal, it
	// does not return a context.
	ResponseSent func(context.Context)

	// Error hook is called when an error occurs while handling a request. The
	// Error is passed as argument to the hook.
	Error func(context.Context, Error) context.Context
}

ServerHooks is a container for callbacks that can instrument a Twirp-generated server. These callbacks all accept a context and return a context. They can use this to add to the request context as it threads through the system, appending values or deadlines to it.

The RequestReceived and RequestRouted hooks are special: they can return errors. If they return a non-nil error, handling for that request will be stopped at that point. The Error hook will be triggered, and the error will be sent to the client. This can be used for stuff like auth checks before deserializing a request.

The RequestReceived hook is always called first, and it is called for every request that the Twirp server handles. The last hook to be called in a request's lifecycle is always ResponseSent, even in the case of an error.

Details on the timing of each hook are documented as comments on the fields of the ServerHooks type.

func ChainHooks

func ChainHooks(hooks ...*ServerHooks) *ServerHooks

ChainHooks creates a new *ServerHooks which chains the callbacks in each of the constituent hooks passed in. Each hook function will be called in the order of the ServerHooks values passed in.

For the erroring hooks, RequestReceived and RequestRouted, any returned errors prevent processing by later hooks.

type ServerOption

type ServerOption func(*ServerOptions)

ServerOption is a functional option for extending a Twirp client.

func WithServerHooks

func WithServerHooks(hooks *ServerHooks) ServerOption

WithServerHooks defines the hooks for a Twirp server.

func WithServerInterceptors

func WithServerInterceptors(interceptors ...Interceptor) ServerOption

WithServerInterceptors defines the interceptors for a Twirp server.

func WithServerJSONSkipDefaults

func WithServerJSONSkipDefaults(skipDefaults bool) ServerOption

WithServerJSONSkipDefaults configures JSON serialization to skip unpopulated or default values in JSON responses, which results in smaller response sizes. This was the default before v7 and can be enabled for full backwards compatibility if required. This is now disabled by default, because JSON serialization is commonly used for manual debugging, in which case it is useful to see the full shape of the response.

func WithServerPathPrefix

func WithServerPathPrefix(prefix string) ServerOption

WithServerPathPrefix specifies a different prefix for routing. If not specified, the "/twirp" prefix is used by default. An empty value "" can be speficied to use no prefix. The clients must be configured to send requests using the same prefix. URL format: "<baseURL>[<prefix>]/<package>.<Service>/<Method>" More info on Twirp docs: https://twitchtv.github.io/twirp/docs/routing.html

type ServerOptions

type ServerOptions struct {
	Interceptors []Interceptor
	Hooks        *ServerHooks

	JSONSkipDefaults bool
	// contains filtered or unexported fields
}

ServerOptions encapsulate the configurable parameters on a Twirp client.

func (*ServerOptions) PathPrefix

func (opts *ServerOptions) PathPrefix() string