proto/

README

proto/

This directory contains the protocol buffer definitions for all Rill components. Instead of placing .proto files in their respective sub-projects, we follow the convention of having a single proto folder to make cross-component imports and codegen easier.

We use Buf to lint and generate protocol buffers. The layout and style of our .proto files follow their style guide.

Defining APIs

We define APIs as gRPC services. All APIs should be defined centrally in this directory, but implemented in their respective sub-package of the monorepo.

For all APIs, we setup gRPC-Gateway to map the gRPC definitions to a RESTful API. The mapping rules are done inline in the proto file, using google.api.http annotations (docs here).

Using protocol buffers to define dual RPC and REST interfaces is a technique widely used at Google. We suggest taking a look at their excellent API design guide, which describes this pattern (notice it's multiple pages).

Generating

After changing a .proto file, you should re-generate the bindings:

1. Install Buf if you haven't already (docs)
2. From the repo root, run:

make proto.generate

Typescript runtime client

We separately have a generated TypeScript client for the runtime in web-common/src/runtime-client. If relevant, you can re-generate it by running:

npm run generate:runtime-client -w web-common

(This is not automated as the frontend may currently be pinned to an older version of the runtime.)

Understanding the buf.gen.yaml files

We use the gRPC gateway OpenAPI plugin to generate OpenAPI specs for the admin and runtime services (which the frontend consumes to generate svelte-query clients).

To get clean OpenAPI specs, we use the allow_merge=true plugin option. Unfortunately, this means we need to separately generate the admin and runtime OpenAPI specs – hence the buf.gen.openapi-SERVICE.yaml files (using multiple buf.gen.yaml files is recommended by Buf when you need more granular builds). The Makefile invokes them using:

cd proto && buf generate --template buf.gen.openapi-admin.yaml --path rill/admin
cd proto && buf generate --template buf.gen.openapi-runtime.yaml --path rill/runtime