protoschema-plugins

README

protoschema-plugins

The protoschema-plugins repository contains a collection of Protobuf plugins that generate different types of schema from protobuf files. This includes:

• PubSub
• BigQuery
• JSON Schema

PubSub Protobuf Schema

Generates a schema for a given protobuf file that can be used as a PubSub schema in the form of a single self-contained messaged normalized to proto2.

Install the protoc-gen-pubsub plugin directly:

go install github.com/bufbuild/protoschema-plugins/cmd/protoc-gen-pubsub@latest

Or reference it as a Remote Plugin in buf.gen.yaml:

version: v1
plugins:
  - plugin: buf.build/bufbuild/protoschema-pubsub
    out: ./gen

For examples see testdata which contains the generated schema for test case definitions found in proto.

BigQuery Protobuf Schema

Generates a Table schema that can be used to initialize a BigQuery table, and a self-contained message normalized to proto2, that can be used to upload data to that BigQuery table.

Install the protoc-gen-bigquery directly:

go install github.com/bufbuild/protoschema-plugins/cmd/protoc-gen-bigquery@latest

Or reference it as a Remote Plugin in buf.gen.yaml:

version: v1
plugins:
  - plugin: buf.build/bufbuild/protoschema-bigquery
    out: ./gen

For examples see testdata which contains the generated schema for test case definitions found in proto.

This supports the gen_bq_schema annotations. For example:

message CustomBigQuery {
  option (gen_bq_schema.bigquery_opts) = {
    table_name: "CustomName"
    use_json_names: true
  };

  int32 int32_field = 1 [(gen_bq_schema.bigquery) = {
    type_override: "TIMESTAMP",
    name: "create_time"
  }];
  string string_field = 2;

  NestedReference nested_reference = 3 [(gen_bq_schema.bigquery) = {ignore: true}];
}

Results in the Table schema:

[
  {
    "name": "create_time",
    "type": "TIMESTAMP"
  },
  {
    "name": "stringField",
    "type": "STRING"
  }
]

and the message:

message CustomBigQuery {
  optional int32 create_time = 1;
  optional string stringField = 2;
}

Limitations

The well-known JSON types are not support in BigQuery. This includes google.protobuf.Struct, google.protobuf.Value and google.protobuf.ListValue. If these types are used in a message, the generated schema will not include them, and the generated message will not include them.

JSON Schema

Generates a JSON Schema for a given protobuf file. This implementation uses the latest JSON Schema Draft 2020-12.

Install the protoc-gen-jsonschema directly:

go install github.com/bufbuild/protoschema-plugins/cmd/protoc-gen-jsonschema@latest

Or reference it as a Remote Plugin in buf.gen.yaml:

version: v1
plugins:
  - plugin: buf.build/bufbuild/protoschema-jsonschema
    out: ./gen

For examples see testdata which contains the generated schema for test case definitions found in proto.

Here is a simple generated schema from the following protobuf:

message Product {
  message Location {
    float lat = 1;
    float long = 2;
  }

  int32 product_id = 1;
  string product_name = 2;
  float price = 3;
  repeated string tags = 4;
  Location location = 5;
}

Results in the following JSON Schema files:

Product.schema.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "product_id": {
      "type": "integer"
    },
    "product_name": {
      "type": "string"
    },
    "price": {
      "type": "number"
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "location": {
      "type": "object",
      "properties": {
        "lat": {
          "type": "number"
        },
        "long": {
          "type": "number"
        }
      },
      "required": ["lat", "long"]
    }
  },
  "required": ["product_id", "product_name", "price", "tags", "location"]
}

Product.Location.schema.json

{
  "$id": "Product.Location.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": false,
  "properties": {
    "lat": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "string"
        },
        {
          "enum": ["NaN", "Infinity", "-Infinity"],
          "type": "string"
        }
      ]
    },
    "long": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "string"
        },
        {
          "enum": ["NaN", "Infinity", "-Infinity"],
          "type": "string"
        }
      ]
    }
  },
  "type": "object"
}

Community

For help and discussion around Protobuf, best practices, and more, join us on Slack.

Status

This project is currently in alpha. The API should be considered unstable and likely to change.

Legal

Offered under the Apache 2 license.