entproto

README

entproto

entproto is a library and cli tool to facilitate the generation of .proto files from an ent.Schema.

Disclaimer: This is an experimental feature, expect the API to change in the near future.

Quick Start

Prerequesites:

• Install protoc: https://grpc.io/docs/protoc-installation/
• Install protoc-gen-go and protoc-gen-go-grpc: https://grpc.io/docs/languages/go/quickstart/

Download the module:

go get -u github.com/linuxluigi/ent-contrib/entproto

Install protoc-gen-entgrpc (ent's gRPC service implementation generator):

go get github.com/linuxluigi/ent-contrib/entproto/cmd/protoc-gen-entgrpc

Annotate the schema with entproto.Message() and all fields with the desired proto field numbers (notice the field number 1 is reserved for the schema's ID field:

package schema

import (
	"entgo.io/ent"
	"entgo.io/ent/schema"
	"entgo.io/ent/schema/field"
	"github.com/linuxluigi/ent-contrib/entproto"
)

type User struct {
	ent.Schema
}

func (User) Annotations() []schema.Annotation {
	return []schema.Annotation{
		entproto.Message(),
		entproto.Service(), // also generate a gRPC service definition
	}
}

func (User) Fields() []ent.Field {
	return []ent.Field{
		field.String("user_name").
			Annotations(entproto.Field(2)),
	}
}

Run the code generation:

go run github.com/linuxluigi/ent-contrib/entproto/cmd/entproto -path ./ent/schema

The proto file is generated under ./ent/proto/entpb/entpb.proto:

// Code generated by entproto. DO NOT EDIT.
syntax = "proto3";

package entpb;

option go_package = "github.com/linuxluigi/ent-contrib/entproto/internal/todo/ent/proto/entpb";

message User {
  int32 id = 1;

  string user_name = 2;
}

In addition, a file named generate.go, which contains a //go:generate directive to invoke protoc and create Go files for the protocol buffers and gRPC services is created adjecent to the .proto file. If a file by that name already exists, this step is skipped. The protoc invocation includes requests for codegen from 3 plugins: protoc-gen-go (standard Go codegen), protoc-gen-go-grpc (standard gRPC codegen) and protoc-gen-entgrpc (an ent-specific protoc plugin that generates service implementations using ent).

To generate the Go files from the .proto file run:

go generate ./ent/proto/...

protoc-gen-entgrpc

protoc-gen-entgrpc is a protoc plugin that generates server code that implements the gRPC interface that was generated from the ent schema. It must receive a path to the ent schema directory which is used to map the schema definitions with the proto definitions to produce correct code:

protoc -I=.. --go_out=.. --go-grpc_out=.. --go_opt=paths=source_relative --entgrpc_out=.. --entgrpc_opt=paths=source_relative,schema_path=../../schema --go-grpc_opt=paths=source_relative entpb/entpb.proto

As mentioned in the section above, this command will be generated for you for each protobuf package directory when you run the entproto command.

The current version generates a full service implementation, an example can be found in entpb/entpb_user_service.go.

Some caveats with the current version:

• Currently only "unique" edges are supported (O2O, O2M). Support for multi-relations will land soon.
• The generated "mutating" methods (Create/Update) currently set all fields, disregarding zero/null values and field nullability.
• All fields are copied from the gRPC request to the ent client, support for making some fields not settable via the service by adding a field/edge annotation is also planned.

// UserService implements UserServiceServer
type UserService struct {
	client *ent.Client
	UnimplementedUserServiceServer
}

func NewUserService(client *ent.Client) *UserService {
	return &UserService{client: client}
}

// Create implements UserServiceServer.Create
func (svc *UserService) Create(ctx context.Context, req *CreateUserRequest) (*User, error) {
	return nil, status.Error(codes.Unimplemented, "error")
}
/// ... and so on

Programmatic code-generation

To programmatically invoke entproto from a custom entc.Generate call, entproto can be used as a gen.Hook. For example:

package main

import (
	"log"

	"github.com/linuxluigi/ent-contrib/entproto"
	"entgo.io/ent/entc"
	"entgo.io/ent/entc/gen"
)

func main() {
	err := entc.Generate("./ent/schema", &gen.Config{
		Hooks: []gen.Hook{
			// Run entproto codegen in addition to normal ent codegen.
			entproto.Hook(),
		},
	})
	if err != nil {
		log.Fatal(err)
	}
}

Message Annotations

ent.Message

By default, entproto will skip all schemas, unless they explicitly opt-in for proto file generation:

type User struct {
	ent.Schema
}

func (User) Annotations() []schema.Annotation {
	return []schema.Annotation{entproto.Message()}
}

By default the proto package name for the generated files will be entpb but it can be specified using a functional option:

func (MessageWithPackageName) Annotations() []schema.Annotation {
	return []schema.Annotation{entproto.Message(
		entproto.PackageName("io.entgo.apps.todo"),
	)}
}

Per the protobuf style guide:

Package name should be in lowercase, and should correspond to the directory hierarchy. e.g., if a file is in my/package/, then the package name should be my.package.

Therefore, protos for a package named io.entgo.apps.todo will be placed under io/entgo/apps/todo. To avoid issues with cyclic dependencies, all messages for a given package are placed in a single file with the name of the last part of the module. In the example above, the generated file name will be todo.proto.

entproto.SkipGen()

To explicitly opt-out of proto file generation, the functional option entproto.SkipGen() can be used:

func (ExplicitSkippedMessage) Annotations() []schema.Annotation {
	return []schema.Annotation{
        entproto.SkipGen(),
    }
}

This is useful in cases where a Mixin is used and its default behavior enables proto generation.

entproto.Service()

entproto supports the generation of simple CRUD gRPC service definitions from ent.Schema

To enable generation of a service definition, add an entproto.Service() annotation:

func (User) Annotations() []schema.Annotation {
	return []schema.Annotation{
		entproto.Message(),
		entproto.Service(),
	}
}

This will generate:

message CreateUserRequest {
  User user = 1;
}

message GetUserRequest {
  int32 id = 1;
}

message UpdateUserRequest {
  User user = 1;
}

message DeleteUserRequest {
  int32 id = 1;
}

service UserService {
  rpc Create ( CreateUserRequest ) returns ( User );

  rpc Get ( GetUserRequest ) returns ( User );

  rpc Update ( UpdateUserRequest ) returns ( User );

  rpc Delete ( DeleteUserRequest ) returns ( google.protobuf.Empty );
}

Method generation can be customized by including the argument entproto.Methods() in the entproto.Service() annotation. entproto.Methods() accepts bit flags to determine what service methods should be generated.

// Generates a Create gRPC service method for the entproto.Service.
entproto.MethodCreate

// Generates a Get gRPC service method for the entproto.Service.
entproto.MethodGet

// Generates an Update gRPC service method for the entproto.Service.
entproto.MethodUpdate

// Generates a Delete gRPC service method for the entproto.Service.
entproto.MethodDelete

// Generates all service methods for the entproto.Service.
// This is the same behavior as not including entproto.Methods.
entproto.MethodAll

To generate a service with multiple methods, bitwise OR the flags.

For example, the ent.Schema can be modified to generate only Create and Get methods:

func (User) Annotations() []schema.Annotation {
	return []schema.Annotation{
		entproto.Message(),
		entproto.Service(
			entproto.Methods(entproto.MethodCreate | entproto.MethodGet),
        ),
	}
}

This will generate:

message CreateUserRequest {
  User user = 1;
}

message GetUserRequest {
  int32 id = 1;
}

service UserService {
  rpc Create ( CreateUserRequest ) returns ( User );

  rpc Get ( GetUserRequest ) returns ( User );
}

Field Annotations

entproto.Field

All fields must be annotated with entproto.Field to specify their proto field numbers

// Fields of the User.
func (User) Fields() []ent.Field {
	return []ent.Field{
		field.String("name").
			Annotations(entproto.Field(2)),
	}
}

The ID field is added to the generated message as well, in the above example it is implicitly defined, but entproto will respect explicitly defined ID fields as well. The above schema would translate to:

message User {
  int32 id = 1;
  string name = 2
}

Field type mappings:

Ent Type	Proto Type	More considerations
TypeBool	bool
TypeTime	google.protobuf.Timestamp
TypeJSON	X
TypeUUID	bytes	When receiving an arbitrary byte slice as input, 16-byte length must be validated
TypeBytes	bytes
TypeEnum	Enum	Proto enums like proto fields require stable numbers to be assigned to each value. Therefore we will need to add an extra annotation to map from field value to tag number.
TypeString	string
TypeOther	X
TypeInt8	int32
TypeInt16	int32
TypeInt32	int32
TypeInt	int32
TypeInt64	int64
TypeUint8	uint32
TypeUint16	uint32
TypeUint32	uint32
TypeUint	uint32
TypeUint64	uint64
TypeFloat32	float
TypeFloat64	double

Validations:

• Field number 1 is reserved for the ID field
• No duplication of field numbers (this is illegal protobuf)
• Only supported ent field types are used

Custom Fields

In some edge cases, it may be required to override the automatic ent <> proto type mapping. This can be done by using the entproto.OverrideType, field option:

field.Uint8("custom_pb").
    Annotations(
        entproto.Field(12,
            entproto.Type(descriptorpb.FieldDescriptorProto_TYPE_UINT64),
        ),
    )

entproto.Enum

Proto Enum options, similar to message fields are assigned a numeric identifier that is expected to remain stable through all versions. This means, that a specific Ent Enum field option must always be translated to the same numeric identifier across the re-generation of the export code.

To accommodate this, we add an additional annotation (entproto.Enum) that maps between the Ent Enum options and their desired proto identifier:

// Fields of the Todo.
func (Todo) Fields() []ent.Field {
	return []ent.Field{
		field.String("task").
			Annotations(entproto.Field(2)),
		field.Enum("status").
			Values("pending", "in_progress", "done").
			Default("pending").
			Annotations(
				entproto.Field(3),
				entproto.Enum(map[string]int32{
					"pending":     0,
					"in_progress": 1,
					"done":        2,
				}),
			),
	}
}

Which is transformed into:

message Todo {
  int32 id = 1;

  string task = 2;

  Status status = 3;

  User user = 4;

  enum Status {
    PENDING = 0;

    IN_PROGRESS = 1;

    DONE = 2;
  }
}

As per the proto3 language guide for enums, the zero value (default) must always be specified. The Proto Style Guide suggests that we use CAPS_WITH_UNDERSCORES for value names, and a suffix of _UNSPECIFIED to the zero value. Ent supports specifying default values for Enum fields. We map this to proto enums in the following manner:

• If no default value is defined for the enum, we generate a <MessageName>_UNSPECIFIED = 0; option on the enum and verify that no option received the 0 number in the enproto.Enum Options field.
• If a default value is defined for the enum, we verify that it receives the 0 value on the Options field.

Edges

Edges are annotated in the same way as fields: using entproto.Field annotation to specify the field number for the generated field. Unique relations are mapped to normal fields, non-unique relations are mapped to repeated fields. For example:

func (BlogPost) Edges() []ent.Edge {
	return []ent.Edge{
		edge.To("author", User.Type).
			Unique().
			Annotations(entproto.Field(4)),
		edge.From("categories", Category.Type).
			Ref("blog_posts").
			Annotations(entproto.Field(5)),
	}
}

func (BlogPost) Fields() []ent.Field {
	return []ent.Field{
		field.String("title").
			Annotations(entproto.Field(2)),
		field.String("body").
			Annotations(entproto.Field(3)),
	}
}

Is transformed to:

message BlogPost {
  int32 id = 1;
  string title = 2;
  string body = 3;
  User author = 4;
  repeated Category categories = 5;
}

Validation:

• Cyclic dependencies are not supported in protobuf - so back references can only be supported if both messages are output to the same proto package. (In the above example, BlogPost, User and Category must be output to the same proto package).

Contributing

Code generation

Please re-generate all code using go generate ./... before checking code in - CI will fail on this check otherwise. To ensure you get the same output as the CI process make sure your local environment has the same protoc, protoc-gen-go and protoc-gen-go-grpc. See the environment setup in the ci.yaml file.

Codegen + Test flow

To rebuild the protoc-gen-entgrpc plugin, regenerate the code and run all tests:

go generate ./cmd/protoc-gen-entgrpc/... &&
  go install github.com/linuxluigi/ent-contrib/entproto/cmd/protoc-gen-entgrpc &&
  go generate ./... &&
  go test ./...

Running in Docker

If you prefer to run code-generation inside a Docker container you can use the provided Dockerfile that mimics the CI environment.

Build the image:

cd entproto
docker build --platform=linux/x86_64 -t entproto-dev .
cd ..

Run the image (from the root contrib/ directory), mounting your local source code into /go/src inside the container:

docker run --platform=linux/x86_64 -it -v $(pwd):/go/src -w /go/src/entproto entproto-dev bash

From within the Docker image, compile and install your current protoc-gen-entgrpc binary, regenerate all code and run the tests.

go install ./cmd/protoc-gen-entgrpc &&
	go install ./cmd/protoc-gen-ent &&
	go generate ./... &&
	go test ./...

Documentation

Index

• Constants
• Variables
• func Enum(opts map[string]int32) *enum
• func Field(num int, options ...FieldOption) schema.Annotation
• func Generate(g *gen.Graph) error
• func Hook() gen.Hook
• func Message(opts ...MessageOption) schema.Annotation
• func Service(opts ...ServiceOption) schema.Annotation
• func Skip() schema.Annotation
• func SkipGen() schema.Annotation
• type Adapter
  • func LoadAdapter(graph *gen.Graph) (*Adapter, error)
  • func (a *Adapter) AllFileDescriptors() map[string]*desc.FileDescriptor
  • func (a *Adapter) FieldMap(schemaName string) (FieldMap, error)
  • func (a *Adapter) GetFileDescriptor(schemaName string) (*desc.FileDescriptor, error)
  • func (a *Adapter) GetMessageDescriptor(schemaName string) (*desc.MessageDescriptor, error)
• type FieldMap
  • func (m FieldMap) Edges() []*FieldMappingDescriptor
  • func (m FieldMap) Enums() []*FieldMappingDescriptor
  • func (m FieldMap) Fields() []*FieldMappingDescriptor
  • func (m FieldMap) ID() *FieldMappingDescriptor
• type FieldMappingDescriptor
  • func (d *FieldMappingDescriptor) EdgeIDPbStructField() string
  • func (d *FieldMappingDescriptor) EdgeIDPbStructFieldDesc() *desc.FieldDescriptor
  • func (d *FieldMappingDescriptor) PbStructField() string
• type FieldOption
  • func Type(typ descriptorpb.FieldDescriptorProto_Type) FieldOption
  • func TypeName(n string) FieldOption
• type MessageOption
  • func PackageName(pkg string) MessageOption
• type Method
  • func (m Method) Is(n Method) bool
• type ServiceOption
  • func Methods(methods Method) ServiceOption

Constants

const (
	DefaultProtoPackageName = "entpb"
	IDFieldNumber           = 1
)

const (
	ServiceAnnotation = "ProtoService"
	// MaxPageSize is the maximum page size that can be returned by a List call. Requesting page sizes larger than
	// this value will return, at most, MaxPageSize entries.
	MaxPageSize = 1000
	// MaxBatchCreateSize is the maximum number of entries that can be created by a single BatchCreate call. Requests
	// exceeding this batch size will return an error.
	MaxBatchCreateSize = 1000
	// MethodCreate generates a Create gRPC service method for the entproto.Service.
	MethodCreate Method = 1 << iota
	// MethodGet generates a Get gRPC service method for the entproto.Service.
	MethodGet
	// MethodUpdate generates an Update gRPC service method for the entproto.Service.
	MethodUpdate
	// MethodDelete generates a Delete gRPC service method for the entproto.Service.
	MethodDelete
	// MethodList generates a List gRPC service method for the entproto.Service.
	MethodList
	// MethodBatchCreate generates a Batch Create gRPC service method for the entproto.Service.
	MethodBatchCreate
	// MethodAll generates all service methods for the entproto.Service. This is the same behavior as not including entproto.Methods.
	MethodAll = MethodCreate | MethodGet | MethodUpdate | MethodDelete | MethodList | MethodBatchCreate
)

const (
	EnumAnnotation = "ProtoEnum"
)

const FieldAnnotation = "ProtoField"

const MessageAnnotation = "ProtoMessage"

const SkipAnnotation = "ProtoSkip"

Variables

var (
	ErrEnumFieldsNotAnnotated = errors.New("entproto: all Enum options must be covered with an entproto.Enum annotation")
)

var (
	ErrSchemaSkipped = errors.New("entproto: schema not annotated with Generate=true")
)

Functions

func Enum

func Enum(opts map[string]int32) *enum

func Field

func Field(num int, options ...FieldOption) schema.Annotation

func Generate

func Generate(g *gen.Graph) error

Generate takes a *gen.Graph and creates .proto files. Next to each .proto file, Generate creates a generate.go file containing a //go:generate directive to invoke protoc and compile Go code from the protobuf definitions. If generate.go already exists next to the .proto file, this step is skipped.

func Hook

func Hook() gen.Hook

Hook returns a gen.Hook that invokes Generate. To use it programatically:

entc.Generate("./ent/schema", &gen.Config{
  Hooks: []gen.Hook{
    entproto.Hook(),
  },
})

func Message

func Message(opts ...MessageOption) schema.Annotation

Message annotates an ent.Schema to specify that protobuf message generation is required for it.

func Service

func Service(opts ...ServiceOption) schema.Annotation

Service annotates an ent.Schema to specify that protobuf service generation is required for it.

func Skip

func Skip() schema.Annotation

Skip annotates an ent.Schema to specify that this field will be skipped during .proto generation.

func SkipGen

func SkipGen() schema.Annotation

SkipGen annotates an ent.Schema to specify that protobuf message generation is not required for it. This is useful in cases where a schema ent.Mixin sets Generated to true and you want to specifically set it to false for this schema.

Types

type Adapter

type Adapter struct {
	// contains filtered or unexported fields
}

Adapter facilitates the transformation of ent gen.Type to desc.FileDescriptors

func LoadAdapter

func LoadAdapter(graph *gen.Graph) (*Adapter, error)

LoadAdapter takes a *gen.Graph and parses it into protobuf file descriptors

func (*Adapter) AllFileDescriptors

func (a *Adapter) AllFileDescriptors() map[string]*desc.FileDescriptor

AllFileDescriptors returns a file descriptor per proto package for each package that contains a successfully parsed ent.Schema

func (*Adapter) FieldMap

func (a *Adapter) FieldMap(schemaName string) (FieldMap, error)

FieldMap returns a FieldMap containing descriptors of all of the mappings between the ent schema field and the protobuf message's field descriptors.

func (*Adapter) GetFileDescriptor

func (a *Adapter) GetFileDescriptor(schemaName string) (*desc.FileDescriptor, error)

GetFileDescriptor returns the proto file descriptor containing the transformed proto message descriptor for `schemaName` along with any other messages in the same protobuf package.

func (*Adapter) GetMessageDescriptor

func (a *Adapter) GetMessageDescriptor(schemaName string) (*desc.MessageDescriptor, error)

GetMessageDescriptor retrieves the protobuf message descriptor for `schemaName`, if an error was returned while trying to parse that error they are returned

type FieldMap

type FieldMap map[string]*FieldMappingDescriptor

FieldMap contains a mapping between the field's name in the ent schema and a FieldMappingDescriptor.

func (FieldMap) Edges

func (m FieldMap) Edges() []*FieldMappingDescriptor

Edges returns the FieldMappingDescriptor for all of the edge fields of the schema. Items are sorted alphabetically on pb field name.

func (FieldMap) Enums

func (m FieldMap) Enums() []*FieldMappingDescriptor

func (FieldMap) Fields

func (m FieldMap) Fields() []*FieldMappingDescriptor

Fields returns the FieldMappingDescriptor for all of the fields of the schema. Items are sorted alphabetically on pb field name.

func (FieldMap) ID

func (m FieldMap) ID() *FieldMappingDescriptor

ID returns the FieldMappingDescriptor for the ID field of the schema.

type FieldMappingDescriptor

type FieldMappingDescriptor struct {
	EntField          *gen.Field
	EntEdge           *gen.Edge
	PbFieldDescriptor *desc.FieldDescriptor
	IsEdgeField       bool
	IsIDField         bool
	IsEnumField       bool
	ReferencedPbType  *desc.MessageDescriptor
}

FieldMappingDescriptor describes the mapping from a protobuf field descriptor to an ent Schema field

func (*FieldMappingDescriptor) EdgeIDPbStructField

func (d *FieldMappingDescriptor) EdgeIDPbStructField() string

EdgeIDPbStructField returns the name for the id field of the entity this edge refers to.

func (*FieldMappingDescriptor) EdgeIDPbStructFieldDesc

func (d *FieldMappingDescriptor) EdgeIDPbStructFieldDesc() *desc.FieldDescriptor

EdgeIDPbStructFieldDesc returns the protobuf field descriptor for the id field of the entity this edge refers to.

func (*FieldMappingDescriptor) PbStructField

func (d *FieldMappingDescriptor) PbStructField() string

PbStructField returns the protobuf field descriptor of this field.

type FieldOption

type FieldOption func(*pbfield)

func Type

func Type(typ descriptorpb.FieldDescriptorProto_Type) FieldOption

Type overrides the default mapping between ent types and protobuf types. Example:

field.Uint8("custom_pb").
	Annotations(
		entproto.Field(2,
			entproto.Type(descriptorpb.FieldDescriptorProto_TYPE_UINT64),
		),
	)

func TypeName

func TypeName(n string) FieldOption

TypeName sets the pb descriptors type name, needed if the Type attribute is TYPE_ENUM or TYPE_MESSAGE.

type MessageOption

type MessageOption func(msg *message)

MessageOption configures the entproto.Message annotation

func PackageName

func PackageName(pkg string) MessageOption

PackageName modifies the generated message's protobuf package name

type Method

type Method uint

func (Method) Is

func (m Method) Is(n Method) bool

Is reports whether method m matches given method n.

type ServiceOption

type ServiceOption func(svc *service)

ServiceOption configures the entproto.Service annotation.

func Methods

func Methods(methods Method) ServiceOption

Methods specifies the gRPC service methods to generate for the entproto.Service.