protocompile

README

NOTICE

This repo has moved. Please see https://github.com/bufbuild/protocompile for the official sources for this library.

If you want to view any issues and PR history from before it moved to the bufbuild organization, see https://github.com/jhump/protocompile-old.

Documentation

Overview

Package protocompile provides the entry point for a high performance native Go protobuf compiler. "Compile" in this case just means parsing and validating source and generating fully-linked descriptors in the end. Unlike the protoc command-line tool, this package does not try to use the descriptors to perform code.

The various sub-packages represent the various compile phases and contain models for the intermediate results. Those phases follow:

1. Parse into AST. Also see: parser.Parse
2. Convert AST to unlinked descriptor protos. Also see: parser.ResultFromAST
3. Link descriptor protos into "rich" descriptors. Also see: linker.Link
4. Interpret custom options. Also see: options.InterpretOptions
5. Generate source code info. Also see: sourceinfo.GenerateSourceInfo

This package provides an easy-to-use interface that does all of the phases relevant, based on the inputs given. It is also capable of taking advantage of multiple CPU cores, so a compilation involving thousands of files can be done very quickly by compiling things in parallel.

Resolvers

A Resolver is how the compiler locates artifacts that are inputs to the compilation. For example, it can load protobuf source code that must be processed. A Resolver could also supply some already-compiled dependencies as fully-linked descriptors, alleviating the need to re-compile them.

A Resolver can provide any of the following in response to a query for an input.

• Source code: If a resolver answers a query with protobuf source, the compiler will parse and compile it.
• AST: If a resolver answers a query with an AST, the parsing step can be skipped, and the rest of the compilation steps will be applied.
• Descriptor proto: If a resolver answers a query with an unlinked proto, only the other compilation steps, including linking, need to be applied.
• Descriptor: If a resolver answers a query with a fully-linked descriptor, nothing further needs to be done. The descriptor is used as-is.

Compilation will use the Resolver to load the files that are to be compiled and also to load all dependencies (i.e. other files imported by those being compiled).

Compiler

A Compiler accepts a list of file names and produces the list of descriptors. A Compiler has several fields that control how it works but only the Resolver field is required. A minimal Compiler, that resolves files by loading them from the file system based on the current working directory, can be had with the following simple snippet:

compiler := protocompile.Compiler{
    Resolver: &protocompile.SourceResolver{},
}

This minimal Compiler will use default parallelism, equal to the number of CPU cores detected; it will not generate source code info in the resulting descriptors; and it will fail fast at the first sign of any error. All of these aspects can be customized by setting other fields.

Index

• func ProtoFromFileDescriptor(f protoreflect.FileDescriptor) *descriptorpb.FileDescriptorProto
• func SourceAccessorFromMap(srcs map[string]string) func(string) (io.ReadCloser, error)
• type Compiler
  • func (c *Compiler) Compile(ctx context.Context, files ...string) (linker.Files, error)
• type CompositeResolver
  • func (f CompositeResolver) FindFileByPath(path string) (SearchResult, error)
• type PanicError
  • func (p PanicError) Error() string
• type Resolver
  • func WithStandardImports(r Resolver) Resolver
• type ResolverFunc
  • func (f ResolverFunc) FindFileByPath(path string) (SearchResult, error)
• type SearchResult
• type SourceResolver
  • func (r *SourceResolver) FindFileByPath(path string) (SearchResult, error)

Functions

func ProtoFromFileDescriptor

func ProtoFromFileDescriptor(f protoreflect.FileDescriptor) *descriptorpb.FileDescriptorProto

ProtoFromFileDescriptor extracts a descriptor proto from the given "rich" descriptor. For file descriptors generated by the compiler, this is an inexpensive and non-lossy operation. File descriptors from other sources however may be expensive (to re-create a proto) and even lossy.

func SourceAccessorFromMap

func SourceAccessorFromMap(srcs map[string]string) func(string) (io.ReadCloser, error)

SourceAccessorFromMap returns a function that can be used as the Accessor field of a SourceResolver that uses the given map to load source. The map keys are file names and the values are the corresponding file contents.

Types

type Compiler

type Compiler struct {
	// Resolves path/file names into source code or intermediate representions
	// for protobuf source files. This is how the compiler loads the files to
	// be compiled as well as all dependencies. This field is the only required
	// field.
	Resolver Resolver
	// The maximum parallelism to use when compiling. If unspecified or set to
	// a non-positive value, then min(runtime.NumCPU(), runtime.GOMAXPROCS(-1))
	// will be used.
	MaxParallelism int
	// A custom error and warning reporter. If unspecified a default reporter
	// is used. A default reporter fails the compilation after encountering any
	// errors and ignores all warnings.
	Reporter reporter.Reporter

	// If true, source code information will be included in the resulting
	// descriptors. Source code information is metadata in the file descriptor
	// that provides position information (i.e. the line and column where file
	// elements were defined) as well as comments.
	//
	// If Resolver returns descriptors or descriptor protos for a file, then
	// those descriptors will not be modified. If they do not already include
	// source code info, they will be left that way when the compile operation
	// concludes. Similarly, if they already have source code info but this flag
	// is false, existing info will be left in place.
	IncludeSourceInfo bool
}

Compiler handles compilation tasks, to turn protobuf source files, or other intermediate representations, into fully linked descriptors.

The compilation process involves five steps for each protobuf source file:

1. Parsing the source into an AST (abstract syntax tree).
2. Converting the AST into descriptor protos.
3. Linking descriptor protos into fully linked descriptors.
4. Interpreting options.
5. Computing source code information.

With fully linked descriptors, code generators and protoc plugins could be invoked (though that step is not implemented by this package and not a responsibility of this type).

func (*Compiler) Compile

func (c *Compiler) Compile(ctx context.Context, files ...string) (linker.Files, error)

Compile compiles the given file names into fully-linked descriptors. The compiler's resolver is used to locate source code (or intermediate artifacts such as parsed ASTs or descriptor protos) and then do what is necessary to transform that into descriptors (parsing, linking, etc).

type CompositeResolver

type CompositeResolver []Resolver

CompositeResolver is a slice of resolvers, which are consulted in order until one can supply a result. If none of the constituent resolvers can supply a result, the error returned by the first resolver is returned. If the slice of resolvers is empty, all operations return protoregistry.NotFound.

func (CompositeResolver) FindFileByPath

func (f CompositeResolver) FindFileByPath(path string) (SearchResult, error)

type PanicError

type PanicError struct {
	// The file that was being processed when the panic occurred
	File string
	// The value returned by recover()
	Value interface{}
	// A formatted stack trace
	Stack string
}

PanicError is an error value that represents a recovered panic. It includes the value returned by recover() as well as the stack trace.

This should generally only be seen if a Resolver implementation panics.

An error returned by a Compiler may wrap a PanicError, so you may need to use errors.As(...) to access panic details.

func (PanicError) Error

func (p PanicError) Error() string

Error implements the error interface. It does NOT include the stack trace. Use a type assertion and query the Stack field directly to access that.

type Resolver

type Resolver interface {
	// FindFileByPath searches for information for the given file path. If no
	// result is available, it should return a non-nil error, such as
	// protoregistry.NotFound.
	FindFileByPath(path string) (SearchResult, error)
}

Resolver is used by the compiler to resolve a proto source file name into some unit that is usable by the compiler. The result could be source for a proto file or it could be an already-parsed AST or descriptor.

func WithStandardImports

func WithStandardImports(r Resolver) Resolver

WithStandardImports returns a new resolver that knows about the same standard imports that are included with protoc.

type ResolverFunc

type ResolverFunc func(string) (SearchResult, error)

ResolverFunc is a simple function type that implements Resolver.

func (ResolverFunc) FindFileByPath

func (f ResolverFunc) FindFileByPath(path string) (SearchResult, error)

type SearchResult

type SearchResult struct {
	// Represents source code for the file. This should be nil if source code
	// is not available. If no field below is set, then the compiler will parse
	// the source code into an AST.
	Source io.Reader
	// Represents the abstract syntax tree for the file. If no field below is
	// set, then the compiler will convert the AST into a descriptor proto.
	AST *ast.FileNode
	// A descriptor proto that represents the file. If the field below is not
	// set, then the compiler will link this proto with its dependencies to
	// produce a linked descriptor.
	Proto *descriptorpb.FileDescriptorProto
	// A fully linked descriptor that represents the file. If this field is set,
	// then the compiler has no additional work to do for this file as it is
	// already compiled.
	Desc protoreflect.FileDescriptor
}

SearchResult represents information about a proto source file. Only one of the various fields must be set, based on what is available for a file. If multiple fields are set, the compiler prefers them in opposite order listed: so it uses a descriptor if present and only falls back to source if nothing else if available.

type SourceResolver

type SourceResolver struct {
	// Optional list of import paths. If present and not empty, then all
	// file paths to find are assumed to be relative to one of these paths.
	// If nil or empty, all file paths to find are assumed to be relative to
	// the current working directory.
	ImportPaths []string
	// Optional function for returning a file's contents. If nil, then
	// os.Open is used to open files on the file system.
	Accessor func(path string) (io.ReadCloser, error)
}

SourceResolver can resolve file names by returning source code. It uses an optional list of import paths to search. By default, it searches the file system.

func (*SourceResolver) FindFileByPath

func (r *SourceResolver) FindFileByPath(path string) (SearchResult, error)