starlarkproto

package
v0.0.0-...-110cdec Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 22, 2024 License: Apache-2.0 Imports: 23 Imported by: 4

Documentation

Overview

Package starlarkproto exposes protobuf messages as Starlark types.

Internally a message is stored as a tree of Starlark values, with some type checking done when manipulating fields, based on proto message descriptors loaded dynamically from a serialized FileDescriptorSet.

For example, reading or assigning to a field not defined in a message will cause a runtime error. Similarly, trying to assign a value of a wrong type to a non-repeated field will fail.

Instantiating messages and default field values

Each proto message in a loaded package is exposed via a constructor function that takes optional keyword arguments and produces a new object of *Message type.

All unassigned fields are implicitly set to their default zero values on first access, including message-typed fields, lists and maps. It means, for example, if a message 'a' has a singular field 'b', that has a field 'c', it is always fine to write 'a.b.c' to read or set 'c' value, without explicitly checking that 'b' is set.

To clear a field, assign None to it (regardless of its type).

References and aliasing

Messages are passed around everywhere by reference. In particular it is possible to have multiple fields pointing to the exact same message, e.g. 

m1 = M()
m2 = M()
a = A()
m1.f = a
m2.f = a
a.i = 123  # changes both m1.f.i and m2.f.i

Note that 'm1.f = a' assignment checks the type of 'a', it should either match type of 'f' identically (no duck typing), or be a dict or None (which will be converted to messages, see below).

Working with repeated fields and maps

Values of repeated fields are represented by special sequence types that behave as strongly-typed lists. Think of them as list[T] types or as hypothetical 'message List<T>' messages.

When assigning a non-None value R to a repeated field F of type list[T], the following rules apply (sequentially):

  1. If R is not a sequence => error.
  2. If R has type list[T'], then a. If T == T', then F becomes an alias of R. b. If T != T' => error.
  3. A new list[T] is instantiated from R and assigned to F.

Notice that rule 2 is exactly like the aliasing rule for messages. This is where "think of them as 'message List<T>'" point of view comes into play.

As a concrete example, consider this: 

m1 = M()
m1.int64s = [1, 2]  # a list is implicitly converted (copied) to list[T]

m2 = M()
m2.int64s = m1.int64s  # points to the exact same list[T] object now
m2.int64s.append(3)    # updates both m1 and m2

m1.int32s = m1.int64s        # error! list[int64] is not list[int32]
m1.int32s = list(m1.int64s)  # works now (by making a copy of a list copy)

Maps behave in the similar way. Think of them as strongly-typed map<K,V> values or as 'message List<Pair<K,V>>' messages. They can be implicitly instantiated from iterable mappings (e.g. dicts).

Auto-conversion of dicts and None's into Messages

When assigning to a message-typed value (be it a field, an element of a list[T] or a value of map<K,V>) dicts are implicitly converted into messages as if via 'T(**d)' call. Similarly, None's are converted into empty messages, as if via 'T()' call.

Differences from starlarkproto (beside using different guts):

  • Message types are instantiated through proto.new_loader().
  • Text marshaller appends\removes trailing '\n' somewhat differently.
  • Text marshaller marshals empty messages as '<>' (without line breaks).
  • Bytes fields are represented as str, not as []uint8{...}.
  • proto.to_jsonpb doesn't have 'emit_defaults' kwarg (it is always False).
  • Better support for proto2 messages.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ProtoLib 

func ProtoLib() starlark.StringDict

ProtoLib returns a dict with single struct named "proto" that holds public Starlark API for working with proto messages.

Exported functions: 

def new_descriptor_set(name=None, blob=None, deps=None):
  """Returns a new DescriptorSet.

  Args:
    name: name of this set for debug and error messages, default is '???'.
    blob: raw serialized FileDescriptorSet, if any.
    deps: an iterable of DescriptorSet's with dependencies, if any.

  Returns:
    New DescriptorSet.
  """

def new_loader(*descriptor_sets):
  """Returns a new proto loader."""

def default_loader():
  """Returns a loader used by default when registering descriptor sets."""

def message_type(msg):
  """Returns proto.MessageType of the given message."""

def to_textpb(msg):
  """Serializes a protobuf message to text proto.

  Args:
    msg: a *Message to serialize.

  Returns:
    A str representing msg in text format.
  """

def to_jsonpb(msg, use_proto_names = None):
  """Serializes a protobuf message to JSONPB string.

  Args:
    msg: a *Message to serialize.
    use_proto_names: boolean, whether to use snake_case in field names
      instead of camelCase. The default is False.

  Returns:
    A str representing msg in JSONPB format.
  """

def to_wirepb(msg):
  """Serializes a protobuf message to a string using binary wire encoding.

  Args:
    msg: a *Message to serialize.

  Returns:
    A str representing msg in binary wire format.
  """

def from_textpb(ctor, body):
  """Deserializes a protobuf message given in text proto form.

  Unknown fields are not allowed.

  Args:
    ctor: a message constructor function.
    body: a string with serialized message.
    discard_unknown: boolean, whether to discard unrecognized fields. The
      default is False.

  Returns:
    Deserialized frozen message constructed via `ctor`.
  """

def from_jsonpb(ctor, body):
  """Deserializes a protobuf message given as JBONPB string.

  Unknown fields are silently skipped.

  Args:
    ctor: a message constructor function.
    body: a string with serialized message.
    discard_unknown: boolean, whether to discard unrecognized fields. The
      default is True.

  Returns:
    Deserialized frozen message constructed via `ctor`.
  """

def from_wirepb(ctor, body):
  """Deserializes a protobuf message given its wire serialization.

  Unknown fields are silently skipped.

  Args:
    ctor: a message constructor function.
    body: a string with serialized message.
    discard_unknown: boolean, whether to discard unrecognized fields. The
      default is True.

  Returns:
    Deserialized frozen message constructed via `ctor`.
  """

def struct_to_textpb(s):
  """Converts a struct to a text proto string.

  Args:
    s: a struct object. May not contain dicts.

  Returns:
    A str containing a text format protocol buffer message.
  """

def clone(msg):
  """Returns a deep copy of a given proto message.

  Args:
    msg: a proto message to make a copy of.

  Returns:
    A deep copy of the message
  """

def has(msg, field):
  """Checks if a proto message has the given optional field set.

  Args:
    msg: a message to check.
    field: a string name of the field to check.

  Returns:
    True if the message has the field set.
  """

func SetDefaultLoader 

func SetDefaultLoader(th *starlark.Thread, l *Loader)

SetDefaultLoader installs the given loader as default in the thread.

It can be obtained via DefaultLoader or proto.default_loader() from Starlark. Note that Starlark code has no way of changing the default loader. It's responsibility of the hosting environment to prepare the default loader (just like it prepares starlark.Thread itself).

func SetMessageCache 

func SetMessageCache(th *starlark.Thread, mc MessageCache)

SetMessageCache installs the given messages cache into the thread.

It will be used by all deserialization operations performed by the Starlark code in this thread.

func ToJSONPB 

func ToJSONPB(msg *Message, useProtoNames bool) ([]byte, error)

ToJSONPB serializes a protobuf message to JSONPB string.

func ToTextPB 

func ToTextPB(msg *Message) ([]byte, error)

ToTextPB serializes a protobuf message to text proto.

func ToWirePB 

func ToWirePB(msg *Message) ([]byte, error)

ToWirePB serializes a protobuf message to binary wire format.

Types

type DescriptorSet 

type DescriptorSet struct {
	// contains filtered or unexported fields
}

DescriptorSet contains FileDescriptorProto of zero or more *.proto files, along with pointers to DescriptorSets with their imports.

A descriptor set can be registered in a Loader. Doing so transitively registers all its dependencies. See Loader.AddDescriptorSet for more details.

Implements starlark.Value and starlark.HasAttrs interfaces. Usage from Starlark side may look like this: 

load(".../wellknown_descpb.star", "wellknown_descpb")
myprotos_descpb = proto.new_descriptor_set(
    name = "myprotos",
    deps = [wellknown_descpb],
    blob = io.read_file("myprotos.descpb"),
)
myprotos_descpb.register()

By default register() registers the descriptor set in the default loader, i.e. ds.register() is same as ds.register(loader=proto.default_loader()). Also note that ds.register(loader=l) is a sugar for l.add_descriptor_set(ds), so ds.register() is same as proto.default_loader().add_descriptor_set(ds), just much shorter.

func NewDescriptorSet 

func NewDescriptorSet(name string, fdps []*descriptorpb.FileDescriptorProto, deps []*DescriptorSet) (*DescriptorSet, error)

NewDescriptorSet evaluates given file descriptors and their dependencies and produces new DescriptorSet if there are no unresolved imports and no duplicated files.

'fdps' should be ordered topologically (i.e. if file A imports file B, then B should precede A in 'fdps' or be somewhere among 'deps'). This is always the case when generating sets via 'protoc --descriptor_set_out=...'.

Note that dependencies can only be specified when creating the descriptor set and can't be changed later. Cycles thus are impossible.

func (*DescriptorSet) Attr 

func (ds *DescriptorSet) Attr(name string) (starlark.Value, error)

Attr returns an attribute given its name (or nil if not present).

func (*DescriptorSet) AttrNames 

func (ds *DescriptorSet) AttrNames() []string

AtrrNames lists available attributes.

func (*DescriptorSet) Freeze 

func (ds *DescriptorSet) Freeze()

Freeze does nothing since DescriptorSet is already immutable.

func (*DescriptorSet) Hash 

func (ds *DescriptorSet) Hash() (uint32, error)

Hash returns unique value associated with this set.

func (*DescriptorSet) String 

func (ds *DescriptorSet) String() string

String returns str(...) representation of the set, for debug messages.

func (*DescriptorSet) Truth 

func (ds *DescriptorSet) Truth() starlark.Bool

Truth returns the truth value of an object.

func (*DescriptorSet) Type 

func (ds *DescriptorSet) Type() string

Type returns a short string describing the value's type.

type Loader 

type Loader struct {
	// contains filtered or unexported fields
}

Loader can instantiate Starlark values that correspond to proto messages.

Holds a pool of descriptors that describe all available proto types. Use AddDescriptorSet to seed it. Once seeded, use Module to get a Starlark module with symbols defined in some registered `*.proto` file.

Loader is also a Starlark value itself, with the following methods:

  • add_descriptor_set(ds) - see AddDescriptorSet.
  • module(path) - see Module.

Can be used concurrently. Non-freezable.

func DefaultLoader 

func DefaultLoader(th *starlark.Thread) *Loader

DefaultLoader returns a loader installed in the thread via SetDefaultLoader.

Returns nil if there's no default loader.

func NewLoader 

func NewLoader() *Loader

NewLoader instantiates a new loader with empty proto registry.

func (*Loader) AddDescriptorSet 

func (l *Loader) AddDescriptorSet(ds *DescriptorSet) error

AddDescriptorSet makes all *.proto files defined in the given descriptor set and all its dependencies available for use from Starlark.

AddDescriptorSet is idempotent in a sense that calling AddDescriptorSet(ds) multiple times with the exact same 'ds' is not an error. But trying to register a proto file through multiple different descriptor sets is an error.

func (*Loader) Attr 

func (l *Loader) Attr(name string) (starlark.Value, error)

Attr returns an attribute given its name (or nil if not present).

func (*Loader) AttrNames 

func (l *Loader) AttrNames() []string

AtrrNames lists available attributes.

func (*Loader) Freeze 

func (l *Loader) Freeze()

Freeze is noop for now.

func (*Loader) Hash 

func (l *Loader) Hash() (uint32, error)

Hash returns an integer assigned to this loader when it was created.

func (*Loader) MessageType 

func (l *Loader) MessageType(desc protoreflect.MessageDescriptor) *MessageType

MessageType creates new (or returns existing) MessageType.

The return value can be used to instantiate Starlark values via Message() or MessageFromProto(m).

func (*Loader) Module 

func (l *Loader) Module(path string) (*starlarkstruct.Module, error)

Module returns a module with top-level definitions from some *.proto file.

The descriptor of this proto file should be registered already via AddDescriptorSet. 'path' here is matched to what's in the descriptor, which is a path to *.proto EXACTLY as it was given to 'protoc'.

The name of the module matches the proto package name (per 'package ...' statement in the proto file).

func (*Loader) String 

func (l *Loader) String() string

String returns str(...) representation of the loader.

func (*Loader) Truth 

func (l *Loader) Truth() starlark.Bool

Truth returns True.

func (*Loader) Type 

func (l *Loader) Type() string

Type returns "proto.Loader".

func (*Loader) Types 

func (l *Loader) Types() *protoregistry.Types

Types returns a registry for looking up or iterating over descriptor types.

type Message 

type Message struct {
	// contains filtered or unexported fields
}

Message is a Starlark value that implements a struct-like type structured like a protobuf message.

Implements starlark.Value, starlark.HasAttrs, starlark.HasSetField and starlark.Comparable interfaces.

Can be instantiated through Loader as loader.MessageType(...).Message() or loader.MessageType(...).MessageFromProto(p).

TODO(vadimsh): Currently not safe for a cross-goroutine use without external locking, even when frozen, due to lazy initialization of default fields on first access.

func FromJSONPB 

func FromJSONPB(typ *MessageType, blob []byte, discardUnknown bool) (*Message, error)

FromJSONPB deserializes a protobuf message given as JBONPB string.

Unlike the equivalent Starlark proto.from_jsonpb(...), this low-level native function doesn't freeze returned messages, but also doesn't use the message cache.

func FromTextPB 

func FromTextPB(typ *MessageType, blob []byte, discardUnknown bool) (*Message, error)

FromTextPB deserializes a protobuf message given in text proto form.

Unlike the equivalent Starlark proto.from_textpb(...), this low-level native function doesn't freeze returned messages, but also doesn't use the message cache.

func FromWirePB 

func FromWirePB(typ *MessageType, blob []byte, discardUnknown bool) (*Message, error)

FromWirePB deserializes a protobuf message given as a wire-encoded blob.

Unlike the equivalent Starlark proto.from_wirepb(...), this low-level native function doesn't freeze returned messages, but also doesn't use the message cache.

func (*Message) Attr 

func (m *Message) Attr(name string) (starlark.Value, error)

Attr is called when a field is read from Starlark code.

func (*Message) AttrNames 

func (m *Message) AttrNames() []string

AttrNames lists available attributes.

func (*Message) CompareSameType 

func (m *Message) CompareSameType(op syntax.Token, y starlark.Value, depth int) (bool, error)

CompareSameType does 'm <op> y' comparison.

func (*Message) Freeze 

func (m *Message) Freeze()

Freeze makes this message immutable.

func (*Message) FromDict 

func (m *Message) FromDict(d starlark.IterableMapping) error

FromDict populates fields of this message based on values in an iterable mapping (usually a starlark.Dict).

Doesn't reset the message. Basically does this: 

for k in d:
  setattr(msg, k, d[k])

Returns an error on type mismatch.

func (*Message) HasProtoField 

func (m *Message) HasProtoField(name string) bool

HasProtoField returns true if the message has the given field initialized.

func (*Message) Hash 

func (m *Message) Hash() (uint32, error)

Hash returns an error, indicating proto messages are not hashable.

func (*Message) IsFrozen 

func (m *Message) IsFrozen() bool

IsFrozen returns true if this message was frozen already.

func (*Message) MessageType 

func (m *Message) MessageType() *MessageType

MessageType returns type information about this message.

func (*Message) SetField 

func (m *Message) SetField(name string, val starlark.Value) error

SetField is called when a field is assigned to from Starlark code.

func (*Message) String 

func (m *Message) String() string

String returns compact text serialization of this message.

func (*Message) ToProto 

func (m *Message) ToProto() proto.Message

ToProto returns a new populated proto message of an appropriate type.

func (*Message) Truth 

func (m *Message) Truth() starlark.Bool

Truth always returns True.

func (*Message) Type 

func (m *Message) Type() string

Type returns full proto message name.

type MessageCache 

type MessageCache interface {
	// Fetch returns a previously stored message or (nil, nil) if missing.
	Fetch(th *starlark.Thread, cache, body string, typ *MessageType) (*Message, error)
	// Store stores a deserialized message.
	Store(th *starlark.Thread, cache, body string, msg *Message) error
}

MessageCache knows how to cache deserialized messages.

Semantically it is a map `(cache, typ, body) => frozen Message`, where `cache` identifies a serialization format, `typ` identifies the type being deserialized and `body` is the raw message body to be deserialized.

Works with frozen messages.

Use SetMessageCache to install a cache into a starlark thread.

type MessageType 

type MessageType struct {
	*starlark.Builtin // the callable, initialize in Loader
	// contains filtered or unexported fields
}

MessageType represents a proto message type and acts as its constructor: it is a Starlark callable that produces instances of Message.

Implements starlark.HasAttrs interface. Attributes represent constructors for nested messages and int values of enums. Note that starlark.HasSetField is not implemented, making values of MessageType immutable.

Stringifying an instance of MessageType (e.g. with `str(...)` in Starlark) returns its full proto type name.

Given a MessageDescriptor, MessageType can be instantiated through Loader as loader.MessageType(...).

func (*MessageType) Attr 

func (t *MessageType) Attr(name string) (starlark.Value, error)

Attr returns either a nested message or an enum value.

func (*MessageType) AttrNames 

func (t *MessageType) AttrNames() []string

AttrNames return names of all nested messages and enum values.

func (*MessageType) Converter 

func (t *MessageType) Converter() typed.Converter

Converter returns an object that can convert Starlark dicts and Nones to values of this message type.

Can be used by typed.List and typed.Dict.

func (*MessageType) Descriptor 

func (t *MessageType) Descriptor() protoreflect.MessageDescriptor

Descriptor returns protobuf type information for this message type.

func (*MessageType) Loader 

func (t *MessageType) Loader() *Loader

Loader returns the loader this message type was loaded from.

func (*MessageType) Message 

func (t *MessageType) Message() *Message

Message instantiates a new empty message of this type.

func (*MessageType) MessageFromProto 

func (t *MessageType) MessageFromProto(p proto.Message) *Message

MessageFromProto instantiates a new message of this type and populates it based on values in the given proto.Message that should have a matching type.

Here "matching type" means p.ProtoReflect().Descriptor() *is* t.Descriptor(). Panics otherwise.

func (*MessageType) String 

func (t *MessageType) String() string

String returns the full proto type name.

func (*MessageType) Type 

func (t *MessageType) Type() string

Type returns "proto.MessageType", it's the type of the message type itself.

It is sort of like a meta-type, i.e. like "type" type in Python.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.