properties

package
v0.0.0-...-f248f4a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 19, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package properties encapsulates all logic and data structures for parsing the input properties and manipulating the output properties of a LUCI Build.

LUCI Build input and output properties are proto Struct objects (effectively JSON objects). As such, they have very little type/schema information, and are cumbersome to directly manipulate/interact with.

Additionally, both reads and writes to the property values must be synchronized within the process and ideally be type-safe (e.g. to prevent manipulating properties in ways which change their schema from one area of the program to another - say a key which is sometimes an number and sometimes a list).

This package provides:

  • transparent, type-safe methods for reading and writing property data to/from proto Message classes as well as Go structs.
  • Goroutine-safe manipulation and sending support.
  • Ability to instantiate `properties` in Context apart from an entire LUCI Build, meaning that code can interact with properties for input and output without any other LUCI/luciexe mechanisms (highly useful for tests).

Data Model

Logically input and output `properties` exist as single JSON objects. This package divides that object into multiple type-safe regions via a Registry. If you are using this with the go.chromium.org/luci/luciexe/build library, see go.chromium.org/luci/luciexe/build.Properties.

The singular Input or Output struct in a LUCI Build are allowed a schema to describe the 'top-level', minus any keys described by a registered namespace. Namespaces other than the top level MUST begin with "$".

For example: 

{
  "some_key": 100,
  "$other key": {
     "sub": "hello"
  },
  "$another": { "lst": [1, 2, 3] }
}

Could be broken into 3 schemas: 

message TopLevel {  // registered to "", meaning top-level
	int some_key = 1;
}

type OtherStruct struct {  // registered to "$other key"
  Sub string `json:"sub"`
}

message Another {  // registered to "$another"
  repeated int lst = 1;
}

Property Types

All of the (Must)?Register(In)?(Out)? functions have additional restrictions on the types that they accept which cannot currently be expressed in terms of Go's generics.

InT, OutT or T must be one of the following (in order of preference):

Using *google.golang.org/protobuf/types/known/structpb.Struct directly implements a pass-through encoding. This will allow you to register properties with ~zero overhead where you need to do something even more custom than using protojson or encoding/json. Note that working with Struct directly can be extremely annoying, so YMMV :).

When parsing input properties, unknown fields will, by default, be logged at `Warning` level and ignored, but you can turn this into a hard error with OptRejectUnknownFields(). This default was selected to ease migration without making property typos completely silent.

If the top level type is a *structpb.Struct or a map type, it will simply collect all otherwise-unaccounted-for top-level keys. In this case, if state.Serialize() would case a top-level property to be overwritten by a namespace, it will return an error.

Namespaces

All of the (Must)?Register(In)?(Out)? functions take a required 'namespace' argument. This is a key within the top-level set of input or output properties.

If namespace is "", this will be the property namespace for the 'top level' property message, otherwise this is a namespace inside the top level properties and MUST begin with "$".

By default, any incoming top-level properties beginning with "$" will be ignored when parsing the input properties, if they don't have an associated schema. This allows callers to pass data in for Go modules which a given build MAY consume, without having this be an error, and still checking that other top-level properties are not misspelled.

Example: 

// the program registered
var topLevel = properties.MustRegister[*struct{
  Specific string `json:"specific"`
  Spelling string `json:"spelling"`
}](..., "")

// but the unaware caller passes
{
  "specific": "property",
  "speling": "oops",
  "$common/module": {
     "somefield": 100
  }
}

This will just result in a logged warning for "speling" - "$common/module" will be ignored. If the program eventually evolves and imports the Go module which registers the "$common/module" namespace, the caller's settings will take effect.

However, if you absolutely want to stamp out these ignored namespaces, you can pass the OptStrictTopLevelFields() option when registering the top level namespace.

Proto vs JSON tradeoffs

Recommendation: Always prefer Protos where you can, except where you are 100% certain that you will never need to interop with other programs. JSON maps and *structpb.Struct can be used to ease migration from JSON -> Proto.

Protos have the advantage that they can be generated for any language and are logically independent of the Go program source. This means that other programs can safely interact (write input properties, read output properties) with this Go program without needing to share source with this program.

However, protos can be cumbersome to generate (though the `cproto` helper in LUCI makes this much easier to manage) vs 'just' Go structs, and sometimes you really do just need something quick.

Think carefully about how your property messages will be used, how migrating from one format to another could be painful, etc. before picking one. If you are unsure, I would recommend to just use a Proto message.

Known Gotcha - Protos in the main package

If you generate protobuf stubs in the same package that you register _at init time_ properties with those protobufs, you MAY see a cryptic panic about some of the protobuf reflection guts being nil/uninitialized. This is because the current protobuf stub code populates its reflection data at init-time, and the order of execution between different init stanzas within the same package is *completely arbitrary*.

To solve this:

  • Move your protos to a sub-package which is imported. This guarantees that the imported package's init will run before any of the importer's init actions.
  • OR: If your package is the `main` package, you can move the property registration inside the `main()` module, before the property registry is Initialized. If your package is not the main module, then please move the protos to a subpackage.
  • OR: You can use a Go struct type instead of a proto - this is less recommended, however, if you need the uniformity of protos to interoperate with other programs (e.g. that will syntheize inputs or parse outputs from your program).

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type RegisterOption 

type RegisterOption func(opts *registerOptions, namespace string, inT, outT reflect.Type) error

A RegisterOption is the way to specify extra behavior when calling any of the (Must)?Register(In)?(Out)? functions.

func OptJSONUseNumber 

func OptJSONUseNumber() RegisterOption

OptJSONUseNumber is a RegisterOption, which will make Register use the json.Number when decoding a number to an `any` as part of a Go struct or map.

Error for non-JSON messages. Error for output-only properties.

func OptProtoUseJSONNames 

func OptProtoUseJSONNames() RegisterOption

OptProtoUseJSONNames returns a RegisterOption, which will make Register use the protobuf default names when serializing proto.Message types. These names are:

  • some_thing -> someThing
  • some_thing [json_name = "wow"] -> wow

By default, this library will use the 'proto' name, so fields in the JSON output appear exactly how the fields exist in in the .proto file. This generally causes much less confusion when working with JSONPB because what you see in the .proto file is what you get in the data which makes gerpping and debugging much easier.

Regardless of this option, when deserializing from Struct, all possible field names are allowed (so, lowerCamelCase, lower_camel_case and json_name annotation).

Error for input-only properties. Error for non-proto messages.

func OptRejectUnknownFields 

func OptRejectUnknownFields() RegisterOption

OptRejectUnknownFields returns a RegisterOption which allows your registered proto or struct to re'ect unknown fields when parsing the input value, rather than just logging them as WARNINGS.

By default this library will log warnings for all unknown fields when parsing the input, which prevents accidental typos, etc. from being completely undetected, but sometimes you need a guard which is even stricter than just logging warnings.

However, setting this option means that migrations need to be handled with care:

  • When adding a field, you MUST add it to the binary, and deploy ALL AFFECTED binaries, before setting the field.
  • When removing a field, make sure to mark the name as reserved in proto, or keep the old field name around in the Go struct to mark it as deprecated. This will allow new binaries to continue to process properties set for older binaries.

Example: 

message Msg {
  string some_field = 1;
}

// by default
{ "somefield": "hello" } => logs WARNING, Msg.some_field will be ""

// with OptRejectUnknownFields()
{ "somefield": "hello" } => logs ERROR and quits

Error for output-only properties.

func OptSkipFrames 

func OptSkipFrames(frames int) RegisterOption

OptSkipFrames returns a RegisterOption which allows you to skip additional frames when Register{Proto,Struct} walk the stack looking for the registration location.

If supplied multiple times, the number of frames skipped accumulates (i.e. `OptSkipFrames(1), OptSkipFrames(1)` is the same as `OptSkipFrames(2)`

func OptStrictTopLevelFields 

func OptStrictTopLevelFields() RegisterOption

OptStrictTopLevelFields returns a RegisterOption which changes the way that top level fields are handled.

By default, any top-level fields not parsed by the top-level namespace are ignored if they start with "$". All (Must)?Register(In)?(Out)? functions enforce that registered namespaces are either "", or begin with a "$". This default is a compromise between completely ignoring typo inputs and also causing builds to fail when passed configuration for a Go module that they happen to not use.

By specifying OptStrictTopLevelFields, ALL extra top-level fields not parsed by the top-level namespace will be errors. Implies OptRejectUnknownFields().

Error if used on non-top-level registrations. Error if top-level input type is a map type. Error for output-only properties.

type RegisteredProperty 

type RegisteredProperty[InT, OutT any] struct {
	RegisteredPropertyIn[InT]
	RegisteredPropertyOut[OutT]
}

RegisteredProperty is the typesafe interface to read the input value of a property, as well as mutate the output value of a property.

Note that this exports all the methods of RegisteredPropertyIn[InT] and RegisteredPropertyOut[OutT], but has no unique methods of its own. It is also possible to directly pass the RegisteredPropertyIn or RegisteredPropertyOut as a way to split this into a read-only or write-only portion.

Obtain this via Register, MustRegister, RegisterInOut or MustRegisterInOut.

func MustRegister 

func MustRegister[T any](r *Registry, namespace string, opts ...RegisterOption) RegisteredProperty[T, T]

MustRegister is a slimmed version of Register, to be used at init()-time where you statically know that there should never be a registration error.

func MustRegisterInOut 

func MustRegisterInOut[InT, OutT any](r *Registry, namespace string, opts ...RegisterOption) RegisteredProperty[InT, OutT]

MustRegisterInOut is a slimmed version of RegisterInOut, to be used at init()-time where you statically know that there should never be a registration error.

func Register 

func Register[T any](r *Registry, namespace string, opts ...RegisterOption) (ret RegisteredProperty[T, T], err error)

Register returns a RegisteredProperty with both In and Out types being T.

Refer to package documentation for the acceptable types for T.

func RegisterInOut 

func RegisterInOut[InT, OutT any](r *Registry, namespace string, opts ...RegisterOption) (ret RegisteredProperty[InT, OutT], err error)

RegisterInOut returns a RegisteredProperty with differint types for InT and OutT. Prefer Register if you need the same input and output types.

Refer to package documentation for the acceptable types for T.

type RegisteredPropertyIn 

type RegisteredPropertyIn[InT any] struct {
	// contains filtered or unexported fields
}

RegisteredPropertyIn is the typesafe interface to read the input value of a property.

Obtain this via RegisterIn or MustRegisterIn.

func MustRegisterIn 

func MustRegisterIn[InT any](r *Registry, namespace string, opts ...RegisterOption) RegisteredPropertyIn[InT]

MustRegisterIn is a slimmed version of RegisterIn, to be used at init()-time where you statically know that there should never be a registration error.

func RegisterIn 

func RegisterIn[InT any](r *Registry, namespace string, opts ...RegisterOption) (ret RegisteredPropertyIn[InT], err error)

RegisterIn returns a RegisteredPropertyIn[InT].

Refer to package documentation for the acceptable types for T.

func (RegisteredPropertyIn[InT]) GetInput 

func (rp RegisteredPropertyIn[InT]) GetInput(ctx context.Context) InT

GetInput returns the initial value for this registered property from the State embedded in `ctx`.

If the input did not contain a value for this property, or State is not present in `ctx`, this returns `nil`.

This will always return the same value - you MUST NOT mutate `T`. If you need to do so, make a copy of it (e.g. with proto.Clone) and then modify the copy.

This will panic if:

  • This RegisteredProperty is uninitialized (i.e. constructed without calling Register).
  • This RegisteredProperty is registered with a different Registry from the one embedded in `ctx`.

func (RegisteredPropertyIn[InT]) GetInputFromState 

func (rp RegisteredPropertyIn[InT]) GetInputFromState(s *State) InT

GetInputFromState is the same as GetInput, but uses a directly supplied State instead of getting it from `ctx`.

type RegisteredPropertyOut 

type RegisteredPropertyOut[OutT any] struct {
	// contains filtered or unexported fields
}

RegisteredPropertyOut is the typesafe interface to mutate the output value of a property.

Obtain this via RegisterOut or MustRegisterOut.

func MustRegisterOut 

func MustRegisterOut[OutT any](r *Registry, namespace string, opts ...RegisterOption) RegisteredPropertyOut[OutT]

MustRegisterOut is a slimmed version of RegisterOut, to be used at init()-time where you statically know that there should never be a registration error.

func RegisterOut 

func RegisterOut[OutT any](r *Registry, namespace string, opts ...RegisterOption) (ret RegisteredPropertyOut[OutT], err error)

RegisterOut returns a RegisteredPropertyOut[OutT].

Refer to package documentation for the acceptable types for T.

func (RegisteredPropertyOut[OutT]) MutateOutput 

func (rp RegisteredPropertyOut[OutT]) MutateOutput(ctx context.Context, cb func(OutT) (mutated bool))

MutateOutput calls `cb` under a Mutex with the current value of this output property from the State embedded in `ctx`.

NOTE: The initial value of the output property is always an empty non-nil message/struct. As a convenience, *structpb.Struct types also have their Fields value initialized.

This callback may observe the current value and optionally modify it. If the callback modifies the value, it must return `mutated=true`. If the value is mutated, this will trigger any `notify` callbacks associated with *State.

If you need to carry any portion of T outside of the callback, clone this inside of `cb` (e.g. with proto.Clone for protos, or other mechanisms for ad-hoc structs). Failure to do this will result in data races which `go test -race` may, or may not, reveal.

This will panic if:

  • This RegisteredProperty is uninitialized (i.e. constructed without calling Register).
  • This RegisteredProperty is registered with a different Registry from the one embedded in `ctx`.
  • State is nil.

func (RegisteredPropertyOut[OutT]) MutateOutputFromState 

func (rp RegisteredPropertyOut[OutT]) MutateOutputFromState(s *State, cb func(OutT) (mutated bool))

MutateOutputFromState is the same as MutateOutput, but uses a directly-supplied State instead of getting it from `ctx`.

func (RegisteredPropertyOut[OutT]) SetOutput 

func (rp RegisteredPropertyOut[OutT]) SetOutput(ctx context.Context, newVal OutT)

SetOutput directly sets the value of this output property under a Mutex on the State embedded in `ctx`.

Note that this does not do any cloning/copying of the value - if you need to maintain a mutable referEnce to `newVal` after this function, copy the value before passing it in.

This will panic if:

  • This RegisteredProperty is uninitialized (i.e. constructed without calling Register).
  • This RegisteredProperty is registered with a different Registry from the one embedded in `ctx`.
  • State is nil.

func (RegisteredPropertyOut[OutT]) SetOutputFromState 

func (rp RegisteredPropertyOut[OutT]) SetOutputFromState(s *State, newVal OutT)

SetOutputFromState is the same as SetOutput, but uses a directly-supplied State instead of getting it from `ctx`.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry contains a mapping for all known property namespaces.

You can add to this registry with Register.

Example 
package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"

	"google.golang.org/protobuf/encoding/protojson"
	"google.golang.org/protobuf/types/known/structpb"

	buildbucketpb "go.chromium.org/luci/buildbucket/proto"

	"go.chromium.org/luci/luciexe/build/properties"
)

// MyRegistry is an example properties.Registry.
//
// NOTE: MOST users of `properties` will be using it indirectly via
// the [go.chromium.org/luci/luciexe/build.Properties] Registry, and
// registering properties directly with the functions in that package.
//
// This example, however, is potentially useful for writing tests, etc. where
// you may want to create your own properties.State and put it in the context
// for tests.
//
// The Registry holds all schema-property registration. It is effectively
// append-only, and becomes immutable as soon as the first State is
// Instantiated from it.
var MyRegistry = &properties.Registry{}

type MyTopLevelStruct struct {
	Key      string `json:"key"`
	OtherKey string `json:"okey"`
}

// InOutProp is an example of using a Go struct for a schema.
var InOutProp = properties.MustRegister[*MyTopLevelStruct](MyRegistry, "")

// BuildLikeProp is an example of using a proto Message as a schema.
//
// Typically these non top-level namespaces would be registered in different Go
// packages, similar to using the go "flag" package.
var BuildLikeProp = properties.MustRegister[*buildbucketpb.Build](MyRegistry, "$buildLike")

// PerfCounters is an example of using a raw Go map to have a schema-less
// namespace.
var PerfCounters = properties.MustRegisterOut[map[string]int](MyRegistry, "$perf_out")

func main() {
	rawInputProperties, err := structpb.NewStruct(map[string]any{
		"key":  "hello",
		"okey": "more top level",

		"$buildLike": map[string]any{
			"id": "12345",
		},
	})
	if err != nil {
		panic(err)
	}

	state, err := MyRegistry.Instantiate(context.Background(), rawInputProperties, nil)
	if err != nil {
		panic(err)
	}
	ctx, err := state.SetInContext(context.Background())
	if err != nil {
		panic(err)
	}

	fmt.Printf("%#v\n", InOutProp.GetInput(ctx))

	// protos print as textpb - use >>> <<< to make this clearer in the output.
	fmt.Printf(">>>%s<<<\n", BuildLikeProp.GetInput(ctx))

	// We can mutate the output property - every output property, even those
	// registered as input/output properties, starts as a blank object. If you
	// wish to start the output value as a copy of the input value, you can use
	// something like:
	//
	//   BuildLikeProp.SetOutput(proto.Clone(BuildLikeProp.GetInput(ctx)))
	//
	// Note that the input value should be treated as read-only; It is up to your
	// program to ensure this is the case.
	BuildLikeProp.MutateOutput(ctx, func(b *buildbucketpb.Build) (mutated bool) {
		b.Builder = &buildbucketpb.BuilderID{Bucket: "buck", Builder: "bld"}
		return true
	})

	PerfCounters.SetOutput(ctx, map[string]int{
		"alpha": 20,
		"yeet":  -1,
	})

	// At any point the program can serialize the state out to a proto Struct.
	//
	// Most programs will not need to do this explicitly; For a luciexe build,
	// this will happen roughly any time any registered property for any namespace
	// is set or mutated. The luciexe/build library also imposes a maximum 1 qps
	// rate limit for outgoing changes, so even if properties are rapidly updated,
	// a maximum of one build update will happen per second.
	rawStruct, vers, consistent, err := state.Serialize()
	if err != nil {
		panic(err)
	}
	fmt.Print("--------\n\n")
	fmt.Println("vers:", vers)
	// consistent would be `false` if another goroutine set/mutated an output
	// property while Serialize was running. Each individual registered namespace
	// is self-consistent, but there is no consistency between different
	// namespaces.
	fmt.Println("consistent:", consistent)
	fmt.Print("--------\n\n")

	blob, err := protojson.Marshal(rawStruct)
	if err != nil {
		panic(err)
	}

	var out bytes.Buffer
	if err := json.Indent(&out, blob, "", "  "); err != nil {
		panic(err)
	}
	fmt.Println(out.String())

}

Output:

&properties_test.MyTopLevelStruct{Key:"hello", OtherKey:"more top level"}
>>>id:12345<<<
--------

vers: 2
consistent: true
--------

{
  "$buildLike": {
    "builder": {
      "bucket": "buck",
      "builder": "bld"
    }
  },
  "$perf_out": {
    "alpha": 20,
    "yeet": -1
  },
  "key": "",
  "okey": ""
}

func (*Registry) Instantiate 

func (r *Registry) Instantiate(ctx context.Context, input *structpb.Struct, notify func(version int64)) (*State, error)

Instantiate generates a new State from this Registry.

Input values will be populated from `input`, if provided.

If `notify` is provided, it will be invoked every time an output property in the State changes.

The version will be monotonically increasing - this version harmonizes with the version returned by State.Serialize, which is intended to allow implementations to not process stale notifications.

The callback is invoked OUTSIDE of a mutex, but synchronously with RegisteredProperty.{Set,Mutate}Output. That is, code which calls RegisteredProperty.{Set,Mutate} will block until notify finishes, but if notify itself blocks, this will not block State.Serialize(), nor will it block other calls to RegisteredProperty.{Set,Mutate} of the same RegisteredProperty. Because of this, `notify` should still execute quickly (e.g. pushing something to a non-blocking channel, such as a go.chromium.org/luci/common/sync/dispatcher.Channel). Because it is called outside of a mutex, you may see many notifications with out-of-order version numbers.

This will finalize the registry, preventing any new registrations. It is valid to generate multiple States from one Registry - they will all operate independently. This characteristic is especially useful in tests, because you can create a single Registry with a single set of registered properties, and then generate a new State for each test case.

type State 

type State struct {
	// contains filtered or unexported fields
}

State holds initial and current states for all the namespaces in a given Registry.

There can be multiple State objects for the same Registry, and all RegisteredProperty objects from that Registry will work with any State from that Registry.

Each State maintains it's own `version` (observable via InstOptNotify and State.Serialize). Any set/modify operation to any property namespace (including the top level namespace) in this State will increment the version by 1.

IN PRACTICE, most programs will only use the State embedded in the context.Context. This is set by go.chromium.org/luci/luciexe/bulid.Main or go.chromium.org/luci/luciexe/bulid.Start. However, direct Instantiation of the go.chromium.org/luci/luciexe/bulid.Properties Registry to get a new State for tests is likely going to be useful, which is why this is a public API separate from the build library itself.

func GetState 

func GetState(ctx context.Context) *State

GetState retrieves the State from the context.

If no State is in the context, returns `nil`.

func (*State) Serialize 

func (s *State) Serialize() (ret *structpb.Struct, startVers int64, consistent bool, err error)

Serialize serializes the entire State to a single NEW Struct proto message.

This also returns a version number which is <= to the actual version number of the returned Struct. You can safely compare this version number to the one emitted by `notify` to discard notify events which are less than or equal to this number.

The returned Struct is fully cloned and can be read or written as you need.

Namespaces with no data (i.e. would be an empty JSON object `{}`) are omitted. This includes the top-level document (so - if no values are present in the overall document at all, this returns nil).

The ONLY way this will return an error is if the top-level namespace is either *Struct or a map type, AND the top-level namespace contains a key which overlaps with one of the other output property namespaces.

This function is non-blocking - it will return the version of the State observed at the beginning of Serialize (startVers), and it will return (consistent==true) if the version observed after the construction of `ret` is the same as startVers. If consistent is false, you can choose to continue with the possibly inconsistent `ret`, and just call Serialize again later, or you can discard `ret` and try again. If the various RegisteredProperties mutating this State very rapidly, it's possible that `consistent` will always be false. If you require a consistent serialization and rapid property updates, you may need to introduce some additional form of synchronization between the property mutators to ensure that Serialize can complete with a consistent serialization.

There is NO synchronization other than bumping the version number up between different properties in this state - multiple goroutines can mutate RegisteredPropertyOuts simultaneously. Manipulation of a single RegisteredPropertyOut is fully synchronized with itself, however.

To get this into JSON form, just use protojson.Marshal on the returned *Struct.

func (*State) SetInContext 

func (s *State) SetInContext(ctx context.Context) (context.Context, error)

SetInContext installs the State into context, returning an unmodified context and an error if `ctx` already includes a State.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.