fieldmask

package
v0.3.12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 30, 2024 License: Apache-2.0 Imports: 12 Imported by: 0

README

Thrift FieldMask RFC

What is thrift fieldmask?

FieldMask is inspired by Protobuf and used to indicates the data that users care about, and filter out useless data, during a RPC call, in order to reducing network package size and accelerating serializing/deserializing process. This tech has been widely used among Protobuf services.

How to construct a fieldmask?

To construct a fieldmask, you need two things:

  • Thrift Path for describing the data you want
  • Type Descriptor for validating the thrift path you pass is compatible with thrift message definition (IDL)
Thrift Path
What is thrift path?

A path string represents a arbitrary endpoint of thrift object. It is used for locating data from thrift root message, and defined from-top-to-bottom. For exapmle, a thrift message defined as below:

struct Example {
    1: string Foo,
    2: i64 Bar
    3: Example Self
}

A thrift path $.Foo represents the string value of Example.Foo, and $.Self.Bar represents the secondary layer i64 value of Example.Self.Bar Since thrift has four nesting types (LIST/SET/MAP/STRUCT), thrift path should also support locating elements in all these types' object, not only STRUCT.

Syntax

Here are basic hypothesis:

  • fieldname is the field name of a field in a struct, it MUST ONLY contain '[a-zA-Z]' alphabet letters, integer numbers and char '_'.
  • index is the index of a element in a list or set, it MUST ONLY contain integer numbers.
  • key is the string-typed key of a element in a map, it can contain any letters, but it MUST be a quoted string.
  • id is the integer-typed key of a element in a map, it MUST ONLY contain integer numbers.
  • except key, ThriftPath shouldn't contains any blank chars (\n\r\b\t).

Here is detailed syntax:

ThriftPath Description
$ the root object,every path must start with it.
.fieldname get the child field of a struct corepsonding to fieldname. For example, $.FieldA.ChildrenB
[index,index...] get any number of elements in an List/Set corepsonding to indices. Indices must be integer.For example: $.FieldList[1,3,4] .Notice: a index beyond actual list size can written but is useless.
{"key","key"...} get any number of values corepsonding to key in a string-typed-key map. For example: $.StrMap{"abcd","1234"}
{id,id...} get the child field with specific id in a integer-typed-key map. For example, $.IntMap{1,2}
* get ALL fields/elements, that is: $.StrMap{*}.FieldX menas gets all the elements' FieldX in a map Root.StrMap; $.List[*].FieldX means get all the elements' FieldX in a list Root.List.
Agreement Of Implementation
  • A empty mask means "PASS ALL" (all field is "PASS")
  • For map of neither-string-nor-integer typed key, only syntax token of all '*' (see above) is allowed in.
  • For safty, required fields which are not in mask ("Filtered") will still be written into message:
    • by default, write current value of the required field;
    • add generate option field_mask_zero_required: write zero value of the required field
  • FieldMask settings must start from the root object.
    • Tips: If you want to set FieldMask from a non-root object and make it effective, you need to add field_mask_halfway option and regenerate the codes. However, there is a latent risk: if different parent objects reference the same child object, and these two parent objects set different fieldmasks, only one parent object's fieldmask relative to this child object will be effective.
Visibility

By default, a field in mask means "PASS" (will be serialized/deserialized), and the other fields not in mask means "REJECT" (won't be serialized/deserialized) -- which is so-called "White List" However, we allow user to use fieldmask as a "Black List", as long as enable option Options.BlackList mode. Under such mode, a field in the mask means "REJECT", and the other fields means "PASS". See main_test.go for detailed usage.

Type Descriptor

Type descriptor is the runtime representation of a message definition, in aligned with Protobuf Descriptor. To get a type descriptor, you must enable thrift reflection feature first, which was introduced in thriftgo v0.3.0. you can generate related codes for this feature using option with_reflection.

How to use fieldmask?

  1. First, you must generates codes for this feature using two options with_fieldmask and with_reflection
$ thriftgo -g with_field_mask,with_reflection ${your_idl}
  1. Create a fieldmask in the initializing phase of your application (recommanded), or just in the bizhandler before you return a response
import (
	"sync"
	"github.com/cloudwego/thriftgo/fieldmask"
	nbase "github.com/cloudwego/thriftgo/test/golang/fieldmask/gen-new/base"
)

var fieldmaskCache sync.Map

func init() {
	// new a obj to get its TypeDescriptor
	obj := nbase.NewBase()
    desc := obj.GetTypeDescriptor()

	// construct a fieldmask with TypeDescriptor and thrift pathes
	fm, err := fieldmask.NewFieldMask(desc,
		"$.Addr", "$.LogID", "$.TrafficEnv.Code", "$.Meta.IntMap{1}", "$.Meta.StrMap{\"1234\"}", "$.Meta.List[1]", "$.Meta.Set[1]")
	if err != nil {
		panic(err)
	}

	// cache it for future usage of nbase.Base
	fieldmaskCache.Store("Mask1ForBase", fm)
}
  • If you want to enable black-list mode of fieldmask, you can create fieldmask like this:
    fm, err := fieldmask.Options{
        BlackListMode: true,
    }.NewFieldMask(desc, "$.Addr")
  1. Now you can use fieldmask in either client-side or server-side
  • For server-side, you can set fieldmask with generated API Set_FieldMask() on your response object. Then the object itself will notice the fieldmask and using it during its serialization
func bizHandler(req any) (*nbase.Base) {
  // handle request ...

  // biz logic: handle and get final response object
  obj := bizBase()

  // Load fieldmask from cache
  fm, _ := fieldmaskCache.Load("Mask1ForBase")
  if fm != nil {
  	// load ok, set fieldmask onto the object using codegen API
  	obj.Set_FieldMask(fm.(*fieldmask.FieldMask))
  }

  return obj
}
  • For client-side: related to the deserialization process of framework. For kitex, it's WIP.

How to pass fieldmask between programs?

Generally, you can add one binary field on your request definition to carry fieldmask, and explicitly serialize/deserialize the fieldmask you are using into/from this field. We provide two encapsulated API for serialization/deserialization:

Benchmark

See (main_test.go)

goos: darwin
goarch: amd64
pkg: github.com/cloudwego/thriftgo/test/golang/fieldmask
cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
BenchmarkWriteWithFieldMask/old-16         	     2188 ns/op	       0 B/op	       0 allocs/op
BenchmarkWriteWithFieldMask/new-16         	     2281 ns/op	       0 B/op	       0 allocs/op
BenchmarkWriteWithFieldMask/new-mask-half-16     1055 ns/op	       0 B/op	       0 allocs/op
BenchmarkReadWithFieldMask/old-16                6187 ns/op	    2124 B/op	      41 allocs/op
BenchmarkReadWithFieldMask/new-16                5675 ns/op	    2268 B/op	      41 allocs/op
BenchmarkReadWithFieldMask/new-mask-half-16      4762 ns/op	    1564 B/op	      31 allocs/op

Explain case names:

  • Write: serialization test
  • Read: deserializtion test
  • old: not generate with_fieldmask API
  • new: generate with_fieldmask API, but not use fieldmask
  • new-mask-half: generate with_fieldmask API and use fieldmask to mask half of the data

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Marshal added in v0.3.4 

func Marshal(fm *FieldMask) ([]byte, error)

Marshal serializes a fieldmask into bytes.

Notice: This API uses cache to accelerate processing, at the cost of increasing memory usage

Types

type FieldMask 

type FieldMask struct {
	// contains filtered or unexported fields
}

FieldMask represents a collection of thrift pathes See

func NewFieldMask 

func NewFieldMask(desc *thrift_reflection.TypeDescriptor, pathes ...string) (*FieldMask, error)

NewFieldMask create a new fieldmask

func Unmarshal added in v0.3.4 

func Unmarshal(data []byte) (*FieldMask, error)

Marshal deserializes a fieldmask from bytes.

Notice: This API uses cache to accelerate processing, at the cost of increasing memory usage

func (*FieldMask) All 

func (self *FieldMask) All() bool

All tells if the mask covers all elements (* or empty or scalar)

func (*FieldMask) Exist 

func (self *FieldMask) Exist() bool

Exist tells if the fieldmask is setted

func (*FieldMask) Field 

func (self *FieldMask) Field(id int16) (*FieldMask, bool)

Field returns the specific sub mask for a given id, and tells if the id in the mask

func (*FieldMask) ForEachChild added in v0.3.11 

func (self *FieldMask) ForEachChild(scanner func(strKey string, intKey int, child *FieldMask) bool)

ForEachChild iterates over underlying children of a FieldMask (if any). A child's key type depends on the type of the FieldMask:

  • FtStruct | FtList | FtIntMap: intKey
  • FtStrMap: strKey

func (*FieldMask) GetPath added in v0.3.11 

func (cur *FieldMask) GetPath(desc *thrift_reflection.TypeDescriptor, path string) (*FieldMask, bool)

getPathAncestor tells if a given path is in current fieldmask, and return the nearest settled ancestor (include itself)

func (*FieldMask) Int 

func (self *FieldMask) Int(id int) (*FieldMask, bool)

Int returns the specific sub mask for a given index, and tells if the index in the mask

func (*FieldMask) IsBlack added in v0.3.11 

func (self *FieldMask) IsBlack() bool

IsBlack tells if the FieldMask is black-list or white-list

func (*FieldMask) MarshalJSON added in v0.3.4 

func (fm *FieldMask) MarshalJSON() ([]byte, error)

MarshalJSON marshals the fieldmask into json.

For example:

  • pathes `[]string{"$.Extra[0].List", "$.Extra[*].Set", "$.Meta.F2{0}", "$.Meta.F2{*}.Addr"}` will produces:
  • `{"path":"$","type":"Struct","children":[{"path":6,"type":"List","children":[{"path":"*","type":"Struct","children":[{"path":4,"type":"List"}]}]},{"path":256,"type":"Struct","children":[{"path":2,"type":"IntMap","children":[{"path":"*","type":"Struct","children":[{"path":0,"type":"Scalar"}]}]}]}]}`

For details:

  • `path` is the path segment of current fieldmask layer
  • `type` is the `FieldMaskType` of the fieldmask -`children` is the chidlren of subsequent pathes
  • each fieldmask always starts with root path "$"
  • path "*" indicates all subsequent path of the fieldmask shares the same sub fieldmask

func (*FieldMask) PathInMask 

func (fm *FieldMask) PathInMask(desc *thrift_reflection.TypeDescriptor, path string) bool

PathInMask tells if a given path is already in current fieldmask

func (*FieldMask) Str 

func (self *FieldMask) Str(id string) (*FieldMask, bool)

Field returns the specific sub mask for a given string, and tells if the string in the mask

func (FieldMask) String 

func (self FieldMask) String(desc *thrift_reflection.TypeDescriptor) string

String pretty prints the structure a FieldMask represents

WARING: This is unstable API, the printer format is not guaranteed

func (*FieldMask) TransferFrom added in v0.3.11 

func (self *FieldMask) TransferFrom(s *fieldMaskTransfer) error

TransferFrom transfroms a FieldMaskTransfer to the FieldMask

func (*FieldMask) Type added in v0.3.11 

func (self *FieldMask) Type() FieldMaskType

Type tells its FieldMaskType, which is decided by the Thrift Path definition.

func (*FieldMask) UnmarshalJSON added in v0.3.4 

func (self *FieldMask) UnmarshalJSON(in []byte) error

UnmarshalJSON unmarshal the fieldmask from json. 

Input JSON **MUST** be according to the schema of `FieldMask.MarshalJSON()`

type FieldMaskType added in v0.3.4 

type FieldMaskType uint8

FieldMaskType indicates the corresponding thrift message type for a fieldmask 

const (
	// Invalid or unsupported thrift type
	FtInvalid FieldMaskType = iota
	// thrift scalar types, including BOOL/I8/I16/I32/I64/DOUBLE/STRING/BINARY, or neither-string-nor-integer-typed-key MAP
	FtScalar
	// thrift LIST/SET
	FtList
	// thrift STRUCT
	FtStruct
	// thrift MAP with string-typed key
	FtStrMap
	// thrift MAP with integer-typed key
	FtIntMap
)

FieldMaskType Enums

func (FieldMaskType) MarshalText added in v0.3.4 

func (ft FieldMaskType) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler

func (*FieldMaskType) UnmarshalText added in v0.3.4 

func (ft *FieldMaskType) UnmarshalText(in []byte) error

UnmarshalText implements encoding.TextUnmarshaler

type Options added in v0.3.11 

type Options struct {
	// BlackListMode enables black-list mode when create FieldMask,
	// which means `Field()/Str()/Int()` will return false for a **Complete** Path in the FieldMask
	BlackListMode bool
}

Options for creating FieldMask

func (Options) NewFieldMask added in v0.3.11 

func (opts Options) NewFieldMask(desc *thrift_reflection.TypeDescriptor, pathes ...string) (*FieldMask, error)

NewFieldMask create a new fieldmask with options

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.