sharding

Documentation

Index

• type Algorithm
• type Decorator
  • func NewDecorator(algorithms []Algorithm, rings []*RingConfig) *Decorator
  • func (a *Decorator) AssignShards(ctx context.Context, resource resource.Resource)
• type RingConfig
  • func (*RingConfig) Descriptor() ([]byte, []int)
  • func (m *RingConfig) GetDefaultFieldPath() string
  • func (m *RingConfig) GetFieldPathForLabel(label string) string
  • func (m *RingConfig) GetFieldPaths() []*RingConfig_FieldPath
  • func (m *RingConfig) GetSize() uint64
  • func (*RingConfig) GotenMessage()
  • func (obj *RingConfig) GotenValidate() error
  • func (m *RingConfig) Marshal() ([]byte, error)
  • func (m *RingConfig) MarshalJSON() ([]byte, error)
  • func (*RingConfig) ProtoMessage()
  • func (m *RingConfig) ProtoReflect() preflect.Message
  • func (m *RingConfig) Reset()
  • func (m *RingConfig) SetFieldPaths(fv []*RingConfig_FieldPath)
  • func (m *RingConfig) SetSize(fv uint64)
  • func (m *RingConfig) String() string
  • func (m *RingConfig) Unmarshal(b []byte) error
  • func (m *RingConfig) UnmarshalJSON(data []byte) error
• type RingConfig_FieldPath
  • func (*RingConfig_FieldPath) Descriptor() ([]byte, []int)
  • func (m *RingConfig_FieldPath) GetLabel() string
  • func (m *RingConfig_FieldPath) GetPath() string
  • func (*RingConfig_FieldPath) GotenMessage()
  • func (obj *RingConfig_FieldPath) GotenValidate() error
  • func (m *RingConfig_FieldPath) Marshal() ([]byte, error)
  • func (m *RingConfig_FieldPath) MarshalJSON() ([]byte, error)
  • func (*RingConfig_FieldPath) ProtoMessage()
  • func (m *RingConfig_FieldPath) ProtoReflect() preflect.Message
  • func (m *RingConfig_FieldPath) Reset()
  • func (m *RingConfig_FieldPath) SetLabel(fv string)
  • func (m *RingConfig_FieldPath) SetPath(fv string)
  • func (m *RingConfig_FieldPath) String() string
  • func (m *RingConfig_FieldPath) Unmarshal(b []byte) error
  • func (m *RingConfig_FieldPath) UnmarshalJSON(data []byte) error

Types

type Algorithm

type Algorithm interface {
	Label() string
	Shardable(resource.Resource) bool
	ComputeHash(context.Context, resource.Resource) (uint64, bool)
}

type Decorator

type Decorator struct {
	// contains filtered or unexported fields
}

func NewDecorator

func NewDecorator(algorithms []Algorithm, rings []*RingConfig) *Decorator

func (*Decorator) AssignShards

func (a *Decorator) AssignShards(ctx context.Context, resource resource.Resource)

type RingConfig

type RingConfig struct {

	// Ring size. It affects possible shard numbers assigned to all relevant resources, which are in range: [0:size).
	Size uint64 `protobuf:"varint,1,opt,name=size,proto3" json:"size,omitempty" firestore:"size"`
	// Field path in the resource where shard number is stored, for example "meta.shards.byProjectIdMod4096". Expected
	// underlying type is int64.
	FieldPaths []*RingConfig_FieldPath `protobuf:"bytes,2,rep,name=field_paths,json=fieldPaths,proto3" json:"field_paths,omitempty" firestore:"fieldPaths"`
	// contains filtered or unexported fields
}

RingConfig is an information about single shard range for one specific purpose on a set of resources. Example:

{
  size: 4096
  field_paths: [
   { path: "meta.shards.byProjectIdMod4096", label: "byProjectId" }
  ]
}

In the example above, we indicate that some set of resources (in this case, those that have project ID specified) are shardable and their respective shard IDs are stored in the field path with label "byProjectId". Also, size of the ring describing this shard range is equal to 4096. Labels are useful when there is more than one field path for given ring. This can happen, if we have two resources which must be within same ring, but have different paths. Or another case, when one resource instance belongs to two different rings of the same type -> for example, IAM/Group resources are bucketed into rings by their identifiers and, if one group belongs to another, their retrospective GroupMember resources are visible in two rings. In other words, resources that form a graph and resources "on the borders" should be visible in both groups. Notes: * Path pointed by FieldPath must be of type int64. Also map<..., int64> is also supported (meta.shards) can be a map. RingConfig in the controller config must be in sync with at least one RingConfig in server side config. Server side config can use multiple rings at once. * While controller uses just one RingConfig at the time, server can use many simultaneously. It is required that labelling across array of RingConfigs is consistent. * Common components for sharding offer ShardDecorator. It is a generic component that decorates each relevant resource with shard ID. It handles common logic. While developer is still responsible for providing algorithm to it, calculation of the actual shard ID (combination of hash + ring size), iterating over all ring configs to provide number for all the rings is done there. This component however requires list of algorithms where each algorithm must tell, via Label, which field path intends to write.

func (*RingConfig) Descriptor

func (*RingConfig) Descriptor() ([]byte, []int)

Deprecated, Use RingConfig.ProtoReflect.Descriptor instead.

func (*RingConfig) GetDefaultFieldPath

func (m *RingConfig) GetDefaultFieldPath() string

func (*RingConfig) GetFieldPathForLabel

func (m *RingConfig) GetFieldPathForLabel(label string) string

func (*RingConfig) GetFieldPaths

func (m *RingConfig) GetFieldPaths() []*RingConfig_FieldPath

func (*RingConfig) GetSize

func (m *RingConfig) GetSize() uint64

func (*RingConfig) GotenMessage

func (*RingConfig) GotenMessage()

func (*RingConfig) GotenValidate

func (obj *RingConfig) GotenValidate() error

func (*RingConfig) Marshal

func (m *RingConfig) Marshal() ([]byte, error)

func (*RingConfig) MarshalJSON

func (m *RingConfig) MarshalJSON() ([]byte, error)

func (*RingConfig) ProtoMessage

func (*RingConfig) ProtoMessage()

func (*RingConfig) ProtoReflect

func (m *RingConfig) ProtoReflect() preflect.Message

func (*RingConfig) Reset

func (m *RingConfig) Reset()

func (*RingConfig) SetFieldPaths

func (m *RingConfig) SetFieldPaths(fv []*RingConfig_FieldPath)

func (*RingConfig) SetSize

func (m *RingConfig) SetSize(fv uint64)

func (*RingConfig) String

func (m *RingConfig) String() string

func (*RingConfig) Unmarshal

func (m *RingConfig) Unmarshal(b []byte) error

func (*RingConfig) UnmarshalJSON

func (m *RingConfig) UnmarshalJSON(data []byte) error

type RingConfig_FieldPath

type RingConfig_FieldPath struct {

	// Path in the resource where shard number is stored.
	Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty" firestore:"path"`
	// Label identifying field path.
	Label string `protobuf:"bytes,2,opt,name=label,proto3" json:"label,omitempty" firestore:"label"`
	// contains filtered or unexported fields
}

func (*RingConfig_FieldPath) Descriptor

func (*RingConfig_FieldPath) Descriptor() ([]byte, []int)

Deprecated, Use RingConfig_FieldPath.ProtoReflect.Descriptor instead.

func (*RingConfig_FieldPath) GetLabel

func (m *RingConfig_FieldPath) GetLabel() string

func (*RingConfig_FieldPath) GetPath

func (m *RingConfig_FieldPath) GetPath() string

func (*RingConfig_FieldPath) GotenMessage

func (*RingConfig_FieldPath) GotenMessage()

func (*RingConfig_FieldPath) GotenValidate

func (obj *RingConfig_FieldPath) GotenValidate() error

func (*RingConfig_FieldPath) Marshal

func (m *RingConfig_FieldPath) Marshal() ([]byte, error)

func (*RingConfig_FieldPath) MarshalJSON

func (m *RingConfig_FieldPath) MarshalJSON() ([]byte, error)

func (*RingConfig_FieldPath) ProtoMessage

func (*RingConfig_FieldPath) ProtoMessage()

func (*RingConfig_FieldPath) ProtoReflect

func (m *RingConfig_FieldPath) ProtoReflect() preflect.Message

func (*RingConfig_FieldPath) Reset

func (m *RingConfig_FieldPath) Reset()

func (*RingConfig_FieldPath) SetLabel

func (m *RingConfig_FieldPath) SetLabel(fv string)

func (*RingConfig_FieldPath) SetPath

func (m *RingConfig_FieldPath) SetPath(fv string)

func (*RingConfig_FieldPath) String

func (m *RingConfig_FieldPath) String() string

func (*RingConfig_FieldPath) Unmarshal

func (m *RingConfig_FieldPath) Unmarshal(b []byte) error

func (*RingConfig_FieldPath) UnmarshalJSON

func (m *RingConfig_FieldPath) UnmarshalJSON(data []byte) error