computer

Documentation

Overview

Package computer contains the compute-specific portions of the DAX architecture. In general, this is a dumb FeatureBase node (or service) which essentially contains the Executor and its interaction with the underlying data.

Index

• Constants
• func MarshalLogMessage(msg LogMessage, encode string) ([]byte, error)
• func NewNopSnapshotterService() *nopSnapshotterService
• func NewNopWritelogService() *nopWritelogService
• type FieldKeyMap
• type ImportMessage
• type ImportRoaringMessage
• type ImportRoaringShardMessage
• type ImportValueMessage
• type LogMessage
  • func UnmarshalLogMessage(b []byte) (LogMessage, error)
• type PartitionKeyMap
• type Registrar
• type RoaringUpdate
• type SnapInfo
• type SnapshotService
• type WriteLogInfo
• type WritelogService

Constants

const (
	EncodeTypeJSON string = "json"
)

Functions

func MarshalLogMessage

func MarshalLogMessage(msg LogMessage, encode string) ([]byte, error)

MarshalLogMessage serializes the log message and prepends additional encoding information to each message. Currently, we prepend three bytes to each log message: byte[0]: encodeVersion - this is currently a constant within the code. If we modify structs such that they encode differently, we'll have to change the constant and keep previous versions of structs for deserialization. byte[1]: encodeType (e.g. "json", etc.) byte[2]: logMessageType

If we get into a situation where we want more flexibility in these message header bytes—for example, if we want to use more than three bytes—we could do something with the first bit of the encodeVersion: if it's 1, that could indicate that there are additional header bytes, and the following seven bits could indicate how many.

func NewNopSnapshotterService

func NewNopSnapshotterService() *nopSnapshotterService

func NewNopWritelogService

func NewNopWritelogService() *nopWritelogService

Types

type FieldKeyMap

type FieldKeyMap struct {
	TableKey   dax.TableKey      `json:"table-key"`
	Field      dax.FieldName     `json:"field"`
	StringToID map[string]uint64 `json:"string-to-id"`
}

type ImportMessage

type ImportMessage struct {
	LogMessage `json:"-"`

	Table      string   `json:"table"`
	Field      string   `json:"field"`
	Partition  int      `json:"partition"`
	Shard      uint64   `json:"shard"`
	RowIDs     []uint64 `json:"row-ids"`
	ColumnIDs  []uint64 `json:"column-ids"`
	RowKeys    []string `json:"row-keys"`
	ColumnKeys []string `json:"column-keys"`
	Timestamps []int64  `json:"timestamps"`
	Clear      bool     `json:"clear"`

	// options
	IgnoreKeyCheck bool `json:"ignore-key-check"`
	Presorted      bool `json:"presorted"`
}

type ImportRoaringMessage

type ImportRoaringMessage struct {
	LogMessage `json:"-"`

	Table           string            `json:"table"`
	Field           string            `json:"field"`
	Partition       int               `json:"partition"`
	Shard           uint64            `json:"shard"`
	Clear           bool              `json:"clear"`
	Action          string            `json:"action"` // [set, clear, overwrite]
	Block           int               `json:"block"`
	Views           map[string][]byte `json:"views"`
	UpdateExistence bool              `json:"update-existence"`
}

type ImportRoaringShardMessage

type ImportRoaringShardMessage struct {
	LogMessage `json:"-"`

	Table     string          `json:"table"`
	Partition int             `json:"partition"`
	Shard     uint64          `json:"shard"`
	Views     []RoaringUpdate `json:"views"`
}

type ImportValueMessage

type ImportValueMessage struct {
	LogMessage `json:"-"`

	Table           string      `json:"table"`
	Field           string      `json:"field"`
	Partition       int         `json:"partition"`
	Shard           uint64      `json:"shard"`
	ColumnIDs       []uint64    `json:"column-ids"`
	ColumnKeys      []string    `json:"column-keys"`
	Values          []int64     `json:"values"`
	FloatValues     []float64   `json:"float-values"`
	TimestampValues []time.Time `json:"timestamp-values"`
	StringValues    []string    `json:"string-values"`
	Clear           bool        `json:"clear"`

	// options
	IgnoreKeyCheck bool `json:"ignore-key-check"`
	Presorted      bool `json:"presorted"`
}

type LogMessage

type LogMessage interface{}

LogMessage is implemented by a variety of types which can be serialized as messages to the Writelogger.

func UnmarshalLogMessage

func UnmarshalLogMessage(b []byte) (LogMessage, error)

UnmarshalLogMessage deserializes the log message based on the log message type info.

type PartitionKeyMap

type PartitionKeyMap struct {
	TableKey   dax.TableKey      `json:"table-key"`
	Partition  dax.PartitionNum  `json:"partition"`
	StringToID map[string]uint64 `json:"string-to-id"`
}

type Registrar

type Registrar interface {
	RegisterNode(ctx context.Context, node *dax.Node) error
	CheckInNode(ctx context.Context, node *dax.Node) error
}

Registrar represents the methods which Computer uses to register itself with the Controller.

type RoaringUpdate

type RoaringUpdate struct {
	Field        string `json:"field"`
	View         string `json:"view"`
	Clear        []byte `json:"clear"`
	Set          []byte `json:"set"`
	ClearRecords bool   `json:"clear-records"`
}

RoaringUpdate is identical to featurebase.RoaringUpdate, but we can't import it due to import cycles. TODO featurebase top level shouldn't import dax stuff... all the types it needs should just be in the top level.

type SnapInfo

type SnapInfo struct {
	Version int
}

SnapInfo holds metadata about a snapshot.

type SnapshotService

type SnapshotService interface {
	Read(bucket string, key string, version int) (io.ReadCloser, error)
	Write(bucket string, key string, version int, rc io.ReadCloser) error
	WriteTo(bucket string, key string, version int, wrTo io.WriterTo) error
	List(bucket, key string) ([]SnapInfo, error)
}

SnapshotService represents the SnapshotService methods which Computer uses. These are typically implemented by both the Snapshotter client.

type WriteLogInfo

type WriteLogInfo SnapInfo

WriteLogInfo holds metadata about a write log.

type WritelogService

type WritelogService interface {
	AppendMessage(bucket string, key string, version int, msg []byte) error
	LogReader(bucket string, key string, version int) (io.ReadCloser, error)
	LogReaderFrom(bucket string, key string, version int, offset int) (io.ReadCloser, error)
	DeleteLog(bucket string, key string, version int) error
	List(bucket, key string) ([]WriteLogInfo, error)
	Lock(bucket, key string) error
	Unlock(bucket, key string) error
}

WritelogService represents the WritelogService methods which Computer uses. These are typically implemented by the Writelogger client.