schema

Documentation

Index

• Constants
• Variables
• func Metadata(keys ...MetadataKey) arrow.Metadata
• func NewFieldFrom(prototype *arrow.Field, transformNode *TransformNode) *arrow.Field
• func NewSchemaFrom(prototype *arrow.Schema, transformTree *TransformNode) *arrow.Schema
• type FieldTransform
• type MetadataKey
• type TransformNode
  • func NewTransformTreeFrom(prototype *arrow.Schema, dictConfig *cfg.Dictionary, ...) (*TransformNode, map[string]*transform2.DictionaryField)
  • func (t *TransformNode) RemoveOptional()
  • func (t *TransformNode) RevertCounters()

Constants

const (
	Optional MetadataKey = iota
	Dictionary8
	Dictionary16
	DeltaEncoding

	OptionalKey   = "#optional"
	DictionaryKey = "#dictionary"
	EncodingKey   = "encoding"

	DeltaEncodingValue = "delta"
)

Variables

var (
	ErrSchemaNotUpToDate = errors.New("schema not up to date")
)

Functions

func Metadata

func Metadata(keys ...MetadataKey) arrow.Metadata

Metadata returns a map of Arrow metadata for the given metadata keys.

func NewFieldFrom

func NewFieldFrom(prototype *arrow.Field, transformNode *TransformNode) *arrow.Field

NewFieldFrom creates a new field from a prototype field and a transformation tree.

func NewSchemaFrom

func NewSchemaFrom(prototype *arrow.Schema, transformTree *TransformNode) *arrow.Schema

NewSchemaFrom creates a new schema from a prototype schema and a transformation tree.

Types

type FieldTransform

type FieldTransform interface {
	Transform(field *arrow.Field) *arrow.Field
	RevertCounters()
}

FieldTransform is an interface to apply a transformation to a field.

type MetadataKey

type MetadataKey int

type TransformNode

type TransformNode struct {
	Children []*TransformNode
	// contains filtered or unexported fields
}

TransformNode is a node in a transformation tree. It can be a leaf node or a node with children.

func NewTransformTreeFrom

func NewTransformTreeFrom(
	prototype *arrow.Schema,
	dictConfig *cfg.Dictionary,
	schemaUpdateRequest *update.SchemaUpdateRequest,
	events *events.Events,
) (*TransformNode, map[string]*transform2.DictionaryField)

NewTransformTreeFrom creates a transformation tree from a prototype schema. The dictIndexType and the field metadata are used to transform the prototype schema into the target schema.

Optional fields: By default all fields marked as optional in the prototype schema are removed from the target schema. This behavior can be changed if data is available for this field.

Dictionary fields: By default all fields marked as dictionary fields in the prototype schema are converted to their dictionary representation. This behavior can be changed if the number of unique values is higher than the size of dictIndexType. If dictIndexType is nil, then fields marked as dictionary fields are not converted to their dictionary representation.

func (*TransformNode) RemoveOptional

func (t *TransformNode) RemoveOptional()

RemoveOptional removes all transformations that are marked as optional. This will take effect on the next cycle of appending data.

To avoid data loss, the methods `AppendNonZero[XYZ]` should be called again after calling this method.

func (*TransformNode) RevertCounters

func (t *TransformNode) RevertCounters()