table

package
v0.125.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 11, 2021 License: MIT Imports: 16 Imported by: 2

Documentation

Index

Constants

View Source 
const BufferSize = 1000

BufferSize represents a constant buffer size to be used by flux that buffer data by the number of rows.

This isn't a required size, but a recommended one that can be shared as a constant around the various standard library functions so that they are more likely to not reorganize data.

This number was chosen because it is the same buffer size that the influxdb storage engine uses when buffering table data. In the future, we may want to make it possible for different sources to report their own buffer sizes so influxdb isn't given an unfair advantage just because these constants are set to the same value.

Variables

This section is empty.

Functions

func Copy added in v0.103.0 

func Copy(t flux.Table) (flux.BufferedTable, error)

Copy returns a buffered copy of the table and consumes the input table. If the input table is already buffered, it "consumes" the input and returns the same table.

The buffered table can then be copied additional times using the BufferedTable.Copy method.

This method should be used sparingly if at all. It will retain each of the buffers of data coming out of a table so the entire table is materialized in memory. For large datasets, this could potentially cause a problem. The allocator is meant to catch when this happens and prevent it.

func Diff 

func Diff(want, got flux.TableIterator, opts ...DiffOption) string

Diff will perform a diff between two table iterators. This will sort the tables within the table iterators and produce a diff of the full output.

func FromBuffer added in v0.114.0 

func FromBuffer(cr flux.ColReader) flux.Table

FromBuffer constructs a flux.Table from a single flux.ColReader.

func Sort 

func Sort(tables flux.TableIterator) (flux.TableIterator, error)

Sort will read a TableIterator and produce another TableIterator where the keys are sorted.

This method will buffer all of the data since it needs to ensure all of the tables are read to avoid any deadlocks. Be careful using this method in performance sensitive areas.

func Stringify 

func Stringify(table flux.Table) string

Stringify will read a table and turn it into a human-readable string.

func Values added in v0.114.0 

func Values(cr flux.ColReader, j int) array.Interface

Values returns the array from the column reader as an array.Interface.

Types

type BufferedBuilder added in v0.114.0 

type BufferedBuilder struct {
	GroupKey  flux.GroupKey
	Columns   []flux.ColMeta
	Buffers   []*arrow.TableBuffer
	Allocator memory.Allocator
}

BufferedBuilder is a table builder that constructs a BufferedTable with zero or more buffers.

func GetBufferedBuilder added in v0.114.0 

func GetBufferedBuilder(key flux.GroupKey, cache *BuilderCache) (builder *BufferedBuilder, created bool)

GetBufferedBuilder is a convenience method for retrieving a BufferedBuilder from the BuilderCache.

func NewBufferedBuilder added in v0.114.0 

func NewBufferedBuilder(key flux.GroupKey, mem memory.Allocator) *BufferedBuilder

NewBufferedBuilder constructs a new BufferedBuilder.

func (*BufferedBuilder) AppendBuffer added in v0.114.0 

func (b *BufferedBuilder) AppendBuffer(cr flux.ColReader) error

AppendBuffer will append a new buffer to this table builder. It ensures the schemas are compatible and will backfill previous buffers with nil for new columns that didn't previously exist.

func (*BufferedBuilder) AppendTable added in v0.114.0 

func (b *BufferedBuilder) AppendTable(tbl flux.Table) error

AppendTable will append all of the table buffers inside of a table to this BufferedBuilder.

This method will take care of normalizing the schema in the case where there is an empty table with no buffers.

func (*BufferedBuilder) Release added in v0.114.0 

func (b *BufferedBuilder) Release()

func (*BufferedBuilder) Table added in v0.114.0 

func (b *BufferedBuilder) Table() (flux.Table, error)

type BufferedTable added in v0.114.0 

type BufferedTable struct {
	GroupKey flux.GroupKey
	Columns  []flux.ColMeta
	Buffers  []flux.ColReader
	// contains filtered or unexported fields
}

BufferedTable represents a table of buffered column readers.

func (*BufferedTable) Cols added in v0.114.0 

func (b *BufferedTable) Cols() []flux.ColMeta

func (*BufferedTable) Do added in v0.114.0 

func (b *BufferedTable) Do(f func(flux.ColReader) error) error

func (*BufferedTable) Done added in v0.114.0 

func (b *BufferedTable) Done()

func (*BufferedTable) Empty added in v0.114.0 

func (b *BufferedTable) Empty() bool

func (*BufferedTable) Key added in v0.114.0 

func (b *BufferedTable) Key() flux.GroupKey

type Builder added in v0.114.0 

type Builder interface {
	// Table will construct a Table from the existing contents.
	// Invoking this method should reset the builder and all allocated
	// memory will be owned by the returned flux.Table.
	Table() (flux.Table, error)

	// Release will release the buffered contents from the builder.
	// This method is unnecessary if Table is called.
	Release()
}

Builder is the minimum interface for constructing a Table.

type BuilderCache added in v0.114.0 

type BuilderCache struct {
	// New will be called to construct a new Builder
	// when a GroupKey that hasn't been seen before is
	// requested. The returned Builder should be empty.
	New func(key flux.GroupKey) Builder

	// Tables contains the cached builders.
	// This can be set before use to customize the
	// method for storing data. If this is null,
	// the default execute.GroupLookup is initialized
	// when the cache is first used.
	Tables KeyLookup
}

BuilderCache hold a mapping of group keys to Builder. When a Builder is requested for a specific group key, the BuilderCache will return a Builder that is unique for that GroupKey.

func (*BuilderCache) DiscardTable added in v0.114.0 

func (d *BuilderCache) DiscardTable(key flux.GroupKey)

func (*BuilderCache) ExpireTable added in v0.114.0 

func (d *BuilderCache) ExpireTable(key flux.GroupKey)

func (*BuilderCache) ForEach added in v0.114.0 

func (d *BuilderCache) ForEach(f func(key flux.GroupKey, builder Builder) error) error

func (*BuilderCache) Get added in v0.114.0 

func (d *BuilderCache) Get(key flux.GroupKey, b interface{}) bool

Get retrieves the Builder for this group key. If one doesn't exist, it will invoke the New function and store it within the Builder. If the builder was newly created, this method returns true for the second parameter. The interface must be a pointer to the type that is created from the New method. This method will use reflection to set the value of the pointer.

func (*BuilderCache) Table added in v0.114.0 

func (d *BuilderCache) Table(key flux.GroupKey) (flux.Table, bool, error)

Table will remove a builder from the cache and construct a flux.Table from the buffered contents.

type Chunk added in v0.125.0 

type Chunk struct {
	// contains filtered or unexported fields
}

Chunk is a horizontal partition of a Table. It is a subset of rows and contains a set of columns known as the group key. It may not contain all columns that have been seen associated with that group key so transformations should verify the existence of columns for each chunk independently.

func ChunkFromBuffer added in v0.125.0 

func ChunkFromBuffer(buf arrow.TableBuffer) Chunk

ChunkFromBuffer will create a Chunk from the TableBuffer.

This function takes ownership of the arrow.TableBuffer and the Chunk goes out of scope at the same time as the arrow.TableBuffer unless Retain is called.

func ChunkFromReader added in v0.125.0 

func ChunkFromReader(cr flux.ColReader) Chunk

ChunkFromReader will create a Chunk from the ColReader.

This function borrows a reference to the data in the ColReader and will go out of scope at the same time as the ColReader unless Retain is called.

func (Chunk) Buffer added in v0.125.0 

func (v Chunk) Buffer() arrow.TableBuffer

Buffer returns the underlying TableBuffer used for this Chunk. This is exposed for use by another package, but this method should never be invoked in normal code.

func (Chunk) Col added in v0.125.0 

func (v Chunk) Col(j int) flux.ColMeta

Col returns the metadata associated with the column.

func (Chunk) Cols added in v0.125.0 

func (v Chunk) Cols() []flux.ColMeta

Cols returns the columns as a slice.

func (Chunk) HasCol added in v0.125.0 

func (v Chunk) HasCol(label string) bool

HasCol returns whether a column with the given name exists.

func (Chunk) Index added in v0.125.0 

func (v Chunk) Index(label string) int

Index returns the index of the column with the given name.

func (Chunk) Key added in v0.125.0 

func (v Chunk) Key() flux.GroupKey

Key returns the columns which are common for each row this view.

func (Chunk) Len added in v0.125.0 

func (v Chunk) Len() int

Len returns the number of rows.

func (Chunk) NCols added in v0.125.0 

func (v Chunk) NCols() int

NCols returns the number of columns in this Chunk.

func (Chunk) Release added in v0.125.0 

func (v Chunk) Release()

Release will release a reference to this buffer.

func (Chunk) Retain added in v0.125.0 

func (v Chunk) Retain()

Retain will retain a reference to this Chunk.

func (Chunk) Values added in v0.125.0 

func (v Chunk) Values(j int) array.Interface

Values returns a reference to the array of values in this Chunk. The returned array is a borrowed reference and the caller can call Retain on the returned array to retain its own reference to the array.

type DiffOption 

type DiffOption interface {
	// contains filtered or unexported methods
}

func DiffContext 

func DiffContext(n int) DiffOption

type Iterator 

type Iterator []flux.Table

func (Iterator) Do 

func (t Iterator) Do(f func(flux.Table) error) error

type KeyLookup added in v0.114.0 

type KeyLookup interface {
	// Lookup will retrieve the value associated with the given key if it exists.
	Lookup(key flux.GroupKey) (interface{}, bool)

	// LookupOrCreate will retrieve the value associated with the given key or,
	// if it does not exist, will invoke the function to create one and set
	// it in the group lookup.
	LookupOrCreate(key flux.GroupKey, fn func() interface{}) interface{}

	// Set will set the value for the given key.
	// It will overwrite an existing value.
	Set(key flux.GroupKey, value interface{})

	// Delete will remove the key from this KeyLookup.
	// It will return the same thing as a call to Lookup.
	Delete(key flux.GroupKey) (v interface{}, found bool)

	// Range will iterate over all groups keys in a stable ordering.
	// Range must not be called within another call to Range.
	// It is safe to call Set/Delete while ranging.
	Range(f func(key flux.GroupKey, value interface{}))

	// Clear will clear the lookup and reset it to contain nothing.
	Clear()
}

KeyLookup is an interface for storing and retrieving items by their group key.

type ProfilerResult added in v0.82.0 

type ProfilerResult struct {
	// contains filtered or unexported fields
}

func NewProfilerResult added in v0.82.0 

func NewProfilerResult(tables ...flux.Table) ProfilerResult

func (*ProfilerResult) Name added in v0.82.0 

func (r *ProfilerResult) Name() string

func (*ProfilerResult) Tables added in v0.82.0 

func (r *ProfilerResult) Tables() flux.TableIterator

Source Files

View all Source files

Directories

Path Synopsis
Package static provides utilities for easily constructing static tables that are meant for tests.
 Package static provides utilities for easily constructing static tables that are meant for tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.