README ¶
table
table provides a DataTable / DataFrame structure similar to pandas and xarray in Python, and Apache Arrow Table, using tensor n-dimensional columns aligned by common outermost row dimension.
See examples/dataproc for a demo of how to use this system for data analysis, paralleling the example in Python Data Science using pandas, to see directly how that translates into this framework.
As a general convention, it is safest, clearest, and quite fast to access columns by name instead of index (there is a map that caches the column indexes), so the base access method names generally take a column name argument, and those that take a column index have an Index
suffix. In addition, we use the Try
suffix for versions that return an error message. It is a bit painful for the writer of these methods but very convenient for the users.
The following packages are included:
-
bitslice is a Go slice of bytes
[]byte
that has methods for setting individual bits, as if it was a slice of bools, while being 8x more memory efficient. This is used for encoding null entries inetensor
, and as a Tensor of bool / bits there as well, and is generally very useful for binary (boolean) data. -
etensor is a Tensor (n-dimensional array) object.
etensor.Tensor
is an interface that applies to many different type-specific instances, such asetensor.Float32
. A tensor is just aetensor.Shape
plus a slice holding the specific data type. Our tensor is based directly on the Apache Arrow project's tensor, and it fully interoperates with it. Arrow tensors are designed to be read-only, and we needed some extra support to make ouretable.Table
work well, so we had to roll our own. Our tensors also interoperate fully with Gonum's 2D-specific Matrix type for the 2D case. -
etable has the
etable.Table
DataTable / DataFrame object, which is useful for many different data analysis and database functions, and also for holding patterns to present to a neural network, and logs of output from the models, etc. Aetable.Table
is just a slice ofetensor.Tensor
columns, that are all aligned along the outer-most row dimension. Index-based indirection, which is essential for efficient Sort, Filter etc, is provided by theetable.IndexView
type, which is an indexed view into a Table. All data processing operations are defined on the IndexView. -
eplot provides an interactive 2D plotting GUI in GoGi for Table data, using the gonum plot plotting package. You can select which columns to plot and specify various basic plot parameters.
-
tensorcore provides an interactive tabular, spreadsheet-style GUI using GoGi for viewing and editing
etable.Table
andetable.Tensor
objects. Thetensorcore.TensorGrid
also provides a colored grid display higher-dimensional tensor data. -
agg provides standard aggregation functions (
Sum
,Mean
,Var
,Std
etc) operating overetable.IndexView
views of Table data. It also defines standardAggFunc
functions such asSumFunc
which can be used forAgg
functions on either a Tensor or IndexView. -
tsragg provides the same agg functions as in
agg
, but operating on all the values in a givenTensor
. Because of the indexed, row-based nature of tensors in a Table, these are not the same as theagg
functions. -
split supports splitting a Table into any number of indexed sub-views and aggregating over those (i.e., pivot tables), grouping, summarizing data, etc.
-
metric provides similarity / distance metrics such as
Euclidean
,Cosine
, orCorrelation
that operate on slices of[]float64
or[]float32
. -
simat provides similarity / distance matrix computation methods operating on
etensor.Tensor
oretable.Table
data. TheSimMat
type holds the resulting matrix and labels for the rows and columns, which has a specialSimMatGrid
view inetview
for visualizing labeled similarity matricies. -
pca provides principal-components-analysis (PCA) and covariance matrix computation functions.
-
clust provides standard agglomerative hierarchical clustering including ability to plot results in an eplot.
-
minmax is home of basic Min / Max range struct, and
norm
has lots of good functions for computing standard norms and normalizing vectors. -
utils has various table-related utility command-line utility tools, including
etcat
which combines multiple table files into one file, including option for averaging column data.
Cheat Sheet
et
is the etable pointer variable for examples below:
Table Access
Scalar columns:
val := et.Float("ColName", row)
str := et.StringValue("ColName", row)
Tensor (higher-dimensional) columns:
tsr := et.Tensor("ColName", row) // entire tensor at cell (a row-level SubSpace of column tensor)
val := et.TensorFloat1D("ColName", row, cellidx) // idx is 1D index into cell tensor
Set Table Value
et.SetFloat("ColName", row, val)
et.SetString("ColName", row, str)
Tensor (higher-dimensional) columns:
et.SetTensor("ColName", row, tsr) // set entire tensor at cell
et.SetTensorFloat1D("ColName", row, cellidx, val) // idx is 1D index into cell tensor
Find Value(s) in Column
Returns all rows where value matches given value, in string form (any number will convert to a string)
rows := et.RowsByString("ColName", "value", etable.Contains, etable.IgnoreCase)
Other options are etable.Equals
instead of Contains
to search for an exact full string, and etable.UseCase
if case should be used instead of ignored.
Index Views (Sort, Filter, etc)
The IndexView provides a list of row-wise indexes into a table, and Sorting, Filtering and Splitting all operate on this index view without changing the underlying table data, for maximum efficiency and flexibility.
ix := etable.NewIndexView(et) // new view with all rows
Sort
ix.SortColumnName("Name", etable.Ascending) // etable.Ascending or etable.Descending
SortedTable := ix.NewTable() // turn an IndexView back into a new Table organized in order of indexes
or:
nmcl := et.ColumnByName("Name") // nmcl is an etensor of the Name column, cached
ix.Sort(func(t *Table, i, j int) bool {
return nmcl.StringValue1D(i) < nmcl.StringValue1D(j)
})
Filter
nmcl := et.ColumnByName("Name") // column we're filtering on
ix.Filter(func(t *Table, row int) bool {
// filter return value is for what to *keep* (=true), not exclude
// here we keep any row with a name that contains the string "in"
return strings.Contains(nmcl.StringValue1D(row), "in")
})
Splits ("pivot tables" etc), Aggregation
Create a table of mean values of "Data" column grouped by unique entries in "Name" column, resulting table will be called "DataMean":
byNm := split.GroupBy(ix, []string{"Name"}) // column name(s) to group by
split.Agg(byNm, "Data", agg.AggMean) //
gps := byNm.AggsToTable(etable.AddAggName) // etable.AddAggName or etable.ColNameOnly for naming cols
Describe (basic stats) all columns in a table:
ix := etable.NewIndexView(et) // new view with all rows
desc := agg.DescAll(ix) // summary stats of all columns
// get value at given column name (from original table), row "Mean"
mean := desc.Float("ColNm", desc.RowsByString("Agg", "Mean", etable.Equals, etable.UseCase)[0])
CSV / TSV file format
Tables can be saved and loaded from CSV (comma separated values) or TSV (tab separated values) files. See the next section for special formatting of header strings in these files to record the type and tensor cell shapes.
Type and Tensor Headers
To capture the type and shape of the columns, we support the following header formatting. We weren't able to find any other widely supported standard (please let us know if there is one that we've missed!)
Here is the mapping of special header prefix characters to standard types:
'$': etensor.STRING,
'%': etensor.FLOAT32,
'#': etensor.FLOAT64,
'|': etensor.INT64,
'@': etensor.UINT8,
'^': etensor.BOOl,
Columns that have tensor cell shapes (not just scalars) are marked as such with the first such column having a <ndim:dim,dim..>
suffix indicating the shape of the cells in this column, e.g., <2:5,4>
indicates a 2D cell Y=5,X=4. Each individual column is then indexed as [ndims:x,y..]
e.g., the first would be [2:0,0]
, then [2:0,1]
etc.
Example
Here's a TSV file for a scalar String column (Name
), a 2D 1x4 tensor float32 column (Input
), and a 2D 1x2 float32 Output
column.
_H: $Name %Input[2:0,0]<2:1,4> %Input[2:0,1] %Input[2:0,2] %Input[2:0,3] %Output[2:0,0]<2:1,2> %Output[2:0,1]
_D: Event_0 1 0 0 0 1 0
_D: Event_1 0 1 0 0 1 0
_D: Event_2 0 0 1 0 0 1
_D: Event_3 0 0 0 1 0 1
Documentation ¶
Index ¶
- Constants
- Variables
- func AddColumn[T string | bool | float32 | float64 | int | int32 | byte](dt *Table, name string) tensor.Tensor
- func AddTensorColumn[T string | bool | float32 | float64 | int | int32 | byte](dt *Table, name string, cellSizes []int, dimNames ...string) tensor.Tensor
- func ConfigFromDataValues(dt *Table, hdrs []string, rec [][]string) error
- func ConfigFromHeaders(dt *Table, hdrs []string, rec [][]string) error
- func ConfigFromTableHeaders(dt *Table, hdrs []string) error
- func DetectTableHeaders(hdrs []string) bool
- func InferDataType(str string) reflect.Kind
- func InsertColumn[T string | bool | float32 | float64 | int | int32 | byte](dt *Table, name string, idx int) tensor.Tensor
- func ShapeFromString(dims string) []int
- func TableColumnType(nm string) (reflect.Kind, string)
- func TableHeaderChar(typ reflect.Kind) byte
- func UpdateSliceTable(st any, dt *Table)
- type Delims
- func (i Delims) Desc() string
- func (i Delims) Int64() int64
- func (i Delims) MarshalText() ([]byte, error)
- func (dl Delims) Rune() rune
- func (i *Delims) SetInt64(in int64)
- func (i *Delims) SetString(s string) error
- func (i Delims) String() string
- func (i *Delims) UnmarshalText(text []byte) error
- func (i Delims) Values() []enums.Enum
- type Filterer
- type IndexView
- func (ix *IndexView) AddIndex(idx int)
- func (ix *IndexView) AddRows(n int)
- func (ix *IndexView) Clone() *IndexView
- func (ix *IndexView) CopyFrom(oix *IndexView)
- func (ix *IndexView) DeleteInvalid()
- func (ix *IndexView) DeleteRows(at, n int)
- func (ix *IndexView) Filter(filterer func(et *Table, row int) bool)
- func (ix *IndexView) FilterColumn(colIndex int, str string, exclude, contains, ignoreCase bool)
- func (ix *IndexView) FilterColumnName(column string, str string, exclude, contains, ignoreCase bool) error
- func (ix *IndexView) InsertRows(at, n int)
- func (ix *IndexView) Len() int
- func (ix *IndexView) Less(i, j int) bool
- func (ix *IndexView) NewTable() *Table
- func (ix *IndexView) OpenCSV(filename core.Filename, delim Delims) error
- func (ix *IndexView) OpenFS(fsys fs.FS, filename string, delim Delims) error
- func (ix *IndexView) Permuted()
- func (ix *IndexView) RowsByString(column string, str string, contains, ignoreCase bool) ([]int, error)
- func (ix *IndexView) RowsByStringIndex(colIndex int, str string, contains, ignoreCase bool) []int
- func (ix *IndexView) SaveCSV(filename core.Filename, delim Delims, headers bool) error
- func (ix *IndexView) Sequential()
- func (ix *IndexView) SetTable(et *Table)
- func (ix *IndexView) Sort(lessFunc func(et *Table, i, j int) bool)
- func (ix *IndexView) SortColumn(colIndex int, ascending bool)
- func (ix *IndexView) SortColumnName(column string, ascending bool) error
- func (ix *IndexView) SortColumnNames(columns []string, ascending bool) error
- func (ix *IndexView) SortColumns(colIndexes []int, ascending bool)
- func (ix *IndexView) SortIndexes()
- func (ix *IndexView) SortStable(lessFunc func(et *Table, i, j int) bool)
- func (ix *IndexView) SortStableColumn(colIndex int, ascending bool)
- func (ix *IndexView) SortStableColumnName(column string, ascending bool) error
- func (ix *IndexView) SortStableColumnNames(columns []string, ascending bool) error
- func (ix *IndexView) SortStableColumns(colIndexes []int, ascending bool)
- func (ix *IndexView) Swap(i, j int)
- func (ix *IndexView) WriteCSV(w io.Writer, delim Delims, headers bool) error
- type LessFunc
- type SplitAgg
- type Splits
- func (spl *Splits) AddAgg(name string, colIndex int) *SplitAgg
- func (spl *Splits) AggByColumnName(name string) (*SplitAgg, error)
- func (spl *Splits) AggByName(name string) (*SplitAgg, error)
- func (spl *Splits) AggsToTable(colName bool) *Table
- func (spl *Splits) AggsToTableCopy(colName bool) *Table
- func (spl *Splits) ByValue(values []string) []int
- func (spl *Splits) Clone() *Splits
- func (spl *Splits) CopyFrom(osp *Splits)
- func (spl *Splits) Delete(idx int)
- func (spl *Splits) DeleteAggs()
- func (spl *Splits) ExtractLevels(levels []int) (*Splits, error)
- func (spl *Splits) Filter(fun func(idx int) bool)
- func (spl *Splits) Len() int
- func (spl *Splits) Less(i, j int) bool
- func (spl *Splits) New(dt *Table, values []string, rows ...int) *IndexView
- func (spl *Splits) ReorderLevels(order []int) error
- func (spl *Splits) SetLevels(levels ...string)
- func (spl *Splits) Sort(lessFunc func(spl *Splits, i, j int) bool)
- func (spl *Splits) SortLevels()
- func (spl *Splits) SortOrder(order []int) error
- func (spl *Splits) Swap(i, j int)
- func (spl *Splits) Table() *Table
- type SplitsLessFunc
- type Table
- func (dt *Table) AddColumn(tsr tensor.Tensor, name string) error
- func (dt *Table) AddColumnOfType(typ reflect.Kind, name string) tensor.Tensor
- func (dt *Table) AddFloat32Column(name string) *tensor.Float32
- func (dt *Table) AddFloat32TensorColumn(name string, cellSizes []int, dimNames ...string) *tensor.Float32
- func (dt *Table) AddFloat64Column(name string) *tensor.Float64
- func (dt *Table) AddFloat64TensorColumn(name string, cellSizes []int, dimNames ...string) *tensor.Float64
- func (dt *Table) AddIntColumn(name string) *tensor.Int
- func (dt *Table) AddIntTensorColumn(name string, cellSizes []int, dimNames ...string) *tensor.Int
- func (dt *Table) AddRows(n int)
- func (dt *Table) AddStringColumn(name string) *tensor.String
- func (dt *Table) AddTensorColumnOfType(typ reflect.Kind, name string, cellSizes []int, dimNames ...string) tensor.Tensor
- func (dt *Table) AppendRows(dt2 *Table)
- func (dt *Table) Clone() *Table
- func (dt *Table) Column(i int) tensor.Tensor
- func (dt *Table) ColumnByName(name string) (tensor.Tensor, error)
- func (dt *Table) ColumnIndex(name string) (int, error)
- func (dt *Table) ColumnIndexesByNames(names ...string) ([]int, error)
- func (dt *Table) ColumnName(i int) string
- func (dt *Table) ConfigFromTable(ft *Table) error
- func (dt *Table) CopyCell(column string, row int, cpt *Table, cpColNm string, cpRow int) error
- func (dt *Table) CopyMetaDataFrom(cp *Table)
- func (dt *Table) DeleteAll()
- func (dt *Table) DeleteColumnIndex(idx int)
- func (dt *Table) DeleteColumnName(name string) error
- func (dt *Table) Float(column string, row int) float64
- func (dt *Table) FloatIndex(column, row int) float64
- func (dt *Table) InsertColumn(tsr tensor.Tensor, name string, idx int) error
- func (dt *Table) InsertKeyColumns(args ...string) *Table
- func (dt *Table) IsValidRow(row int) error
- func (dt *Table) NumColumns() int
- func (dt *Table) NumRows() int
- func (dt *Table) OpenCSV(filename core.Filename, delim Delims) error
- func (dt *Table) OpenFS(fsys fs.FS, filename string, delim Delims) error
- func (dt *Table) ReadCSV(r io.Reader, delim Delims) error
- func (dt *Table) ReadCSVRow(rec []string, row int)
- func (dt *Table) RowsByString(column string, str string, contains, ignoreCase bool) ([]int, error)
- func (dt *Table) RowsByStringIndex(column int, str string, contains, ignoreCase bool) []int
- func (dt *Table) SaveCSV(filename core.Filename, delim Delims, headers bool) error
- func (dt *Table) SetFloat(column string, row int, val float64) error
- func (dt *Table) SetFloatIndex(column, row int, val float64) error
- func (dt *Table) SetMetaData(key, val string)
- func (dt *Table) SetNumRows(rows int) *Table
- func (dt *Table) SetString(column string, row int, val string) error
- func (dt *Table) SetStringIndex(column, row int, val string) error
- func (dt *Table) SetTensor(column string, row int, val tensor.Tensor) error
- func (dt *Table) SetTensorFloat1D(column string, row int, idx int, val float64) error
- func (dt *Table) SetTensorIndex(column, row int, val tensor.Tensor) error
- func (dt *Table) StringIndex(column, row int) string
- func (dt *Table) StringValue(column string, row int) string
- func (dt *Table) TableHeaders() []string
- func (dt *Table) Tensor(column string, row int) tensor.Tensor
- func (dt *Table) TensorFloat1D(column string, row int, idx int) float64
- func (dt *Table) TensorIndex(column, row int) tensor.Tensor
- func (dt *Table) UpdateColumnNameMap() error
- func (dt *Table) WriteCSV(w io.Writer, delim Delims, headers bool) error
- func (dt *Table) WriteCSVHeaders(w io.Writer, delim Delims) (int, error)
- func (dt *Table) WriteCSVRow(w io.Writer, row int, delim Delims) error
- func (dt *Table) WriteCSVRowWriter(cw *csv.Writer, row int, ncol int) error
Constants ¶
const ( // Ascending specifies an ascending sort direction for table Sort routines Ascending = true // Descending specifies a descending sort direction for table Sort routines Descending = false )
const ( // Headers is passed to CSV methods for the headers arg, to use headers // that capture full type and tensor shape information. Headers = true // NoHeaders is passed to CSV methods for the headers arg, to not use headers NoHeaders = false )
const ( // ColumnNameOnly means resulting agg table just has the original column name, no aggregation name ColumnNameOnly bool = true // AddAggName means resulting agg table columns have aggregation name appended AddAggName = false )
use these for arg to ArgsToTable*
const ( // Contains means the string only needs to contain the target string (see Equals) Contains bool = true // Equals means the string must equal the target string (see Contains) Equals = false // IgnoreCase means that differences in case are ignored in comparing strings IgnoreCase = true // UseCase means that case matters when comparing strings UseCase = false )
Named arg values for Contains, IgnoreCase
Variables ¶
var TableHeaderToType = map[byte]reflect.Kind{ '$': reflect.String, '%': reflect.Float32, '#': reflect.Float64, '|': reflect.Int, '^': reflect.Bool, }
TableHeaderToType maps special header characters to data type
Functions ¶
func AddColumn ¶
func AddColumn[T string | bool | float32 | float64 | int | int32 | byte](dt *Table, name string) tensor.Tensor
AddColumn adds a new column to the table, of given type and column name (which must be unique). The cells of this column hold a single scalar value: see AddColumnTensor for n-dimensional cells.
func AddTensorColumn ¶
func AddTensorColumn[T string | bool | float32 | float64 | int | int32 | byte](dt *Table, name string, cellSizes []int, dimNames ...string) tensor.Tensor
AddTensorColumn adds a new n-dimensional column to the table, of given type, column name (which must be unique), and dimensionality of each _cell_. An outer-most Row dimension will be added to this dimensionality to create the tensor column.
func ConfigFromDataValues ¶
ConfigFromDataValues configures a Table based on data types inferred from the string representation of given records, using header names if present.
func ConfigFromHeaders ¶
ConfigFromHeaders attempts to configure Table based on the headers. for non-table headers, data is examined to determine types.
func ConfigFromTableHeaders ¶
ConfigFromTableHeaders attempts to configure a Table based on special table headers
func DetectTableHeaders ¶
DetectTableHeaders looks for special header characters -- returns true if found
func InferDataType ¶
InferDataType returns the inferred data type for the given string only deals with float64, int, and string types
func InsertColumn ¶ added in v0.2.0
func InsertColumn[T string | bool | float32 | float64 | int | int32 | byte](dt *Table, name string, idx int) tensor.Tensor
InsertColumn inserts a new column to the table, of given type and column name (which must be unique), at given index. The cells of this column hold a single scalar value.
func ShapeFromString ¶
ShapeFromString parses string representation of shape as N:d,d,..
func TableColumnType ¶
TableColumnType parses the column header for special table type information
func TableHeaderChar ¶
TableHeaderChar returns the special header character based on given data type
func UpdateSliceTable ¶
UpdateSliceTable updates given Table with data from the given slice of structs, which must be the same type as used to configure the table
Types ¶
type Delims ¶
type Delims int32 //enums:enum
Delim are standard CSV delimiter options (Tab, Comma, Space)
const ( // Tab is the tab rune delimiter, for TSV tab separated values Tab Delims = iota // Comma is the comma rune delimiter, for CSV comma separated values Comma // Space is the space rune delimiter, for SSV space separated value Space // Detect is used during reading a file -- reads the first line and detects tabs or commas Detect )
const DelimsN Delims = 4
DelimsN is the highest valid value for type Delims, plus one.
func DelimsValues ¶
func DelimsValues() []Delims
DelimsValues returns all possible values for the type Delims.
func (Delims) MarshalText ¶
MarshalText implements the encoding.TextMarshaler interface.
func (*Delims) SetString ¶
SetString sets the Delims value from its string representation, and returns an error if the string is invalid.
func (*Delims) UnmarshalText ¶
UnmarshalText implements the encoding.TextUnmarshaler interface.
type Filterer ¶ added in v0.2.1
Filterer is a function used for filtering that returns true if Table row should be included in the current filtered view of the table, and false if it should be removed.
type IndexView ¶
type IndexView struct { // Table that we are an indexed view onto Table *Table // current indexes into Table Indexes []int // contains filtered or unexported fields }
IndexView is an indexed wrapper around an table.Table that provides a specific view onto the Table defined by the set of indexes. This provides an efficient way of sorting and filtering a table by only updating the indexes while doing nothing to the Table itself. To produce a table that has data actually organized according to the indexed order, call the NewTable method. IndexView views on a table can also be organized together as Splits of the table rows, e.g., by grouping values along a given column.
func NewIndexView ¶
NewIndexView returns a new IndexView based on given table, initialized with sequential idxes
func (*IndexView) AddRows ¶
AddRows adds n rows to end of underlying Table, and to the indexes in this view
func (*IndexView) CopyFrom ¶
CopyFrom copies from given other IndexView (we have our own unique copy of indexes)
func (*IndexView) DeleteInvalid ¶
func (ix *IndexView) DeleteInvalid()
DeleteInvalid deletes all invalid indexes from the list. Call this if rows (could) have been deleted from table.
func (*IndexView) DeleteRows ¶
DeleteRows deletes n rows of indexes starting at given index in the list of indexes
func (*IndexView) Filter ¶
Filter filters the indexes into our Table using given Filter function. The Filter function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes.
func (*IndexView) FilterColumn ¶
FilterColumn sorts the indexes into our Table according to values in given column index, using string representation of column values. Includes rows with matching values unless exclude is set. If contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity. Only valid for 1-dimensional columns.
func (*IndexView) FilterColumnName ¶
func (ix *IndexView) FilterColumnName(column string, str string, exclude, contains, ignoreCase bool) error
FilterColumnName filters the indexes into our Table according to values in given column name, using string representation of column values. Includes rows with matching values unless exclude is set. If contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity. Only valid for 1-dimensional columns. Returns error if column name not found.
func (*IndexView) InsertRows ¶
InsertRows adds n rows to end of underlying Table, and to the indexes starting at given index in this view
func (*IndexView) NewTable ¶
NewTable returns a new table with column data organized according to the indexes
func (*IndexView) OpenCSV ¶
OpenCSV reads a table idx view from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. If the file was saved from table with headers, then these have full configuration information for tensor type and dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.
func (*IndexView) OpenFS ¶
OpenFS is the version of IndexView.OpenCSV that uses an fs.FS filesystem.
func (*IndexView) Permuted ¶
func (ix *IndexView) Permuted()
Permuted sets indexes to a permuted order -- if indexes already exist then existing list of indexes is permuted, otherwise a new set of permuted indexes are generated
func (*IndexView) RowsByString ¶
func (ix *IndexView) RowsByString(column string, str string, contains, ignoreCase bool) ([]int, error)
RowsByString returns the list of *our indexes* whose row in the table has given string value in given column name (de-reference our indexes to get actual row). if contains, only checks if row contains string; if ignoreCase, ignores case. returns error message for invalid column name. Use named args for greater clarity.
func (*IndexView) RowsByStringIndex ¶
RowsByStringIndex returns the list of *our indexes* whose row in the table has given string value in given column index (de-reference our indexes to get actual row). if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.
func (*IndexView) SaveCSV ¶
SaveCSV writes a table index view to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate column headers that capture the type and tensor cell geometry of the columns, enabling full reloading of exactly the same table format and data (recommended). Otherwise, only the data is written.
func (*IndexView) Sequential ¶
func (ix *IndexView) Sequential()
Sequential sets indexes to sequential row-wise indexes into table
func (*IndexView) SetTable ¶
SetTable sets as indexes into given table with sequential initial indexes
func (*IndexView) Sort ¶
Sort sorts the indexes into our Table using given Less function. The Less function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes.
func (*IndexView) SortColumn ¶
SortColumn sorts the indexes into our Table according to values in given column index, using either ascending or descending order. Only valid for 1-dimensional columns.
func (*IndexView) SortColumnName ¶
SortColumnName sorts the indexes into our Table according to values in given column name, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.
func (*IndexView) SortColumnNames ¶
SortColumnNames sorts the indexes into our Table according to values in given column names, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.
func (*IndexView) SortColumns ¶
SortColumns sorts the indexes into our Table according to values in given list of column indexes, using either ascending or descending order for all of the columns. Only valid for 1-dimensional columns.
func (*IndexView) SortIndexes ¶
func (ix *IndexView) SortIndexes()
SortIndexes sorts the indexes into our Table directly in numerical order, producing the native ordering, while preserving any filtering that might have occurred.
func (*IndexView) SortStable ¶
SortStable stably sorts the indexes into our Table using given Less function. The Less function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes. It is *essential* that it always returns false when the two are equal for the stable function to actually work.
func (*IndexView) SortStableColumn ¶
SortStableColumn sorts the indexes into our Table according to values in given column index, using either ascending or descending order. Only valid for 1-dimensional columns.
func (*IndexView) SortStableColumnName ¶
SortStableColumnName sorts the indexes into our Table according to values in given column name, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.
func (*IndexView) SortStableColumnNames ¶
SortStableColumnNames sorts the indexes into our Table according to values in given column names, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.
func (*IndexView) SortStableColumns ¶
SortStableColumns sorts the indexes into our Table according to values in given list of column indexes, using either ascending or descending order for all of the columns. Only valid for 1-dimensional columns.
func (*IndexView) WriteCSV ¶
WriteCSV writes only rows in table idx view to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate column headers that capture the type and tensor cell geometry of the columns, enabling full reloading of exactly the same table format and data (recommended). Otherwise, only the data is written.
type LessFunc ¶
LessFunc is a function used for sort comparisons that returns true if Table row i is less than Table row j -- these are the raw row numbers, which have already been projected through indexes when used for sorting via Indexes.
type SplitAgg ¶
type SplitAgg struct { // the name of the aggregation operation performed, e.g., Sum, Mean, etc Name string // column index on which the aggregation was performed -- results will have same shape as cells in this column ColumnIndex int // aggregation results -- outer index is length of splits, inner is the length of the cell shape for the column Aggs [][]float64 }
SplitAgg contains aggregation results for splits
type Splits ¶
type Splits struct { // the list of index views for each split Splits []*IndexView // levels of indexes used to organize the splits -- each split contains the full outer product across these index levels. for example, if the split was generated by grouping over column values, then these are the column names in order of grouping. the splits are not automatically sorted hierarchically by these levels but e.g., the GroupBy method produces that result -- use the Sort methods to explicitly sort. Levels []string // the values of the index levels associated with each split. The outer dimension is the same length as Splits, and the inner dimension is the levels. Values [][]string // aggregate results, one for each aggregation operation performed -- split-level data is contained within each SplitAgg struct -- deleting a split removes these aggs but adding new splits just invalidates all existing aggs (they are automatically deleted). Aggs []*SplitAgg // contains filtered or unexported fields }
Splits is a list of indexed views into a given Table, that represent a particular way of splitting up the data, e.g., whenever a given column value changes.
It is functionally equivalent to the MultiIndex in python's pandas: it has multiple levels of indexes as listed in the Levels field, which then have corresponding Values for each split. These index levels can be re-ordered, and new Splits or IndexViews's can be created from subsets of the existing levels. The Values are stored simply as string values, as this is the most general type and often index values are labels etc.
For Splits created by the splits.GroupBy function for example, each index Level is the column name that the data was grouped by, and the Values for each split are then the values of those columns. However, any arbitrary set of levels and values can be used, e.g., as in the splits.GroupByFunc function.
Conceptually, a given Split always contains the full "outer product" of all the index levels -- there is one split for each unique combination of values along each index level. Thus, removing one level collapses across those values and moves the corresponding indexes into the remaining split indexes.
You can Sort and Filter based on the index values directly, to reorganize the splits and drop particular index values, etc.
Splits also maintains Aggs aggregate values for each split, which can be computed using standard aggregation methods over data columns, using the split.Agg* functions.
The table code contains the structural methods for managing the Splits data. See split package for end-user methods to generate different kinds of splits, and perform aggregations, etc.
func (*Splits) AggByColumnName ¶
AggByColumnName returns Agg results for given column name, optionally including :Name agg name appended, where Name is the name given to the Agg result (e.g., Mean for a standard Mean agg). Returns error message if not found.
func (*Splits) AggByName ¶
AggByName returns Agg results for given name, which does NOT include the column name, just the name given to the Agg result (e.g., Mean for a standard Mean agg). Returns error message if not found.
func (*Splits) AggsToTable ¶
AggsToTable returns a Table containing this Splits' aggregate data. Must have Levels and Aggs all created as in the split.Agg* methods. if colName == ColumnNameOnly, then the name of the columns for the Table is just the corresponding agg column name -- otherwise it also includes the name of the aggregation function with a : divider (e.g., Name:Mean)
func (*Splits) AggsToTableCopy ¶
AggsToTableCopy returns a Table containing this Splits' aggregate data and a copy of the first row of data for each split for all non-agg cols, which is useful for recording other data that goes along with aggregated values. Must have Levels and Aggs all created as in the split.Agg* methods. if colName == ColumnNameOnly, then the name of the columns for the Table is just the corresponding agg column name -- otherwise it also includes the name of the aggregation function with a : divider (e.g., Name:Mean)
func (*Splits) ByValue ¶
ByValue finds split indexes by matching to split values, returns nil if not found. values are used in order as far as they go and any remaining values are assumed to match, and any empty values will match anything. Can use this to access different subgroups within overall set of splits.
func (*Splits) CopyFrom ¶
CopyFrom copies from other Splits -- we get our own unique copy of everything
func (*Splits) Delete ¶
Delete deletes split at given index -- use this to coordinate deletion of Splits, Values, and Aggs values for given split
func (*Splits) DeleteAggs ¶
func (spl *Splits) DeleteAggs()
DeleteAggs deletes all existing aggregation data
func (*Splits) ExtractLevels ¶
ExtractLevels returns a new Splits that only has the given levels of indexes, in their given order, with the other levels removed and their corresponding indexes merged into the appropriate remaining levels. Any existing aggregation data is not retained in the new splits.
func (*Splits) New ¶
New adds a new split to the list for given table, and with associated values, which are copied before saving into Values list, and any number of rows from the table associated with this split (also copied). Any existing Aggs are deleted by this.
func (*Splits) ReorderLevels ¶
ReorderLevels re-orders the index levels according to the given new ordering indexes e.g., []int{1,0} will move the current level 0 to level 1, and 1 to level 0 no checking is done to ensure these are sensible beyond basic length test -- behavior undefined if so. Typically you want to call SortLevels after this.
func (*Splits) SetLevels ¶
SetLevels sets the Levels index names -- must match actual index dimensionality of the Values. This is automatically done by e.g., GroupBy, but must be done manually if creating custom indexes.
func (*Splits) SortLevels ¶
func (spl *Splits) SortLevels()
SortLevels sorts the splits according to the current index level ordering of values i.e., first index level is outer sort dimension, then within that is the next, etc
type SplitsLessFunc ¶
SplitsLessFunc is a function used for sort comparisons that returns true if split i is less than split j
type Table ¶
type Table struct { // columns of data, as tensor.Tensor tensors Columns []tensor.Tensor `display:"no-inline"` // the names of the columns ColumnNames []string // number of rows, which is enforced to be the size of the outer-most dimension of the column tensors Rows int `edit:"-"` // the map of column names to column numbers ColumnNameMap map[string]int `display:"-"` // misc meta data for the table. We use lower-case key names following the struct tag convention: name = name of table; desc = description; read-only = gui is read-only; precision = n for precision to write out floats in csv. For Column-specific data, we look for ColumnName: prefix, specifically ColumnName:desc = description of the column contents, which is shown as tooltip in the tensorcore.Table, and :width for width of a column MetaData map[string]string }
Table is a table of data, with columns of tensors, each with the same number of Rows (outer-most dimension).
func NewSliceTable ¶
NewSliceTable returns a new Table with data from the given slice of structs.
func (*Table) AddColumn ¶
AddColumn adds the given tensor as a column to the table, returning an error and not adding if the name is not unique. Automatically adjusts the shape to fit the current number of rows.
func (*Table) AddColumnOfType ¶
AddColumnOfType adds a new scalar column to the table, of given reflect type, column name (which must be unique), The cells of this column hold a single (scalar) value of given type. Supported types are string, bool (for tensor.Bits), float32, float64, int, int32, and byte.
func (*Table) AddFloat32Column ¶
AddFloat32Column adds a new float32 column with given name. The cells of this column hold a single scalar value.
func (*Table) AddFloat32TensorColumn ¶
func (dt *Table) AddFloat32TensorColumn(name string, cellSizes []int, dimNames ...string) *tensor.Float32
AddFloat32TensorColumn adds a new n-dimensional float32 column with given name and dimensionality of each _cell_. An outer-most Row dimension will be added to this dimensionality to create the tensor column.
func (*Table) AddFloat64Column ¶
AddFloat64Column adds a new float64 column with given name. The cells of this column hold a single scalar value.
func (*Table) AddFloat64TensorColumn ¶
func (dt *Table) AddFloat64TensorColumn(name string, cellSizes []int, dimNames ...string) *tensor.Float64
AddFloat64TensorColumn adds a new n-dimensional float64 column with given name and dimensionality of each _cell_. An outer-most Row dimension will be added to this dimensionality to create the tensor column.
func (*Table) AddIntColumn ¶
AddIntColumn adds a new int column with given name. The cells of this column hold a single scalar value.
func (*Table) AddIntTensorColumn ¶
AddIntTensorColumn adds a new n-dimensional int column with given name and dimensionality of each _cell_. An outer-most Row dimension will be added to this dimensionality to create the tensor column.
func (*Table) AddStringColumn ¶
AddStringColumn adds a new String column with given name. The cells of this column hold a single string value.
func (*Table) AddTensorColumnOfType ¶
func (dt *Table) AddTensorColumnOfType(typ reflect.Kind, name string, cellSizes []int, dimNames ...string) tensor.Tensor
AddTensorColumnOfType adds a new n-dimensional column to the table, of given reflect type, column name (which must be unique), and dimensionality of each _cell_. An outer-most Row dimension will be added to this dimensionality to create the tensor column. Supported types are string, bool (for tensor.Bits), float32, float64, int, int32, and byte.
func (*Table) AppendRows ¶
AppendRows appends shared columns in both tables with input table rows
func (*Table) ColumnByName ¶
ColumnByName returns the tensor at given column name, with error message if not found. Returns nil if not found
func (*Table) ColumnIndex ¶
ColumnIndex returns the index of the given column name, along with an error if not found.
func (*Table) ColumnIndexesByNames ¶
ColumnIndexesByNames returns the indexes of the given column names. idxs have -1 if name not found.
func (*Table) ColumnName ¶
ColumnName returns the name of given column
func (*Table) ConfigFromTable ¶ added in v0.3.3
ConfigFromTable configures the columns of this table according to the values in the first two columns of given format table, conventionally named Name, Type (but names are not used), which must be of the string type.
func (*Table) CopyCell ¶
CopyCell copies into cell at given column, row from cell in other table. It is robust to differences in type; uses destination cell type. Returns error if column names are invalid.
func (*Table) CopyMetaDataFrom ¶
CopyMetaDataFrom copies meta data from other table
func (*Table) DeleteColumnIndex ¶
DeleteColumnIndex deletes column of given index
func (*Table) DeleteColumnName ¶
DeleteColumnName deletes column of given name. returns error if not found.
func (*Table) Float ¶
Float returns the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns NaN if column is not a 1-dimensional tensor or col name not found, or row not valid.
func (*Table) FloatIndex ¶
FloatIndex returns the float64 value of cell at given column, row index for columns that have 1-dimensional tensors. Returns NaN if column is not a 1-dimensional tensor or row not valid.
func (*Table) InsertColumn ¶ added in v0.2.0
InsertColumn inserts the given tensor as a column to the table at given index, returning an error and not adding if the name is not unique. Automatically adjusts the shape to fit the current number of rows.
func (*Table) InsertKeyColumns ¶ added in v0.3.3
InsertKeyColumns returns a copy of the given Table with new columns having given values, inserted at the start, used as legend keys etc. args must be in pairs: column name, value. All rows get the same value.
func (*Table) IsValidRow ¶
IsValidRow returns error if the row is invalid
func (*Table) NumColumns ¶
NumColumns returns the number of columns
func (*Table) OpenCSV ¶
OpenCSV reads a table from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. If the file was saved from table with headers, then these have full configuration information for tensor type and dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.
func (*Table) OpenFS ¶
OpenFS is the version of Table.OpenCSV that uses an fs.FS filesystem.
func (*Table) ReadCSV ¶
ReadCSV reads a table from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. If the file was saved from table with headers, then these have full configuration information for tensor type and dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.
func (*Table) ReadCSVRow ¶
ReadCSVRow reads a record of CSV data into given row in table
func (*Table) RowsByString ¶
RowsByString returns the list of rows that have given string value in given column name. returns nil & error if name invalid. if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.
func (*Table) RowsByStringIndex ¶
RowsByStringIndex returns the list of rows that have given string value in given column index. if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.
func (*Table) SaveCSV ¶
SaveCSV writes a table to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate column headers that capture the type and tensor cell geometry of the columns, enabling full reloading of exactly the same table format and data (recommended). Otherwise, only the data is written.
func (*Table) SetFloat ¶
SetFloat sets the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors.
func (*Table) SetFloatIndex ¶
SetFloatIndex sets the float64 value of cell at given column, row index for columns that have 1-dimensional tensors.
func (*Table) SetMetaData ¶
SetMetaData sets given meta-data key to given value, safely creating the map if not yet initialized. Standard Keys are: * name -- name of table * desc -- description of table * read-only -- makes gui read-only (inactive edits) for tensorcore.Table * ColumnName:* -- prefix for all column-specific meta-data
- desc -- description of column
func (*Table) SetNumRows ¶
SetNumRows sets the number of rows in the table, across all columns if rows = 0 then effective number of rows in tensors is 1, as this dim cannot be 0
func (*Table) SetString ¶
SetString sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns true if set.
func (*Table) SetStringIndex ¶
SetStringIndex sets the string value of cell at given column, row index for columns that have 1-dimensional tensors. Returns true if set.
func (*Table) SetTensor ¶
SetTensor sets the tensor value of cell at given column (by name), row index for columns that have n-dimensional tensors. Returns true if set.
func (*Table) SetTensorFloat1D ¶
SetTensorFloat1D sets the tensor cell's float cell value at given 1D index within cell, at given column (by name), row index for columns that have n-dimensional tensors. Returns true if set.
func (*Table) SetTensorIndex ¶
SetTensorIndex sets the tensor value of cell at given column, row index for columns that have n-dimensional tensors. Returns true if set.
func (*Table) StringIndex ¶
StringIndex returns the string value of cell at given column, row index for columns that have 1-dimensional tensors. Returns "" if column is not a 1-dimensional tensor or row not valid.
func (*Table) StringValue ¶
StringValue returns the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns "" if column is not a 1-dimensional tensor or row not valid.
func (*Table) TableHeaders ¶
TableHeaders generates special header strings from the table with full information about type and tensor cell dimensionality.
func (*Table) Tensor ¶
Tensor returns the tensor SubSpace for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns nil on any error.
func (*Table) TensorFloat1D ¶
TensorFloat1D returns the float value of a Tensor cell's cell at given 1D offset within cell, for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns 0 on any error.
func (*Table) TensorIndex ¶
TensorIndex returns the tensor SubSpace for given column, row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns nil if column is a 1-dimensional tensor or there is any error from the tensor.Tensor.SubSpace call.
func (*Table) UpdateColumnNameMap ¶
UpdateColumnNameMap updates the column name map, returning an error if any of the column names are duplicates.
func (*Table) WriteCSV ¶
WriteCSV writes a table to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate column headers that capture the type and tensor cell geometry of the columns, enabling full reloading of exactly the same table format and data (recommended). Otherwise, only the data is written.
func (*Table) WriteCSVHeaders ¶
WriteCSVHeaders writes headers to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). Returns number of columns in header
func (*Table) WriteCSVRow ¶
WriteCSVRow writes given row to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg)