etable

package
v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 5, 2022 License: BSD-3-Clause Imports: 18 Imported by: 209

Documentation

Overview

Package etable provides the etable.Table structure which provides a DataTable or DataFrame data representation, which is a collection of columnar data all having the same number of rows.

Each column is an etensor.Tensor, so it can represent scalar or higher dimensional data per each cell (row x column location) in the Table. Thus, scalar data is represented using a 1D Tensor where the 1 dimension is the rows of the table, and likewise higher dimensional data always has the outer-most dimension as the row.

All tensors MUST have RowMajor stride layout for consistency, with the outer-most dimension as the row dimension, which is enforced to be the same across all columns.

The tensor columns can be individually converted to / from arrow.Tensors and conversion between arrow.Table is planned, along with inter-conversion with relevant gonum structures including the planned dframe.Frame.

Native support is provided for basic CSV, TSV I/O, including the C++ emergent standard TSV format with full type information in the first row column headers.

The etable.IdxView is an indexed view into a Table, which is used for all data-processing operations such as Sort, Filter, Split (groupby), and for aggregating data as in a pivot-table.

See etable/agg package for aggregation functions that operate on the IdxView to perform standard aggregation operations such as Sum, Mean, etc.

Other relevant examples of DataTable-like structures: * https://github.com/apache/arrow/tree/master/go/arrow Table * http://xarray.pydata.org/en/stable/index.html * https://pandas.pydata.org/pandas-docs/stable/reference/frame.html * https://www.rdocumentation.org/packages/base/versions/3.4.3/topics/data.frame * https://github.com/tobgu/qframe * https://github.com/kniren/gota

Index

Constants

View Source 
const (
	// Contains means the string only needs to contain the target string (see Equals)
	Contains bool = true
	// Equals means the string must equal the target string (see Contains)
	Equals = false
	// IgnoreCase means that differences in case are ignored in comparing strings
	IgnoreCase = true
	// UseCase means that case matters when comparing strings
	UseCase = false
)

Named arg values for Contains, IgnoreCase

View Source 
const (
	// Ascending specifies an ascending sort direction for etable Sort routines
	Ascending = true

	// Descending specifies a descending sort direction for etable Sort routines
	Descending = false
)
View Source 
const (
	//	Headers is passed to CSV methods for the headers arg, to use headers
	Headers = true

	// NoHeaders is passed to CSV methods for the headers arg, to not use headers
	NoHeaders = false
)
View Source 
const (
	// ColNameOnly means resulting agg table just has the original column name, no aggregation name
	ColNameOnly bool = true
	// AddAggName means resulting agg table columns have aggregation name appended
	AddAggName = false
)

use these for arg to ArgsToTable*

View Source 
const (
	Version     = "v1.1.2"
	GitCommit   = "1643946"          // the commit JUST BEFORE the release
	VersionDate = "2022-06-05 11:40" // UTC
)

Variables

View Source 
var EmerHdrCharToType = map[byte]etensor.Type{
	'$': etensor.STRING,
	'%': etensor.FLOAT32,
	'#': etensor.FLOAT64,
	'|': etensor.INT64,
	'@': etensor.UINT8,
	'&': etensor.STRING,
	'^': etensor.BOOL,
}
View Source 
var EmerHdrTypeToChar map[etensor.Type]byte
View Source 
var IdxViewProps = ki.Props{
	"ToolBar": ki.PropSlice{
		{"AddRows", ki.Props{
			"icon": "plus",
			"Args": ki.PropSlice{
				{"N Rows", ki.Props{
					"default": 1,
				}},
			},
		}},
		{"SortColName", ki.Props{
			"label": "Sort...",
			"desc":  "sort by given column name",
			"icon":  "edit",
			"Args": ki.PropSlice{
				{"Column Name", ki.Props{
					"width": 20,
				}},
				{"Ascending", ki.Props{}},
			},
		}},
		{"FilterColName", ki.Props{
			"label": "Filter...",
			"desc":  "Filter rows by values in given column name, using string representation.  Includes matches unless exclude is set.  contains matches if column contains value, otherwise must be entire value.",
			"icon":  "search",
			"Args": ki.PropSlice{
				{"Column Name", ki.Props{
					"width": 20,
				}},
				{"Value", ki.Props{
					"width": 50,
				}},
				{"Exclude", ki.Props{}},
				{"Contains", ki.Props{}},
				{"Ignore Case", ki.Props{}},
			},
		}},
		{"Sequential", ki.Props{
			"label": "Show All",
			"desc":  "show all rows in the table (undo any filtering and sorting)",
			"icon":  "update",
		}},

		{"sep-file", ki.BlankProp{}},
		{"OpenCSV", ki.Props{
			"label": "Open CSV...",
			"icon":  "file-open",
			"desc":  "Open CSV-formatted data (or any delimeter) -- also recognizes emergent-style headers",
			"Args": ki.PropSlice{
				{"File Name", ki.Props{
					"ext": ".tsv,.csv",
				}},
				{"Delimiter", ki.Props{
					"default": Tab,
				}},
			},
		}},
		{"SaveCSV", ki.Props{
			"label": "Save CSV...",
			"icon":  "file-save",
			"desc":  "Save CSV-formatted data (or any delimiter) -- header outputs emergent-style header data (recommended)",
			"Args": ki.PropSlice{
				{"File Name", ki.Props{
					"ext": ".tsv,.csv",
				}},
				{"Delimiter", ki.Props{
					"default": Tab,
				}},
				{"Headers", ki.Props{
					"default": true,
					"desc":    "output C++ emergent-style headers that have type and tensor geometry information",
				}},
			},
		}},
	},
}
View Source 
var KiT_Delims = kit.Enums.AddEnum(DelimsN, kit.NotBitFlag, nil)
View Source 
var KiT_IdxView = kit.Types.AddType(&IdxView{}, IdxViewProps)
View Source 
var KiT_Table = kit.Types.AddType(&Table{}, TableProps)
View Source 
var TableProps = ki.Props{
	"ToolBar": ki.PropSlice{
		{"AddRows", ki.Props{
			"icon": "plus",
			"Args": ki.PropSlice{
				{"N Rows", ki.Props{
					"default": 1,
				}},
			},
		}},
		{"SetNumRows", ki.Props{
			"label": "Set N Rows",
			"icon":  "new",
			"Args": ki.PropSlice{
				{"N Rows", ki.Props{
					"default-field": "Rows",
				}},
			},
		}},
		{"sep-file", ki.BlankProp{}},
		{"OpenCSV", ki.Props{
			"label": "Open CSV...",
			"icon":  "file-open",
			"desc":  "Open CSV-formatted data (or any delimeter) -- also recognizes emergent-style headers",
			"Args": ki.PropSlice{
				{"File Name", ki.Props{
					"ext": ".tsv,.csv",
				}},
				{"Delimiter", ki.Props{
					"default": Tab,
				}},
			},
		}},
		{"SaveCSV", ki.Props{
			"label": "Save CSV...",
			"icon":  "file-save",
			"desc":  "Save CSV-formatted data (or any delimiter) -- header outputs emergent-style header data (recommended)",
			"Args": ki.PropSlice{
				{"File Name", ki.Props{
					"ext": ".tsv,.csv",
				}},
				{"Delimiter", ki.Props{
					"default": Tab,
				}},
				{"Headers", ki.Props{
					"default": true,
					"desc":    "output C++ emergent-style headers that have type and tensor geometry information",
				}},
			},
		}},
	},
}

Functions

func DetectEmerHeaders added in v1.0.0 

func DetectEmerHeaders(hdrs []string) bool

DetectEmerHeaders looks for emergent header special characters -- returns true if found

func EmerColType 

func EmerColType(nm string) (etensor.Type, string)

EmerColType parses the column header for type information using the emergent naming convention

func FilterNull 

func FilterNull(et *Table, row int) bool

FilterNull is a FilterFunc that filters out all rows that have a Null value in a 1D (scalar) column, according to the IsNull flag

func InferDataType 

func InferDataType(str string) etensor.Type

InferDataType returns the inferred data type for the given string only deals with float64, int, and string types

func ShapeFromString 

func ShapeFromString(dims string) []int

ShapeFromString parses string representation of shape as N:d,d,..

Types

type Column 

type Column struct {
	Name      string       `desc:"name of column -- must be unique for a table"`
	Type      etensor.Type `desc:"data type, using etensor types which are isomorphic with arrow.Type"`
	CellShape []int        `` /* 168-byte string literal not displayed */
	DimNames  []string     `desc:"names of the dimensions within the CellShape -- 'Row' will be added to outer dimension"`
}

Column specifies everything about a column -- can be used for constructing tables

type Delims added in v1.0.0 

type Delims int32

Delim are standard CSV delimiter options (Tab, Comma, Space) 

const (
	// Tab is the tab rune delimiter, for TSV tab separated values
	Tab Delims = iota

	// Comma is the comma rune delimiter, for CSV comma separated values
	Comma

	// Space is the space rune delimiter, for SSV space separated value
	Space

	// Detect is used during reading a file -- reads the first line and detects tabs or commas
	Detect

	DelimsN
)

func (*Delims) FromString added in v1.0.0 

func (i *Delims) FromString(s string) error

func (Delims) MarshalJSON added in v1.0.0 

func (ev Delims) MarshalJSON() ([]byte, error)

func (Delims) Rune added in v1.0.0 

func (dl Delims) Rune() rune

func (Delims) String added in v1.0.0 

func (i Delims) String() string

func (*Delims) UnmarshalJSON added in v1.0.0 

func (ev *Delims) UnmarshalJSON(b []byte) error

type FilterFunc 

type FilterFunc func(et *Table, row int) bool

FilterFunc is a function used for filtering that returns true if Table row should be included in the current filtered view of the table, and false if it should be removed.

type IdxView 

type IdxView struct {
	Table *Table `desc:"Table that we are an indexed view onto"`
	Idxs  []int  `desc:"current indexes into Table"`
	// contains filtered or unexported fields
}

IdxView is an indexed wrapper around an etable.Table that provides a specific view onto the Table defined by the set of indexes. This provides an efficient way of sorting and filtering a table by only updating the indexes while doing nothing to the Table itself. To produce a table that has data actually organized according to the indexed order, call the NewTable method. IdxView views on a table can also be organized together as Splits of the table rows, e.g., by grouping values along a given column.

func NewIdxView 

func NewIdxView(et *Table) *IdxView

NewIdxView returns a new IdxView based on given table, initialized with sequential idxes

func (*IdxView) AddIndex 

func (ix *IdxView) AddIndex(idx int)

AddIndex adds a new index to the list

func (*IdxView) AddRows added in v1.0.0 

func (ix *IdxView) AddRows(n int)

AddRows adds n rows to end of underlying Table, and to the indexes in this view

func (*IdxView) AggCol 

func (ix *IdxView) AggCol(colIdx int, ini float64, fun etensor.AggFunc) []float64

AggCol applies given aggregation function to each element in the given column, using float64 conversions of the values. init is the initial value for the agg variable. Operates independently over each cell on n-dimensional columns and returns the result as a slice of values per cell.

func (*IdxView) Clone 

func (ix *IdxView) Clone() *IdxView

Clone returns a copy of the current index view with its own index memory

func (*IdxView) CopyFrom 

func (ix *IdxView) CopyFrom(oix *IdxView)

CopyFrom copies from given other IdxView (we have our own unique copy of indexes)

func (*IdxView) DeleteInvalid added in v1.0.6 

func (ix *IdxView) DeleteInvalid()

DeleteInvalid deletes all invalid indexes from the list. Call this if rows (could) have been deleted from table.

func (*IdxView) DeleteRows added in v1.0.0 

func (ix *IdxView) DeleteRows(at, n int)

DeleteRows deletes n rows of indexes starting at given index in the list of indexes

func (*IdxView) Filter 

func (ix *IdxView) Filter(filterFunc func(et *Table, row int) bool)

Filter filters the indexes into our Table using given Filter function. The Filter function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes.

func (*IdxView) FilterCol added in v1.0.0 

func (ix *IdxView) FilterCol(colIdx int, str string, exclude, contains, ignoreCase bool)

FilterCol sorts the indexes into our Table according to values in given column index, using string representation of column values. Includes rows with matching values unless exclude is set. If contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity. Only valid for 1-dimensional columns.

func (*IdxView) FilterColName added in v1.0.0 

func (ix *IdxView) FilterColName(colNm string, str string, exclude, contains, ignoreCase bool) error

FilterColName filters the indexes into our Table according to values in given column name, using string representation of column values. Includes rows with matching values unless exclude is set. If contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IdxView) InsertRows added in v1.0.0 

func (ix *IdxView) InsertRows(at, n int)

InsertRows adds n rows to end of underlying Table, and to the indexes starting at given index in this view

func (*IdxView) Len 

func (ix *IdxView) Len() int

Len returns the length of the index list

func (*IdxView) Less 

func (ix *IdxView) Less(i, j int) bool

Less calls the LessFunc for sorting

func (*IdxView) NewTable 

func (ix *IdxView) NewTable() *Table

NewTable returns a new table with column data organized according to the indexes

func (*IdxView) OpenCSV added in v1.0.0 

func (ix *IdxView) OpenCSV(filename gi.FileName, delim Delims) error

OpenCSV reads a table idx view from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. The C++ emergent column headers are parsed -- these have full configuration information for tensor dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.

func (*IdxView) Permuted 

func (ix *IdxView) Permuted()

Permuted sets indexes to a permuted order -- if indexes already exist then existing list of indexes is permuted, otherwise a new set of permuted indexes are generated

func (*IdxView) RowsByString 

func (ix *IdxView) RowsByString(colNm string, str string, contains, ignoreCase bool) []int

RowsByString returns the list of *our indexes* whose row in the table has given string value in given column name (de-reference our indexes to get actual row). if contains, only checks if row contains string; if ignoreCase, ignores case. returns nil if name invalid -- see also Try. Use named args for greater clarity.

func (*IdxView) RowsByStringIdx 

func (ix *IdxView) RowsByStringIdx(colIdx int, str string, contains, ignoreCase bool) []int

RowsByStringIdx returns the list of *our indexes* whose row in the table has given string value in given column index (de-reference our indexes to get actual row). if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.

func (*IdxView) RowsByStringTry 

func (ix *IdxView) RowsByStringTry(colNm string, str string, contains, ignoreCase bool) ([]int, error)

RowsByStringTry returns the list of *our indexes* whose row in the table has given string value in given column name (de-reference our indexes to get actual row). if contains, only checks if row contains string; if ignoreCase, ignores case. returns error message for invalid column name. Use named args for greater clarity.

func (*IdxView) SaveCSV added in v1.0.0 

func (ix *IdxView) SaveCSV(filename gi.FileName, delim Delims, headers bool) error

SaveCSV writes a table idx view to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate C++ emergent-tyle column headers. These headers have full configuration information for the tensor columns. Otherwise, only the data is written.

func (*IdxView) Sequential 

func (ix *IdxView) Sequential()

Sequential sets indexes to sequential row-wise indexes into table

func (*IdxView) SetTable 

func (ix *IdxView) SetTable(et *Table)

SetTable sets as indexes into given table with sequential initial indexes

func (*IdxView) Sort 

func (ix *IdxView) Sort(lessFunc func(et *Table, i, j int) bool)

Sort sorts the indexes into our Table using given Less function. The Less function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes.

func (*IdxView) SortCol 

func (ix *IdxView) SortCol(colIdx int, ascending bool)

SortCol sorts the indexes into our Table according to values in given column index, using either ascending or descending order. Only valid for 1-dimensional columns.

func (*IdxView) SortColName added in v1.0.0 

func (ix *IdxView) SortColName(colNm string, ascending bool) error

SortColName sorts the indexes into our Table according to values in given column name, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IdxView) SortColNames added in v1.0.0 

func (ix *IdxView) SortColNames(colNms []string, ascending bool) error

SortColNames sorts the indexes into our Table according to values in given column names, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IdxView) SortCols 

func (ix *IdxView) SortCols(colIdxs []int, ascending bool)

SortCols sorts the indexes into our Table according to values in given list of column indexes, using either ascending or descending order for all of the columns. Only valid for 1-dimensional columns.

func (*IdxView) SortIdxs added in v1.0.0 

func (ix *IdxView) SortIdxs()

SortIdxs sorts the indexes into our Table directly in numerical order, producing the native ordering, while preserving any filtering that might have occurred.

func (*IdxView) SortStable added in v1.0.10 

func (ix *IdxView) SortStable(lessFunc func(et *Table, i, j int) bool)

SortStable stably sorts the indexes into our Table using given Less function. The Less function operates directly on row numbers into the Table as these row numbers have already been projected through the indexes. It is *essential* that it always returns false when the two are equal for the stable function to actually work.

func (*IdxView) SortStableCol added in v1.0.10 

func (ix *IdxView) SortStableCol(colIdx int, ascending bool)

SortStableCol sorts the indexes into our Table according to values in given column index, using either ascending or descending order. Only valid for 1-dimensional columns.

func (*IdxView) SortStableColName added in v1.0.10 

func (ix *IdxView) SortStableColName(colNm string, ascending bool) error

SortStableColName sorts the indexes into our Table according to values in given column name, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IdxView) SortStableColNames added in v1.0.10 

func (ix *IdxView) SortStableColNames(colNms []string, ascending bool) error

SortStableColNames sorts the indexes into our Table according to values in given column names, using either ascending or descending order. Only valid for 1-dimensional columns. Returns error if column name not found.

func (*IdxView) SortStableCols added in v1.0.10 

func (ix *IdxView) SortStableCols(colIdxs []int, ascending bool)

SortStableCols sorts the indexes into our Table according to values in given list of column indexes, using either ascending or descending order for all of the columns. Only valid for 1-dimensional columns.

func (*IdxView) Swap 

func (ix *IdxView) Swap(i, j int)

Swap switches the indexes for i and j

func (*IdxView) WriteCSV added in v1.0.0 

func (ix *IdxView) WriteCSV(w io.Writer, delim Delims, headers bool) error

WriteCSV writes only rows in table idx view to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate C++ emergent-style column headers. These headers have full configuration information for the tensor columns. Otherwise, only the data is written.

type LessFunc 

type LessFunc func(et *Table, i, j int) bool

LessFunc is a function used for sort comparisons that returns true if Table row i is less than Table row j -- these are the raw row numbers, which have already been projected through indexes when used for sorting via Idxs.

type Schema 

type Schema []Column

Schema specifies all of the columns of a table, sufficient to create the table It is just a slice list of Columns

func SchemaFromEmerHeaders 

func SchemaFromEmerHeaders(hdrs []string) (Schema, error)

SchemaFromEmerHeaders attempts to configure a Table Schema based on emergent DataTable headers

func SchemaFromHeaders 

func SchemaFromHeaders(hdrs []string, rec [][]string) (Schema, error)

SchemaFromHeaders attempts to configure a Table Schema based on the headers for non-Emergent headers, data is examined to

func SchemaFromPlainHeaders 

func SchemaFromPlainHeaders(hdrs []string, rec [][]string) (Schema, error)

SchemaFromPlainHeaders configures a Table Schema based on plain headers. All columns are of type String and must be converted later to numerical types as appropriate.

type SplitAgg 

type SplitAgg struct {
	Name   string      `desc:"the name of the aggregation operation performed, e.g., Sum, Mean, etc"`
	ColIdx int         `desc:"column index on which the aggregation was performed -- results will have same shape as cells in this column"`
	Aggs   [][]float64 `desc:"aggregation results -- outer index is length of splits, inner is the length of the cell shape for the column"`
}

SplitAgg contains aggregation results for splits

func (*SplitAgg) Clone 

func (sa *SplitAgg) Clone() *SplitAgg

Clone returns a cloned copy of our SplitAgg

func (*SplitAgg) CopyFrom 

func (sa *SplitAgg) CopyFrom(osa *SplitAgg)

CopyFrom copies from other SplitAgg -- we get our own unique copy of everything

type Splits 

type Splits struct {
	Splits []*IdxView  `desc:"the list of index views for each split"`
	Levels []string    `` /* 418-byte string literal not displayed */
	Values [][]string  `` /* 155-byte string literal not displayed */
	Aggs   []*SplitAgg `` /* 261-byte string literal not displayed */
	// contains filtered or unexported fields
}

Splits is a list of indexed views into a given Table, that represent a particular way of splitting up the data, e.g., whenever a given column value changes.

It is functionally equivalent to the MultiIndex in python's pandas: it has multiple levels of indexes as listed in the Levels field, which then have corresponding Values for each split. These index levels can be re-ordered, and new Splits or IdxViews's can be created from subsets of the existing levels. The Values are stored simply as string values, as this is the most general type and often index values are labels etc.

For Splits created by the splits.GroupBy function for example, each index Level is the column name that the data was grouped by, and the Values for each split are then the values of those columns. However, any arbitrary set of levels and values can be used, e.g., as in the splits.GroupByFunc function.

Conceptually, a given Split always contains the full "outer product" of all the index levels -- there is one split for each unique combination of values along each index level. Thus, removing one level collapses across those values and moves the corresponding indexes into the remaining split indexes.

You can Sort and Filter based on the index values directly, to reorganize the splits and drop particular index values, etc.

Splits also maintains Aggs aggregate values for each split, which can be computed using standard aggregation methods over data columns, using the split.Agg* functions.

The etable code contains the structural methods for managing the Splits data. See split package for end-user methods to generate different kinds of splits, and perform aggregations, etc.

func (*Splits) AddAgg 

func (spl *Splits) AddAgg(name string, colIdx int) *SplitAgg

AddAgg adds a new set of aggregation results for the Splits

func (*Splits) AggByColName 

func (spl *Splits) AggByColName(name string) *SplitAgg

AggByColName returns Agg results for given column name, optionally including :Name agg name appended, where Name is the name given to the Agg result (e.g., Mean for a standard Mean agg). Returns nil if not found. See also Try version for error message.

func (*Splits) AggByColNameTry 

func (spl *Splits) AggByColNameTry(name string) (*SplitAgg, error)

AggByColNameTry returns Agg results for given column name, optionally including :Name agg name appended, where Name is the name given to the Agg result (e.g., Mean for a standard Mean agg). Returns error message if not found.

func (*Splits) AggByName 

func (spl *Splits) AggByName(name string) *SplitAgg

AggByName returns Agg results for given name, which does NOT include the column name, just the name given to the Agg result (e.g., Mean for a standard Mean agg). See also AggByColName. Returns nil if not found. See also Try version for error message.

func (*Splits) AggByNameTry 

func (spl *Splits) AggByNameTry(name string) (*SplitAgg, error)

AggByNameTry returns Agg results for given name, which does NOT include the column name, just the name given to the Agg result (e.g., Mean for a standard Mean agg). See also AggByColName. Returns error message if not found.

func (*Splits) AggsToTable 

func (spl *Splits) AggsToTable(colName bool) *Table

AggsToTable returns a Table containing this Splits' aggregate data. Must have Levels and Aggs all created as in the split.Agg* methods. if colName == ColNameOnly, then the name of the columns for the Table is just the corresponding agg column name -- otherwise it also includes the name of the aggregation function with a : divider (e.g., Name:Mean)

func (*Splits) AggsToTableCopy added in v1.0.0 

func (spl *Splits) AggsToTableCopy(colName bool) *Table

AggsToTableCopy returns a Table containing this Splits' aggregate data and a copy of the first row of data for each split for all non-agg cols, which is useful for recording other data that goes along with aggregated values. Must have Levels and Aggs all created as in the split.Agg* methods. if colName == ColNameOnly, then the name of the columns for the Table is just the corresponding agg column name -- otherwise it also includes the name of the aggregation function with a : divider (e.g., Name:Mean)

func (*Splits) ByValue 

func (spl *Splits) ByValue(values []string) []int

ByValue finds split indexes by matching to split values, returns nil if not found. values are used in order as far as they go and any remaining values are assumed to match, and any empty values will match anything. Can use this to access different subgroups within overall set of splits.

func (*Splits) Clone 

func (spl *Splits) Clone() *Splits

Clone returns a cloned copy of our splits

func (*Splits) CopyFrom 

func (spl *Splits) CopyFrom(osp *Splits)

CopyFrom copies from other Splits -- we get our own unique copy of everything

func (*Splits) Delete 

func (spl *Splits) Delete(idx int)

Delete deletes split at given index -- use this to coordinate deletion of Splits, Values, and Aggs values for given split

func (*Splits) DeleteAggs 

func (spl *Splits) DeleteAggs()

DeleteAggs deletes all existing aggregation data

func (*Splits) ExtractLevels 

func (spl *Splits) ExtractLevels(levels []int) (*Splits, error)

ExtractLevels returns a new Splits that only has the given levels of indexes, in their given order, with the other levels removed and their corresponding indexes merged into the appropriate remaining levels. Any existing aggregation data is not retained in the new splits.

func (*Splits) Filter 

func (spl *Splits) Filter(fun func(idx int) bool)

Filter removes any split for which given function returns false

func (*Splits) Len 

func (spl *Splits) Len() int

Len returns number of splits

func (*Splits) Less 

func (spl *Splits) Less(i, j int) bool

Less calls the LessFunc for sorting

func (*Splits) New 

func (spl *Splits) New(dt *Table, values []string, rows ...int) *IdxView

New adds a new split to the list for given table, and with associated values, which are copied before saving into Values list, and any number of rows from the table associated with this split (also copied). Any existing Aggs are deleted by this.

func (*Splits) ReorderLevels 

func (spl *Splits) ReorderLevels(order []int) error

ReorderLevels re-orders the index levels according to the given new ordering indexes e.g., []int{1,0} will move the current level 0 to level 1, and 1 to level 0 no checking is done to ensure these are sensible beyond basic length test -- behavior undefined if so. Typically you want to call SortLevels after this.

func (*Splits) SetLevels 

func (spl *Splits) SetLevels(levels ...string)

SetLevels sets the Levels index names -- must match actual index dimensionality of the Values. This is automatically done by e.g., GroupBy, but must be done manually if creating custom indexes.

func (*Splits) Sort 

func (spl *Splits) Sort(lessFunc func(spl *Splits, i, j int) bool)

Sort sorts the splits according to the given Less function.

func (*Splits) SortLevels 

func (spl *Splits) SortLevels()

SortLevels sorts the splits according to the current index level ordering of values i.e., first index level is outer sort dimension, then within that is the next, etc

func (*Splits) SortOrder 

func (spl *Splits) SortOrder(order []int) error

SortOrder sorts the splits according to the given ordering of index levels which can be a subset as well

func (*Splits) Swap 

func (spl *Splits) Swap(i, j int)

Swap switches the indexes for i and j

func (*Splits) Table 

func (spl *Splits) Table() *Table

Table returns the table from the first split (should be same for all) returns nil if no splits yet

type SplitsLessFunc 

type SplitsLessFunc func(spl *Splits, i, j int) bool

SplitsLessFunc is a function used for sort comparisons that returns true if split i is less than split j

type Table 

type Table struct {
	Cols       []etensor.Tensor  `view:"no-inline" desc:"columns of data, as etensor.Tensor tensors"`
	ColNames   []string          `desc:"the names of the columns"`
	Rows       int               `inactive:"+" desc:"number of rows, which is enforced to be the size of the outer-most dimension of the column tensors"`
	ColNameMap map[string]int    `view:"-" desc:"the map of column names to column numbers"`
	MetaData   map[string]string `` /* 435-byte string literal not displayed */
}

etable.Table is the emer DataTable structure, containing columns of etensor tensors. All tensors MUST have RowMajor stride layout!

func New 

func New(sc Schema, rows int) *Table

New returns a new Table constructed from given Schema. The actual tensor number of rows is enforced to be > 0, because we cannot have a null dimension in tensor shape

func NewTable 

func NewTable(name string) *Table

func (*Table) AddCol 

func (dt *Table) AddCol(tsr etensor.Tensor, name string) error

AddCol adds the given tensor as a column to the table. returns error if it is not a RowMajor organized tensor, and automatically adjusts the shape to fit the current number of rows.

func (*Table) AddRows 

func (dt *Table) AddRows(n int)

AddRows adds n rows to each of the columns

func (*Table) AppendRows added in v1.0.1 

func (dt *Table) AppendRows(dt2 *Table)

AppendRows appends shared columns in both tables with input table rows

func (*Table) CellFloat 

func (dt *Table) CellFloat(colNm string, row int) float64

CellFloat returns the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns NaN if column is not a 1-dimensional tensor or col name not found, or row not valid.

func (*Table) CellFloatIdx 

func (dt *Table) CellFloatIdx(col, row int) float64

CellFloatIdx returns the float64 value of cell at given column, row index for columns that have 1-dimensional tensors. Returns NaN if column is not a 1-dimensional tensor or row not valid.

func (*Table) CellFloatTry 

func (dt *Table) CellFloatTry(colNm string, row int) (float64, error)

CellFloatTry returns the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor, or row not valid.

func (*Table) CellString 

func (dt *Table) CellString(colNm string, row int) string

CellString returns the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns "" if column is not a 1-dimensional tensor or row not valid.

func (*Table) CellStringIdx 

func (dt *Table) CellStringIdx(col, row int) string

CellStringIdx returns the string value of cell at given column, row index for columns that have 1-dimensional tensors. Returns "" if column is not a 1-dimensional tensor or row not valid.

func (*Table) CellStringTry 

func (dt *Table) CellStringTry(colNm string, row int) (string, error)

CellStringTry returns the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor, or row not valid.

func (*Table) CellTensor 

func (dt *Table) CellTensor(colNm string, row int) etensor.Tensor

CellTensor returns the tensor SubSpace for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns nil on any error -- see Try version for error returns.

func (*Table) CellTensorFloat1D added in v1.0.0 

func (dt *Table) CellTensorFloat1D(colNm string, row int, idx int) float64

CellTensorFloat1D returns the float value of a Tensor cell's cell at given 1D offset within cell, for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns 0 on any error -- see Try version for error returns.

func (*Table) CellTensorFloat1DTry added in v1.0.0 

func (dt *Table) CellTensorFloat1DTry(colNm string, row int, idx int) (float64, error)

CellTensorFloat1DTry returns the float value of a Tensor cell's cell at given 1D offset within cell, for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns any error.

func (*Table) CellTensorIdx 

func (dt *Table) CellTensorIdx(col, row int) etensor.Tensor

CellTensorIdx returns the tensor SubSpace for given column, row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns nil if column is a 1-dimensional tensor or there is any error from the etensor.Tensor.SubSpace call.

func (*Table) CellTensorTry 

func (dt *Table) CellTensorTry(colNm string, row int) (etensor.Tensor, error)

CellTensorTry returns the tensor SubSpace for given column (by name), row index for columns that have higher-dimensional tensors so each row is represented by an n-1 dimensional tensor, with the outer dimension being the row number. Returns an error if column is a 1-dimensional tensor or any error from the etensor.Tensor.SubSpace call.

func (*Table) Clone added in v1.0.0 

func (dt *Table) Clone() *Table

Clone returns a complete copy of this table

func (*Table) Col 

func (dt *Table) Col(i int) etensor.Tensor

Col returns the tensor at given column index

func (*Table) ColByName 

func (dt *Table) ColByName(name string) etensor.Tensor

ColByName returns the tensor at given column name without any error messages -- just returns nil if not found

func (*Table) ColByNameTry 

func (dt *Table) ColByNameTry(name string) (etensor.Tensor, error)

ColByNameTry returns the tensor at given column name, if not found, returns error

func (*Table) ColIdx 

func (dt *Table) ColIdx(name string) int

ColIdx returns the index of the given column name. returns -1 if name not found -- see Try version for error message.

func (*Table) ColIdxTry 

func (dt *Table) ColIdxTry(name string) (int, error)

ColIdxTry returns the index of the given column name, along with an error if not found.

func (*Table) ColIdxsByNames 

func (dt *Table) ColIdxsByNames(names []string) []int

ColIdxsByNames returns the indexes of the given column names. idxs have -1 if name not found -- see Try version for error message.

func (*Table) ColIdxsByNamesTry 

func (dt *Table) ColIdxsByNamesTry(names []string) ([]int, error)

ColsIdxsByNamesTry returns the indexes of the given column names, along with an error if any not found.

func (*Table) ColName 

func (dt *Table) ColName(i int) string

ColName returns the name of given column

func (*Table) CopyCell added in v1.0.0 

func (dt *Table) CopyCell(colNm string, row int, cpt *Table, cpColNm string, cpRow int) error

CopyCell copies into cell at given col, row from cell in other table. It is robust to differences in type -- uses destination cell type. Returns error if column names are invalid.

func (*Table) CopyMetaDataFrom added in v1.0.0 

func (dt *Table) CopyMetaDataFrom(cp *Table)

CopyMetaDataFrom copies meta data from other table

func (*Table) DeleteAll added in v1.0.39 

func (dt *Table) DeleteAll()

DeleteAll deletes all columns -- full reset

func (*Table) DeleteColIdx added in v1.0.9 

func (dt *Table) DeleteColIdx(idx int)

DeleteColIdx deletes column of given index

func (*Table) DeleteColName added in v1.0.9 

func (dt *Table) DeleteColName(name string) error

DeleteColName deletes column of given name.

func (*Table) EmerHeaders 

func (dt *Table) EmerHeaders() []string

EmerHeaders generates emergent DataTable header strings from the table. These have full information about type and tensor cell dimensionality.

func (*Table) IsValidRow 

func (dt *Table) IsValidRow(row int) bool

IsValidRow returns true if the row is valid

func (*Table) IsValidRowTry 

func (dt *Table) IsValidRowTry(row int) error

IsValidRowTry returns an error message if the row is not valid.

func (*Table) NumCols 

func (dt *Table) NumCols() int

NumCols returns the number of columns (arrow / dframe api)

func (*Table) NumRows 

func (dt *Table) NumRows() int

NumRows returns the number of rows (arrow / dframe api)

func (*Table) OpenCSV 

func (dt *Table) OpenCSV(filename gi.FileName, delim Delims) error

OpenCSV reads a table from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. The C++ emergent column headers are parsed -- these have full configuration information for tensor dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.

func (*Table) ReadCSV 

func (dt *Table) ReadCSV(r io.Reader, delim Delims) error

ReadCSV reads a table from a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg), using the Go standard encoding/csv reader conforming to the official CSV standard. If the table does not currently have any columns, the first row of the file is assumed to be headers, and columns are constructed therefrom. The C++ emergent column headers are parsed -- these have full configuration information for tensor dimensionality. If the table DOES have existing columns, then those are used robustly for whatever information fits from each row of the file.

func (*Table) ReadCSVRow added in v1.0.0 

func (dt *Table) ReadCSVRow(rec []string, row int)

ReadCSVRow reads a record of CSV data into given row in table

func (*Table) RowsByString 

func (dt *Table) RowsByString(colNm string, str string, contains, ignoreCase bool) []int

RowsByString returns the list of rows that have given string value in given column name. returns nil if name invalid -- see also Try. if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.

func (*Table) RowsByStringIdx 

func (dt *Table) RowsByStringIdx(colIdx int, str string, contains, ignoreCase bool) []int

RowsByStringIdx returns the list of rows that have given string value in given column index. if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.

func (*Table) RowsByStringTry 

func (dt *Table) RowsByStringTry(colNm string, str string, contains, ignoreCase bool) ([]int, error)

RowsByStringTry returns the list of rows that have given string value in given column name. returns error message for invalid column name. if contains, only checks if row contains string; if ignoreCase, ignores case. Use named args for greater clarity.

func (*Table) SaveCSV 

func (dt *Table) SaveCSV(filename gi.FileName, delim Delims, headers bool) error

SaveCSV writes a table to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate C++ emergent-tyle column headers. These headers have full configuration information for the tensor columns. Otherwise, only the data is written.

func (*Table) Schema 

func (dt *Table) Schema() Schema

Schema returns the Schema (column properties) for this table

func (*Table) SetCellFloat 

func (dt *Table) SetCellFloat(colNm string, row int, val float64) bool

SetCellFloat sets the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors.

func (*Table) SetCellFloatIdx 

func (dt *Table) SetCellFloatIdx(col, row int, val float64) bool

SetCellFloatIdx sets the float64 value of cell at given column, row index for columns that have 1-dimensional tensors. Returns true if set.

func (*Table) SetCellFloatTry 

func (dt *Table) SetCellFloatTry(colNm string, row int, val float64) error

SetCellFloatTry sets the float64 value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor.

func (*Table) SetCellString 

func (dt *Table) SetCellString(colNm string, row int, val string) bool

SetCellString sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns true if set.

func (*Table) SetCellStringIdx 

func (dt *Table) SetCellStringIdx(col, row int, val string) bool

SetCellStringIdx sets the string value of cell at given column, row index for columns that have 1-dimensional tensors. Returns true if set.

func (*Table) SetCellStringTry 

func (dt *Table) SetCellStringTry(colNm string, row int, val string) error

SetCellStringTry sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor.

func (*Table) SetCellTensor 

func (dt *Table) SetCellTensor(colNm string, row int, val etensor.Tensor) bool

SetCellTensor sets the tensor value of cell at given column (by name), row index for columns that have n-dimensional tensors. Returns true if set.

func (*Table) SetCellTensorFloat1D added in v1.0.0 

func (dt *Table) SetCellTensorFloat1D(colNm string, row int, idx int, val float64) bool

SetCellTensorFloat1D sets the tensor cell's float cell value at given 1D index within cell, at given column (by name), row index for columns that have n-dimensional tensors. Returns true if set.

func (*Table) SetCellTensorFloat1DTry added in v1.0.0 

func (dt *Table) SetCellTensorFloat1DTry(colNm string, row int, idx int, val float64) error

SetCellTensorFloat1DTry sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor.

func (*Table) SetCellTensorIdx 

func (dt *Table) SetCellTensorIdx(col, row int, val etensor.Tensor) bool

SetCellTensorIdx sets the tensor value of cell at given column, row index for columns that have n-dimensional tensors. Returns true if set.

func (*Table) SetCellTensorTry 

func (dt *Table) SetCellTensorTry(colNm string, row int, val etensor.Tensor) error

SetCellTensorTry sets the string value of cell at given column (by name), row index for columns that have 1-dimensional tensors. Returns an error if column not found, or column is not a 1-dimensional tensor.

func (*Table) SetFromSchema 

func (dt *Table) SetFromSchema(sc Schema, rows int)

SetFromSchema configures table from given Schema. The actual tensor number of rows is enforced to be > 0, because we cannot have a null dimension in tensor shape. does not preserve any existing columns / data.

func (*Table) SetMetaData 

func (dt *Table) SetMetaData(key, val string)

SetMetaData sets given meta-data key to given value, safely creating the map if not yet initialized. Standard Keys are: * name -- name of table * desc -- description of table * read-only -- makes gui read-only (inactive edits) for etview.TableView * ColName:* -- prefix for all column-specific meta-data

  • desc -- description of column

func (*Table) SetNumRows 

func (dt *Table) SetNumRows(rows int)

SetNumRows sets the number of rows in the table, across all columns if rows = 0 then effective number of rows in tensors is 1, as this dim cannot be 0

func (*Table) UpdateColNameMap 

func (dt *Table) UpdateColNameMap()

UpdateColNameMap updates the column name map

func (*Table) WriteCSV 

func (dt *Table) WriteCSV(w io.Writer, delim Delims, headers bool) error

WriteCSV writes a table to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). If headers = true then generate C++ emergent-style column headers. These headers have full configuration information for the tensor columns. Otherwise, only the data is written.

func (*Table) WriteCSVHeaders 

func (dt *Table) WriteCSVHeaders(w io.Writer, delim Delims) (int, error)

WriteCSVHeaders writes headers to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg). Returns number of columns in header

func (*Table) WriteCSVRow 

func (dt *Table) WriteCSVRow(w io.Writer, row int, delim Delims) error

WriteCSVRow writes given row to a comma-separated-values (CSV) file (where comma = any delimiter, specified in the delim arg)

func (*Table) WriteCSVRowWriter 

func (dt *Table) WriteCSVRowWriter(cw *csv.Writer, row int, ncol int) error

WriteCSVRowWriter uses csv.Writer to write one row

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.