datafs

README

datafs: a virtual filesystem for data

TODO: write docs

Constraints

• no pointers -- GPU does not like pointers -- use Set / As accessors
• names within directory must be unique

Documentation

Index

• func NewTable(dir *Data, names ...string) (*table.Table, error)
• func NewTensor[T string | bool | float32 | float64 | int | int32 | byte](dir *Data, name string, sizes []int, names ...string) (tensor.Tensor, error)
• func PlotColumnZeroOne() *plotcore.ColumnOptions
• type Data
  • func New[T any](dir *Data, names ...string) (*Data, error)
  • func NewData(dir *Data, name string) (*Data, error)
  • func NewDir(name string, parent ...*Data) (*Data, error)
  • func (d *Data) Add(it *Data) error
  • func (d *Data) AsFloat32() (float32, bool)
  • func (d *Data) AsFloat64() (float64, bool)
  • func (d *Data) AsInt() (int, bool)
  • func (d *Data) AsString() (string, bool)
  • func (d *Data) AsTable() *table.Table
  • func (d *Data) AsTensor() tensor.Tensor
  • func (d *Data) Bytes() []byte
  • func (d *Data) Calc() error
  • func (d *Data) CalcAll() error
  • func (d *Data) DataType() reflect.Kind
  • func (d *Data) DirAtPath(dir string) (*Data, error)
  • func (d *Data) DirNamesAlpha() []string
  • func (d *Data) DirNamesByTime() []string
  • func (d *Data) DirTable(fun func(item *Data) bool) *table.Table
  • func (d *Data) FlatItemsByTimeFunc(fun func(item *Data) bool) []*Data
  • func (d *Data) FlatItemsFunc(fun func(item *Data) bool) []*Data
  • func (d *Data) Info() (fs.FileInfo, error)
  • func (d *Data) IsDir() bool
  • func (d *Data) IsNumeric() bool
  • func (d *Data) Item(name string) *Data
  • func (d *Data) Items(names ...string) ([]*Data, error)
  • func (d *Data) ItemsByTimeFunc(fun func(item *Data) bool) []*Data
  • func (d *Data) ItemsFunc(fun func(item *Data) bool) []*Data
  • func (d *Data) KnownFileInfo() fileinfo.Known
  • func (d *Data) Mkdir(name string) (*Data, error)
  • func (d *Data) ModTime() time.Time
  • func (d *Data) Mode() fs.FileMode
  • func (d *Data) Name() string
  • func (d *Data) Open(name string) (fs.File, error)
  • func (d *Data) Path() string
  • func (d *Data) PlotColumnOptions() *plotcore.ColumnOptions
  • func (d *Data) ReadDir(dir string) ([]fs.DirEntry, error)
  • func (d *Data) ReadFile(name string) ([]byte, error)
  • func (d *Data) SetCalcFunc(fun func() error)
  • func (d *Data) SetFloat32(v float32) bool
  • func (d *Data) SetFloat64(v float64) bool
  • func (d *Data) SetInt(v int) bool
  • func (d *Data) SetMetaItems(key string, value any, names ...string) error
  • func (d *Data) SetPlotColumnOptions(opts *plotcore.ColumnOptions, names ...string) error
  • func (d *Data) SetString(v string) bool
  • func (d *Data) Size() int64
  • func (d *Data) Stat(name string) (fs.FileInfo, error)
  • func (d *Data) Sub(dir string) (fs.FS, error)
  • func (d *Data) Sys() any
  • func (d *Data) Type() fs.FileMode
• type DirFile
  • func (f *DirFile) Close() error
  • func (f *DirFile) ReadDir(n int) ([]fs.DirEntry, error)
• type File
  • func (f *File) Close() error
  • func (f *File) Stat() (fs.FileInfo, error)
• type Sizer

Functions

func NewTable

func NewTable(dir *Data, names ...string) (*table.Table, error)

NewTable makes new table.Table(s) in given directory, for given name(s) (at least one is required). All names must be unique in the directory. Returns the first table created, for immediate use of one item.

func NewTensor

func NewTensor[T string | bool | float32 | float64 | int | int32 | byte](dir *Data, name string, sizes []int, names ...string) (tensor.Tensor, error)

NewTensor returns a new Tensor of given data type, shape sizes, and optional dimension names, in given directory Data item. The name must be unique in the directory.

func PlotColumnZeroOne

func PlotColumnZeroOne() *plotcore.ColumnOptions

PlotColumnZeroOne returns plot options with a fixed 0-1 range

Types

type Data

type Data struct {
	// Parent is the parent data directory
	Parent *Data

	// Meta has metadata, including standardized support for
	// plotting options, compute functions.
	Meta metadata.Data

	// Value is the underlying value of data;
	// is a map[string]*Data for directories.
	Value any
	// contains filtered or unexported fields
}

Data is a single item of data, the "file" or "directory" in the data filesystem.

func New

func New[T any](dir *Data, names ...string) (*Data, error)

New adds new data item(s) of given basic type to given directory, with given name(s) (at least one is required). Values are initialized to zero value for type. All names must be unique in the directory. Returns the first item created, for immediate use of one value.

func NewData

func NewData(dir *Data, name string) (*Data, error)

NewData returns a new Data item in given directory Data item, which can be nil. If not a directory, an error will be generated. The modTime is automatically set to now, and can be used for sorting by order created. The name must be unique within parent.

func NewDir

func NewDir(name string, parent ...*Data) (*Data, error)

NewDir returns a new datafs directory with given name. if parent != nil and a directory, this dir is added to it. if name is empty, then it is set to "root", the root directory. Note that "/" is not allowed for the root directory in Go fs. Names must be unique within a directory.

func (*Data) Add

func (d *Data) Add(it *Data) error

Add adds an item to this directory data item. The only errors are if this item is not a directory, or the name already exists. Names must be unique within a directory.

func (*Data) AsFloat32

func (d *Data) AsFloat32() (float32, bool)

AsFloat32 returns data as a float32 if it is a scalar value that can be so converted. Returns false if not.

func (*Data) AsFloat64

func (d *Data) AsFloat64() (float64, bool)

AsFloat64 returns data as a float64 if it is a scalar value that can be so converted. Returns false if not.

func (*Data) AsInt

func (d *Data) AsInt() (int, bool)

AsInt returns data as a int if it is a scalar value that can be so converted. Returns false if not.

func (*Data) AsString

func (d *Data) AsString() (string, bool)

AsString returns data as a string if it is a scalar value that can be so converted. Returns false if not.

func (*Data) AsTable

func (d *Data) AsTable() *table.Table

AsTable returns the data as a table if it is one, else nil.

func (*Data) AsTensor

func (d *Data) AsTensor() tensor.Tensor

AsTensor returns the data as a tensor if it is one, else nil.

func (*Data) Bytes

func (d *Data) Bytes() []byte

Bytes returns the byte-wise representation of the data Value. This is the actual underlying data, so make a copy if it can be unintentionally modified or retained more than for immediate use.

func (*Data) Calc

func (d *Data) Calc() error

Calc calls function set by Data.SetCalcFunc to compute an updated Value for this data item. Returns an error if func not set, or any error from func itself. Function is stored as CalcFunc in Metadata.

func (*Data) CalcAll

func (d *Data) CalcAll() error

CalcAll calls function set by Data.SetCalcFunc for all items in this directory and all of its subdirectories. Calls Calc on items from FlatItemsByTimeFunc(nil)

func (*Data) DataType

func (d *Data) DataType() reflect.Kind

DataType returns the type of the data elements in the tensor. Bool is returned for the Bits tensor type.

func (*Data) DirAtPath

func (d *Data) DirAtPath(dir string) (*Data, error)

DirAtPath returns directory at given relative path from this starting dir.

func (*Data) DirNamesAlpha

func (d *Data) DirNamesAlpha() []string

DirNamesAlpha returns the names of items in the directory sorted alphabetically. Data must be dir by this point.

func (*Data) DirNamesByTime

func (d *Data) DirNamesByTime() []string

DirNamesByTime returns the names of items in the directory sorted by modTime (order added). Data must be dir by this point.

func (*Data) DirTable

func (d *Data) DirTable(fun func(item *Data) bool) *table.Table

DirTable returns a table.Table for this directory item, with columns as the Tensor elements in the directory and any subdirectories, from FlatItemsByTimeFunc using given filter function. This is a convenient mechanism for creating a plot of all the data in a given directory. If such was previously constructed, it is returned from "DirTable" Metadata key where the table is stored. Row count is updated to current max row. Delete that key to reconstruct if items have changed.

func (*Data) FlatItemsByTimeFunc

func (d *Data) FlatItemsByTimeFunc(fun func(item *Data) bool) []*Data

FlatItemsByTimeFunc returns all "leaf" (non directory) data items in given directory, recursively descending into directories to return a flat list of the entire subtree, filtered by given function, in time order (i.e., order added). The function can filter out directories to prune the tree. If func is nil, all items are returned.

func (*Data) FlatItemsFunc

func (d *Data) FlatItemsFunc(fun func(item *Data) bool) []*Data

FlatItemsFunc returns all "leaf" (non directory) data items in given directory, recursively descending into directories to return a flat list of the entire subtree, filtered by given function, in alpha order. The function can filter out directories to prune the tree. If func is nil, all items are returned.

func (*Data) Info

func (d *Data) Info() (fs.FileInfo, error)

func (*Data) IsDir

func (d *Data) IsDir() bool

func (*Data) IsNumeric

func (d *Data) IsNumeric() bool

IsNumeric returns true if the [DataType] is a basic scalar numerical value, e.g., float32, int, etc.

func (*Data) Item

func (d *Data) Item(name string) *Data

Item returns data item in given directory by name. This is for fast access and direct usage of known items, and it will crash if item is not found or this data is not a directory.

func (*Data) Items

func (d *Data) Items(names ...string) ([]*Data, error)

Items returns data items in given directory by name. error reports any items not found, or if not a directory.

func (*Data) ItemsByTimeFunc

func (d *Data) ItemsByTimeFunc(fun func(item *Data) bool) []*Data

ItemsByTimeFunc returns data items in given directory filtered by given function, in time order (i.e., order added). If func is nil, all items are returned. Any directories within this directory are returned, unless specifically filtered.

func (*Data) ItemsFunc

func (d *Data) ItemsFunc(fun func(item *Data) bool) []*Data

ItemsFunc returns data items in given directory filtered by given function, in alpha order. If func is nil, all items are returned. Any directories within this directory are returned, unless specifically filtered.

func (*Data) KnownFileInfo

func (d *Data) KnownFileInfo() fileinfo.Known

func (*Data) Mkdir

func (d *Data) Mkdir(name string) (*Data, error)

Mkdir creates a new directory with the specified name. The only error is if this item is not a directory.

func (*Data) ModTime

func (d *Data) ModTime() time.Time

func (*Data) Mode

func (d *Data) Mode() fs.FileMode

func (*Data) Name

func (d *Data) Name() string

func (*Data) Open

func (d *Data) Open(name string) (fs.File, error)

Open opens the given data Value within this datafs filesystem.

func (*Data) Path

func (d *Data) Path() string

Path returns the full path to this data item

func (*Data) PlotColumnOptions

func (d *Data) PlotColumnOptions() *plotcore.ColumnOptions

PlotColumnOptions returns plotting options if they have been set, else nil.

func (*Data) ReadDir

func (d *Data) ReadDir(dir string) ([]fs.DirEntry, error)

ReadDir returns the contents of the given directory within this filesystem. Use "." (or "") to refer to the current directory.

func (*Data) ReadFile

func (d *Data) ReadFile(name string) ([]byte, error)

ReadFile reads the named file and returns its contents. A successful call returns a nil error, not io.EOF. (Because ReadFile reads the whole file, the expected EOF from the final Read is not treated as an error to be reported.)

The caller is permitted to modify the returned byte slice. This method should return a copy of the underlying data.

func (*Data) SetCalcFunc

func (d *Data) SetCalcFunc(fun func() error)

SetCalcFunc sets a function to compute an updated Value for this data item. Function is stored as CalcFunc in Metadata. Can be called by Data.Calc method.

func (*Data) SetFloat32

func (d *Data) SetFloat32(v float32) bool

SetFloat32 sets data from given float32 if it is a scalar value that can be so set. Returns false if not.

func (*Data) SetFloat64

func (d *Data) SetFloat64(v float64) bool

SetFloat64 sets data from given float64 if it is a scalar value that can be so set. Returns false if not.

func (*Data) SetInt

func (d *Data) SetInt(v int) bool

SetInt sets data from given int if it is a scalar value that can be so set. Returns false if not.

func (*Data) SetMetaItems

func (d *Data) SetMetaItems(key string, value any, names ...string) error

SetMetaItems sets given metadata for items in given directory with given names. Returns error for any items not found.

func (*Data) SetPlotColumnOptions

func (d *Data) SetPlotColumnOptions(opts *plotcore.ColumnOptions, names ...string) error

SetPlotColumnOptions sets given plotting options for named items within this directory (stored in Metadata).

func (*Data) SetString

func (d *Data) SetString(v string) bool

SetString sets data from given string if it is a scalar value that can be so set. Returns false if not.

func (*Data) Size

func (d *Data) Size() int64

Size returns the size of known data Values, or it uses the Sizer interface, otherwise returns 0.

func (*Data) Stat

func (d *Data) Stat(name string) (fs.FileInfo, error)

Stat returns a FileInfo describing the file. If there is an error, it should be of type *PathError.

func (*Data) Sub

func (d *Data) Sub(dir string) (fs.FS, error)

Sub returns a data FS corresponding to the subtree rooted at dir.

func (*Data) Sys

func (d *Data) Sys() any

Sys returns the metadata for Value

func (*Data) Type

func (d *Data) Type() fs.FileMode

type DirFile

type DirFile struct {
	File
	// contains filtered or unexported fields
}

DirFile represents a directory data item for reading, as fs.ReadDirFile.

func (*DirFile) Close

func (f *DirFile) Close() error

func (*DirFile) ReadDir

func (f *DirFile) ReadDir(n int) ([]fs.DirEntry, error)

type File

type File struct {
	bytes.Reader
	Data *Data
	// contains filtered or unexported fields
}

File represents a data item for reading, as an fs.File. All io functionality is handled by bytes.Reader.

func (*File) Close

func (f *File) Close() error

func (*File) Stat

func (f *File) Stat() (fs.FileInfo, error)

type Sizer

type Sizer interface {
	Sizeof() int64
}

Sizer is an interface to allow an arbitrary data Value to report its size in bytes. Size is automatically computed for known basic data Values supported by datafs directly.