pixidb

package module
v0.0.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 5, 2024 License: MIT Imports: 14 Imported by: 0

README

pixidb

GoReportCard test

PixiDB is a pure Go library and database supporting queries over dense (spatial) pixelized data sets.

Documentation

Index

Constants

View Source 
const (
	DataFileExt     = ".dat"
	MetadataFileExt = ".meta.json"
	MaxPagesInCache = 64
)
View Source 
const (
	ProjectionKey string = "projection"
	CreatedAt     string = "created-at"
)
View Source 
const ChecksumSize int = 4

4 bytes for int32 checksum in each page

View Source 
const TableFileExt string = ".tbl.json"

Variables

View Source 
var (
	ErrZeroColumns = errors.New("cannot create a table with zero columns")
)

Functions

This section is empty.

Types

type Column 

type Column struct {
	Name    string
	Type    ColumnType
	Default Value
}

The metadata that describes a column of data in the table. Each column has a name used to refer to it in queries. The type describes the range of values able to be stored in the column (and their in-memory size), and the default value will prepopulate the column's slot in every row when the table is created. There are no nullable columns in PixiDB.

func NewColumnEncoded 

func NewColumnEncoded(name string, ctype ColumnType, defval Value) Column

Create a new column description with the given name, type, and encoded default value for the type.

func NewColumnFloat32 

func NewColumnFloat32(name string, defval float32) Column

Create a new Float32-sized column with the given name and default value.

func NewColumnFloat64 

func NewColumnFloat64(name string, defval float64) Column

Create a new Float64-sized column with the given name and default value.

func NewColumnInt16 

func NewColumnInt16(name string, defval int16) Column

Create a new Int16-sized column with the given name and default value.

func NewColumnInt32 

func NewColumnInt32(name string, defval int32) Column

Create a new Int32-sized column with the given name and default value.

func NewColumnInt64 

func NewColumnInt64(name string, defval int64) Column

Create a new Int64-sized column with the given name and default value.

func NewColumnInt8 

func NewColumnInt8(name string, defval int8) Column

Create a new Int8-sized column with the given name and default value.

func NewColumnUint16 

func NewColumnUint16(name string, defval uint16) Column

Create a new Uint16-sized column with the given name and default value.

func NewColumnUint32 

func NewColumnUint32(name string, defval uint32) Column

Create a new Uint32-sized column with the given name and default value.

func NewColumnUint64 

func NewColumnUint64(name string, defval uint64) Column

Create a new Uint64-sized column with the given name and default value.

func NewColumnUint8 

func NewColumnUint8(name string, defval uint8) Column

Create a new Uint8-sized column with the given name and default value.

func NewColumnUnencoded 

func NewColumnUnencoded(name string, ctype ColumnType, defval any) Column

Create a new column description with the given name and type, and a default Go value that is encoded before being assigned to the column.

func (Column) EncodeValue 

func (c Column) EncodeValue(val any) Value

Encodes a Go value according to the type of the column. The type of the input Go value should match the specified type of the column.

func (Column) Size 

func (c Column) Size() int

The number of bytes that values of this column will consume on disk.

type ColumnNotFoundError 

type ColumnNotFoundError struct {
	Store  string
	Column string
}

func NewColumnNotFoundError 

func NewColumnNotFoundError(store string, column string) *ColumnNotFoundError

func (ColumnNotFoundError) Error 

func (c ColumnNotFoundError) Error() string

type ColumnProjection 

type ColumnProjection struct {
	// contains filtered or unexported fields
}

type ColumnType 

type ColumnType int16

Type representing the PixiDB 'types' of values that can be stored in a field in a table. These types are the 'atomic' types of PixiDB. 

const (
	ColumnTypeInt8 ColumnType = iota
	ColumnTypeUint8
	ColumnTypeInt16
	ColumnTypeUint16
	ColumnTypeInt32
	ColumnTypeUint32
	ColumnTypeInt64
	ColumnTypeUint64
	ColumnTypeFloat32
	ColumnTypeFloat64
)

func (ColumnType) EncodeValue 

func (c ColumnType) EncodeValue(val any) Value

Given a standard Go value, encodes it according to the type of the column. The column type must match the type of the Go value.

func (ColumnType) Size 

func (c ColumnType) Size() int

The size in bytes of this particular column type.

type CylindricalEquirectangularIndexer 

type CylindricalEquirectangularIndexer struct {
	Parallel float64 `json:"parallel"`
	Grid     ProjectionlessIndexer
	// contains filtered or unexported fields
}

Indexing into a sphere of pixels projected via a cylindrical equirectangular projection. 0,0 is the bottom left corner of the projection space, i.e. (XMin, YMin) => (0, 0). Supports both row-major and column-major order of the grid, which changes how efficient certain consecutive x- or y-accesses are, but does not change where x,y coordinates refer to.

func NewCylindricalEquirectangularIndexer 

func NewCylindricalEquirectangularIndexer(parallel float64, width int, height int, rowMajor bool) CylindricalEquirectangularIndexer

Create a new indexer into a grid with the cylindrical equirectangular projection, focused at the given latitude. Many common projections can be created this way.

func (CylindricalEquirectangularIndexer) Name 

func (c CylindricalEquirectangularIndexer) Name() string

func (CylindricalEquirectangularIndexer) Projection 

func (c CylindricalEquirectangularIndexer) Projection() flatsphere.Projection

func (CylindricalEquirectangularIndexer) Size 

func (c CylindricalEquirectangularIndexer) Size() int

func (CylindricalEquirectangularIndexer) ToIndex 

func (c CylindricalEquirectangularIndexer) ToIndex(loc Location) (int, error)

type Database 

type Database struct {
	// contains filtered or unexported fields
}

func NewDatabase 

func NewDatabase(dbPath string) (*Database, error)

func OpenDatabase 

func OpenDatabase(dbPath string) (*Database, error)

func (*Database) Checkpoint 

func (d *Database) Checkpoint() error

func (*Database) Create 

func (d *Database) Create(tableName string, indexer LocationIndexer, columns ...Column) error

func (*Database) Drop 

func (d *Database) Drop(tableName string) error

func (*Database) GetColumns 

func (d *Database) GetColumns(tableName string) ([]Column, error)

func (*Database) GetMetadata 

func (d *Database) GetMetadata(tableName string, key string) (string, error)

func (*Database) GetRows 

func (d *Database) GetRows(tableName string, columns []string, locations ...Location) (ResultSet, error)

func (*Database) GetTableNames 

func (d *Database) GetTableNames() ([]string, error)

func (*Database) SetMetadata 

func (d *Database) SetMetadata(tableName string, key string, value string) error

func (*Database) SetRows 

func (d *Database) SetRows(tableName string, columns []string, locations []Location, values [][]Value) (int, error)

func (*Database) Table added in v0.0.3 

func (d *Database) Table(name string) *Table

type FlatHealpixIndexer 

type FlatHealpixIndexer struct {
	Scheme healpix.HealpixScheme `json:"scheme"`
	Order  healpix.HealpixOrder  `json:"order"`
	// contains filtered or unexported fields
}

Pixelizes a sphere using the HEALPix pixelisation method. This indexer promises a single resolution pixelization, where every pixel has the same angular area. Provides storage options of both ring and nested schemes, for making certain data-access patterns more efficient.

func NewFlatHealpixIndexer 

func NewFlatHealpixIndexer(order healpix.HealpixOrder, scheme healpix.HealpixScheme) FlatHealpixIndexer

func (FlatHealpixIndexer) Name 

func (h FlatHealpixIndexer) Name() string

func (FlatHealpixIndexer) Projection 

func (h FlatHealpixIndexer) Projection() flatsphere.Projection

func (FlatHealpixIndexer) Size 

func (h FlatHealpixIndexer) Size() int

func (FlatHealpixIndexer) ToIndex 

func (h FlatHealpixIndexer) ToIndex(loc Location) (int, error)

type GridLocation 

type GridLocation struct {
	X int
	Y int
}

type IndexLocation 

type IndexLocation int

type Location 

type Location interface{}

type LocationIndexer 

type LocationIndexer interface {
	ToIndex(Location) (int, error)
	Projection() flatsphere.Projection
	Name() string
	Size() int
}

Common functionality for converting between various different coordinate systems and pixel indices within a store.

type LocationNotSupportedError 

type LocationNotSupportedError struct {
	Projection string
	Location   Location
}

func NewLocationNotSupportedError 

func NewLocationNotSupportedError(projection string, location Location) *LocationNotSupportedError

func (LocationNotSupportedError) Error 

func (l LocationNotSupportedError) Error() string

type LocationOutOfBoundsError 

type LocationOutOfBoundsError struct {
	Location Location
}

func NewLocationOutOfBoundsError 

func NewLocationOutOfBoundsError(location Location) LocationOutOfBoundsError

func (LocationOutOfBoundsError) Error 

func (l LocationOutOfBoundsError) Error() string

type MercatorCutoffIndexer 

type MercatorCutoffIndexer struct {
	NorthCutoff float64 `json:"northCutoff"`
	SouthCutoff float64 `json:"southCutoff"`

	Grid ProjectionlessIndexer
	// contains filtered or unexported fields
}

Indexing into a sphere of pixels project via a standard Mercator projection. Because Mercator diverges at the poles, two cutoff parameters are provided for the northern and southern latitudes. These cutoff parallels will mark the boundaries of the top and bottom of the grid respectively. Supports either row-major or column-major storage of the data for particular access patterns.

func NewMercatorCutoffIndexer 

func NewMercatorCutoffIndexer(northCutoff float64, southCutoff float64, width int, height int, rowMajor bool) MercatorCutoffIndexer

func (MercatorCutoffIndexer) Name 

func (m MercatorCutoffIndexer) Name() string

func (MercatorCutoffIndexer) Projection 

func (m MercatorCutoffIndexer) Projection() flatsphere.Projection

func (MercatorCutoffIndexer) Size 

func (m MercatorCutoffIndexer) Size() int

func (MercatorCutoffIndexer) ToIndex 

func (m MercatorCutoffIndexer) ToIndex(loc Location) (int, error)

type NestLocation 

type NestLocation int

type Page 

type Page struct {
	// contains filtered or unexported fields
}

Wrapper struct for a page that has been loaded into memory. Contains a 'dirty' flag to mark the cached page as having received an update in the data that needs to be flushed to disk.

type Pagemaster 

type Pagemaster struct {
	// contains filtered or unexported fields
}

Abstracts the data access and caching in memory of a large file using a fixed page size. Individual operations intended to be threadsafe and allow for concurrency while maintaining efficiency. This abstraction also implements basic checksumming to validate the integrity of the data. A checksum is stored just before each page in the disk file, but this piece of data is not included in the slices returned by any of the public facing methods of this type. https://en.wikipedia.org/wiki/The_Pagemaster

func NewPagemaster 

func NewPagemaster(path string, maxCache int) *Pagemaster

Create a new cached data layer to access the file on disk location at `path`, with the specified number of pages allowed in the cache. No disk side effect. Must call Initialize afterward if the path is to a newly created (empty) file.

func (*Pagemaster) ClearCache 

func (p *Pagemaster) ClearCache()

Empties the cache of all pages. Does not destroy the data in the pages, so if those are still referenced elsewhere they will not be garbage collected. No disk side effect.

func (*Pagemaster) FlushAllPages 

func (p *Pagemaster) FlushAllPages() error

Writes all pages marked dirty to the disk, locking access to the cache and the file until writing is complete. If a page write files, the process is stopped and an error is returned, but only the successfully written pages will be marked clean. The page on which the write errored, and the remaining dirty pages, will still be marked dirty if the managing process wants to retry flushing.

func (*Pagemaster) FlushPage 

func (p *Pagemaster) FlushPage(pageIndex int) error

Writes the page in the cache to disk, whether it is dirty or not. Marks the page as clean afterward. If the page does not exist in the cache, no action is taken. If the write is unsuccessful, the page dirtiness status will be left unchanged.

func (*Pagemaster) GetChunk 

func (p *Pagemaster) GetChunk(pageIndex int, offset int, size int) ([]byte, error)

Essentially the same actions as GetPage, but returns a portion of the page data at the given byte offset.

func (*Pagemaster) GetPage 

func (p *Pagemaster) GetPage(pageIndex int) ([]byte, error)

Get the page with the sequential index given. If the page exists in the cache, does not access the disk. Otherwise, loads the page into the cache and returns it.

func (*Pagemaster) Initialize 

func (p *Pagemaster) Initialize(pages int, page []byte) error

For pagemasters created over newly created empty files, this function will initialize the file with the given number of pages, each page filled with the same given template of data. If a write to the file fails, all of the writes that have succeeded to that point will not be undone. However, future calls to Initialize (e.g. a rety), will write over any data that was written previously.

func (*Pagemaster) LoadPage 

func (p *Pagemaster) LoadPage(pageIndex int) ([]byte, error)

Retrieve the page at the given index from disk, load it into the cache, and return the data. Always skips cache to read from disk. If the cache is full, a different page is removed from the cache before the requested page is added.

func (*Pagemaster) MaxPagesInCache 

func (p *Pagemaster) MaxPagesInCache() int

The maximum number of pages allowed in the cache.

func (*Pagemaster) PageSize 

func (p *Pagemaster) PageSize() int

The number of bytes that be written to per page in the file.

func (*Pagemaster) PagesInCache 

func (p *Pagemaster) PagesInCache() int

The current number of pages in the cache.

func (*Pagemaster) SetChunk 

func (p *Pagemaster) SetChunk(pageIndex int, offset int, chunk []byte) error

Similar to SetPage but only updates the specified portion of data in the page.

func (*Pagemaster) SetPage 

func (p *Pagemaster) SetPage(pageIndex int, page []byte) error

Sets the data for the page at the given index, and marks the cache entry as dirty. If the page does not yet exist in the cache, it will exist in the cache afterwards, potentially unloading a different page to make room.

type ProjectedLocation 

type ProjectedLocation struct {
	X float64
	Y float64
}

type Projection 

type Projection []ColumnProjection

type ProjectionlessIndexer 

type ProjectionlessIndexer struct {
	Width    int  `json:"width"`
	Height   int  `json:"height"`
	RowMajor bool `json:"rowmajor"`
}

Simple indexing into a grid, no spherical projection provided by this indexer. Supports either row-major or column-major storage of the data for particular access patterns.

func NewProjectionlessIndexer 

func NewProjectionlessIndexer(width int, height int, rowMajor bool) ProjectionlessIndexer

func (ProjectionlessIndexer) Name 

func (p ProjectionlessIndexer) Name() string

func (ProjectionlessIndexer) Projection 

func (p ProjectionlessIndexer) Projection() flatsphere.Projection

func (ProjectionlessIndexer) Size 

func (p ProjectionlessIndexer) Size() int

func (ProjectionlessIndexer) ToIndex 

func (p ProjectionlessIndexer) ToIndex(loc Location) (int, error)

type RectangularLocation 

type RectangularLocation struct {
	X float64
	Y float64
	Z float64
}

func (RectangularLocation) ToSpherical 

func (r RectangularLocation) ToSpherical() SphericalLocation

type ResultSet 

type ResultSet struct {
	Columns []Column
	Rows    [][]Value
}

type RingLocation 

type RingLocation int

type Row 

type Row []byte

func (Row) Project 

func (r Row) Project(proj Projection) []Value

type SphericalLocation 

type SphericalLocation struct {
	Latitude  float64
	Longitude float64
}

type Store 

type Store struct {
	// The name by which the store can be referenced in queries. Also the final folder in the path
	// in which the data file for this store is kept.
	Name      string   `json:"-"`
	ColumnSet []Column `json:"columns"`
	Rows      int      `json:"rows"`
	// contains filtered or unexported fields
}

A simple set of rows, divided into fixed-size columns. The number of rows and columns both are known ahead of time, and the most efficient access pattern is by row index. A store keeps all of its data compact in one flat file, storing variable size metadata in a separate structured file.

func NewStore 

func NewStore(path string, rows int, columns ...Column) (*Store, error)

func OpenStore 

func OpenStore(path string) (*Store, error)

func (*Store) Checkpoint 

func (s *Store) Checkpoint() error

func (*Store) DefaultRow 

func (s *Store) DefaultRow() []byte

func (*Store) Drop 

func (s *Store) Drop() error

func (*Store) FilterColumns 

func (s *Store) FilterColumns(proj Projection) []Column

func (*Store) GetRowAt 

func (s *Store) GetRowAt(index int) (Row, error)

func (*Store) GetValueAt 

func (s *Store) GetValueAt(index int) (Value, error)

Cheat method when a store has only a single column and we don't need to do any projection (because it's the only column)

func (*Store) Path 

func (s *Store) Path() string

func (*Store) Projection 

func (s *Store) Projection(columns ...string) (Projection, error)

func (*Store) RowSize 

func (s *Store) RowSize() int

func (*Store) RowsPerPage 

func (s *Store) RowsPerPage() int

func (*Store) SetRowAt 

func (s *Store) SetRowAt(index int, row Row) error

func (*Store) SetValueAt added in v0.0.7 

func (s *Store) SetValueAt(column string, index int, val Value) error

type Table 

type Table struct {
	Indexer     LocationIndexer   `json:"indexer"`
	IndexerName string            `json:"indexerName"`
	Metadata    map[string]string `json:"metadata"`
	// contains filtered or unexported fields
}

func NewTable 

func NewTable(path string, indexer LocationIndexer, columns ...Column) (*Table, error)

func OpenTable 

func OpenTable(path string) (*Table, error)

func (*Table) Checkpoint 

func (t *Table) Checkpoint() error

func (*Table) Drop 

func (t *Table) Drop() error

func (*Table) GetRows 

func (t *Table) GetRows(projectedColumns []string, locations ...Location) (ResultSet, error)

func (*Table) Name 

func (t *Table) Name() string

func (*Table) Path 

func (t *Table) Path() string

func (*Table) SetMetadata 

func (t *Table) SetMetadata(key string, value string) error

func (*Table) SetRows 

func (t *Table) SetRows(columns []string, locations []Location, values [][]Value) (int, error)

func (*Table) SetValue added in v0.0.7 

func (t *Table) SetValue(column string, location Location, value Value) error

func (*Table) UnmarshalJSON 

func (t *Table) UnmarshalJSON(b []byte) error

type TableNotFoundError 

type TableNotFoundError struct {
	Table string
}

func NewTableNotFoundError 

func NewTableNotFoundError(tableName string) TableNotFoundError

func (TableNotFoundError) Error 

func (t TableNotFoundError) Error() string

type UniqueLocation 

type UniqueLocation int

type Value 

type Value []byte

func NewFloat32Value added in v0.0.5 

func NewFloat32Value(val float32) Value

func NewFloat64Value added in v0.0.5 

func NewFloat64Value(val float64) Value

func NewInt16Value added in v0.0.5 

func NewInt16Value(val int16) Value

func NewInt32Value added in v0.0.5 

func NewInt32Value(val int32) Value

func NewInt64Value added in v0.0.5 

func NewInt64Value(val int64) Value

func NewInt8Value added in v0.0.5 

func NewInt8Value(val int8) Value

func NewUint16Value added in v0.0.5 

func NewUint16Value(val uint16) Value

func NewUint32Value added in v0.0.5 

func NewUint32Value(val uint32) Value

func NewUint64Value added in v0.0.5 

func NewUint64Value(val uint64) Value

func NewUint8Value added in v0.0.5 

func NewUint8Value(val uint8) Value

func (Value) AsFloat32 

func (v Value) AsFloat32() float32

func (Value) AsFloat64 

func (v Value) AsFloat64() float64

func (Value) AsInt16 

func (v Value) AsInt16() int16

func (Value) AsInt32 

func (v Value) AsInt32() int32

func (Value) AsInt64 

func (v Value) AsInt64() int64

func (Value) AsInt8 

func (v Value) AsInt8() int8

func (Value) AsUint16 

func (v Value) AsUint16() uint16

func (Value) AsUint32 

func (v Value) AsUint32() uint32

func (Value) AsUint64 

func (v Value) AsUint64() uint64

func (Value) AsUint8 

func (v Value) AsUint8() uint8

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.