updog

README

README for updog

What's updog?

Not much, what's up with you?!

updog is a columnar index to very quickly run count queries, optionally allowing to group the result by specific columns. If you're familiar with SQL, its functionality is equivalent to running SELECT x, y, z, COUNT(*) FROM ... WHERE ... GROUP BY x, y, z queries.

The data source to query on (the FROM part of the SQL query) is limited to a single index (which you can imagine as a table), while the query expression (the WHERE clause of the SQL query) is currently limited to the operators =, NOT, AND and OR. At the moment, updog does not provide a textual query language. Queries need to be constructed as Query objects instead.

See the Go Reference for further details and a full documentation of the API.

License

This project is licensed under the MIT license - see the LICENSE file for details.

Documentation

Index

• type BigIndexWriter
  • func NewBigIndexWriter(db *bbolt.DB, tempDB *bbolt.DB) (*BigIndexWriter, error)
  • func (idx *BigIndexWriter) AddRow(values map[string]string) (uint32, error)
  • func (idx *BigIndexWriter) Flush() error
• type Cache
• type CacheMetrics
• type CounterMetric
• type ExprAnd
  • func (e *ExprAnd) String() string
• type ExprEqual
  • func (e *ExprEqual) String() string
• type ExprNot
  • func (e *ExprNot) String() string
• type ExprOr
  • func (e *ExprOr) String() string
• type Expression
• type HistogramMetric
• type Index
  • func OpenIndex(file string, opts ...IndexOption) (*Index, error)
  • func OpenIndexFromBoltDatabase(db *bbolt.DB, opts ...IndexOption) (*Index, error)
  • func (idx *Index) Close() error
  • func (idx *Index) Execute(q *Query) (*Result, error)
  • func (idx *Index) GetSchema() *Schema
• type IndexMetrics
• type IndexOption
  • func WithCache(cache Cache) IndexOption
  • func WithIndexMetrics(metrics *IndexMetrics) IndexOption
  • func WithPreloadedData() IndexOption
• type IndexWriter
  • func NewIndexWriter(filename string) *IndexWriter
  • func (idx *IndexWriter) AddRow(values map[string]string) (uint32, error)
  • func (idx *IndexWriter) Flush() error
  • func (idx *IndexWriter) WriteToBoltDatabase(db *bbolt.DB) error
• type LRUCache
  • func NewLRUCache(maxSizeBytes uint64, opts ...LRUCacheOption) *LRUCache
  • func (c *LRUCache) Get(key uint64) (*roaring.Bitmap, bool)
  • func (c *LRUCache) Put(key uint64, bm *roaring.Bitmap)
• type LRUCacheOption
  • func WithCacheMetrics(metrics *CacheMetrics) LRUCacheOption
• type Query
• type Result
• type ResultField
• type ResultGroup
• type Schema
• type SchemaColumn
• type SchemaColumnValue

Types

type BigIndexWriter

type BigIndexWriter struct {
	// contains filtered or unexported fields
}

func NewBigIndexWriter

func NewBigIndexWriter(db *bbolt.DB, tempDB *bbolt.DB) (*BigIndexWriter, error)

func (*BigIndexWriter) AddRow

func (idx *BigIndexWriter) AddRow(values map[string]string) (uint32, error)

func (*BigIndexWriter) Flush

func (idx *BigIndexWriter) Flush() error

type Cache

type Cache interface {
	Get(key uint64) (bm *roaring.Bitmap, found bool)
	Put(key uint64, bm *roaring.Bitmap)
}

Cache is the interface that needs to be fulfilled for a cache implementation.

type CacheMetrics

type CacheMetrics struct {
	CacheHit  CounterMetric
	CacheMiss CounterMetric
	GetCall   CounterMetric
	PutCall   CounterMetric
}

CacheMetrics contains the counter metrics relating to a cache. It is currently mainly used in the LRUCache implementation.

type CounterMetric

type CounterMetric interface {
	Inc()
}

type ExprAnd

type ExprAnd struct {
	Exprs []Expression
}

func (*ExprAnd) String

func (e *ExprAnd) String() string

type ExprEqual

type ExprEqual struct {
	Column string
	Value  string
}

func (*ExprEqual) String

func (e *ExprEqual) String() string

type ExprNot

type ExprNot struct {
	Expr Expression
}

func (*ExprNot) String

func (e *ExprNot) String() string

type ExprOr

type ExprOr struct {
	Exprs []Expression
}

func (*ExprOr) String

func (e *ExprOr) String() string

type Expression

type Expression interface {
	String() string
	// contains filtered or unexported methods
}

type HistogramMetric

type HistogramMetric interface {
	Observe(float64)
}

type Index

type Index struct {
	// contains filtered or unexported fields
}

Index represents an index to run queries on. You have to create objects using the OpenIndex or OpenIndexFromBoltDatabase constructor functions.

func OpenIndex

func OpenIndex(file string, opts ...IndexOption) (*Index, error)

OpenIndex opens an index file previously created using the IndexWriter.

func OpenIndexFromBoltDatabase

func OpenIndexFromBoltDatabase(db *bbolt.DB, opts ...IndexOption) (*Index, error)

OpenIndexFromBoltDatabase opens an index directly from a bbolt database. For this to work correctly, the index should be created using the IndexWriter.

func (*Index) Close

func (idx *Index) Close() error

Close closes the index, including the associated bbolt database.

func (*Index) Execute

func (idx *Index) Execute(q *Query) (*Result, error)

Execute runs the provided query on the index and returns the query result.

func (*Index) GetSchema

func (idx *Index) GetSchema() *Schema

type IndexMetrics

type IndexMetrics struct {
	ExecuteDuration HistogramMetric
}

type IndexOption

type IndexOption func(idx *Index) error

func WithCache

func WithCache(cache Cache) IndexOption

WithCache is an option for OpenIndex and OpenIndexFromBoltDatabase to set a cache for caching queries, including partial queries.

func WithIndexMetrics

func WithIndexMetrics(metrics *IndexMetrics) IndexOption

WithIndexMetrics is an option for OpenIndex and OpenIndexFromBoltDatabase to set an IndexMetrics object.

func WithPreloadedData

func WithPreloadedData() IndexOption

WithPreloadedData is an option for OpenIndex and OpenIndexFromBoltDatabase to preload all data into memory to allow for faster queries. Only use this if all data from the index file will fit into the available memory.

type IndexWriter

type IndexWriter struct {
	// contains filtered or unexported fields
}

IndexWriter is a helper type to create a columnar index by adding row data. It needs to be created using the NewIndexWriter constructor function.

func NewIndexWriter

func NewIndexWriter(filename string) *IndexWriter

NewIndexWriter creates a new IndexWriter object. IndexWriter is used to add row data and to write the corresponding index data to a persistent store.

func (*IndexWriter) AddRow

func (idx *IndexWriter) AddRow(values map[string]string) (uint32, error)

AddRow adds a row of data and returns its row ID. The row data must be provided as map, where the keys contain the column names, and the values the corresponding column values.

func (*IndexWriter) Flush

func (idx *IndexWriter) Flush() error

WriteToFile writes the index data to the provided file.

func (*IndexWriter) WriteToBoltDatabase

func (idx *IndexWriter) WriteToBoltDatabase(db *bbolt.DB) error

WriteToBoltDatabase writes the index data directly to a badger database.

type LRUCache

type LRUCache struct {
	// contains filtered or unexported fields
}

LRUCache is a size-bounded cache with a LRU cache replacement policy. You have to use the NewLRUCache constructor function to create an instance of it.

func NewLRUCache

func NewLRUCache(maxSizeBytes uint64, opts ...LRUCacheOption) *LRUCache

NewLRUCache returns a new LRUCache object.

func (*LRUCache) Get

func (c *LRUCache) Get(key uint64) (*roaring.Bitmap, bool)

Get returns the bitmap associated with the provided key, if available.

func (*LRUCache) Put

func (c *LRUCache) Put(key uint64, bm *roaring.Bitmap)

Put stores the provided bitmap under the provided key in the cache. It ensures that the maximum size of the LRU cache is kept, by evicting other cached elements if necessary.

type LRUCacheOption

type LRUCacheOption func(c *LRUCache)

func WithCacheMetrics

func WithCacheMetrics(metrics *CacheMetrics) LRUCacheOption

WithCacheMetrics is an option for LRUCache to set a CacheMetrics object.

type Query

type Query struct {
	// Expr is the expression you want to limit your query on. You can use the types ExprEqual, ExprNot,
	// ExprAnd and ExprOr to construct your expression.
	Expr Expression

	// GroupBy is a list of column names you want to group by. The result will then contain the
	// results for all available values of all the listed columns for which a non-zero count result
	// was determined.
	GroupBy []string
	// contains filtered or unexported fields
}

Query describes a count query to execute on an index. updog allows you to run the equivalent of SQL queries like `SELECT x, y, z, COUNT(*) WHERE ... GROUP BY x, y, z`.

type Result

type Result struct {
	// Count is the total count of rows that matched the query expression.
	Count uint64

	// Groups contains a list of grouped results. If no GroupBy list was provided
	// in the query, this list will be empty.
	Groups []ResultGroup
}

Result contains the query result.

type ResultField

type ResultField struct {
	Column string
	Value  string
}

ResultField contains a single column name and value. It is used in ResultGroup objects.

type ResultGroup

type ResultGroup struct {
	// Fields contains a list of result fields for that group, each field consisting of the
	// column name and value.
	Fields []ResultField

	// Count contains the determined count for the list of result fields.
	Count uint64
}

ResultGroup contains a single grouped result.

type Schema

type Schema struct {
	Columns []SchemaColumn
}

type SchemaColumn

type SchemaColumn struct {
	Name   string
	Values []SchemaColumnValue
}

type SchemaColumnValue

type SchemaColumnValue struct {
	Value string
}