model

README

Model

Package model is a convenience wrapper around what the Store provides. It's main responsibility is to maintain indexes that would otherwise be maintaned by the users to enable different queries on the same data.

Usage

The following snippets will this piece of code prepends them.

import(
    model "github.com/micro/micro/v3/service/model"
    fs "github.com/micro/micro/v3/service/store/file"
)

type User struct {
	ID      string `json:"id"`
 	Name string    `json:"name"`
	Age     int    `json:"age"`
	HasPet  bool   `json:"hasPet"`
	Created int64  `json:"created"`
	Tag     string `json:"tag"`
	Updated int64  `json:"updated"`
}

Query by field equality

For each field we want to query on we have to create an index. Index by id is provided by default to each DB, there is no need to specify it.

ageIndex := model.ByEquality("age")

db := model.New(fs.NewStore(), User{}, []model.Index{(ageIndex})

err := db.Create(User{
    ID: "1",
    Name: "Alice",
    Age: 20,
})
if err != nil {
    // handle save error
}
err := db.Create(User{
    ID: "2",
    Name: "Jane",
    Age: 22
})
if err != nil {
    // handle save error
}

err = db.Read(model.Equals("age", 22), &users)
if err != nil {
	// handle list error
}
fmt.Println(users)

// will print
// [{"id":"2","name":"Jane","age":22}]

Reading all records in an index

Reading can be done without specifying a value:

db.Read(Equals("age", nil), &users)

Readings will be unordered, ascending ordered or descending ordered depending on the ordering settings of the index.

Ordering

Indexes by default are ordered. If we want to turn this behaviour off:

ageIndex.Order.Type = OrderTypeUnordered

ageQuery := model.Equals("age", 22)
ageQuery.Order.Type = OrderTypeUnordered

Filtering by one field, ordering by other

typeIndex := ByEquality("type")
typeIndex.Order = Order{
	Type:      OrderTypeDesc,
	FieldName: "age",
}

// Results will be ordered by age
db.Read(typeIndex.ToQuery("a-certain-type-value"))

By default the ordering field is the same as the filtering field.

Reverse order

ageQuery.Desc = true

Queries must match indexes

It is important to note that queries must match indexes. The following index-query pairs match (separated by an empty line)

// Ascending ordered index by age
index := model.Equality("age")
// Read ascending ordered by age
query := model.Equals("age", nil)
// Read ascending ordered by age where age = 20
query2 := model.Equals("age", 20) 

// Descending ordered index by age
index := model.Equality("age")
index.Order.Type = OrderTypeDesc
// Read descending ordered by age
query := model.Equals("age", nil)
query.Order.Type = OrderTypeDesc
// Read descending ordered by age where age = 20
query2 := model.Equals("age", 20)
query2.Order.Type = OrderTypeDesc

// Unordered index by age
index := model.Equality("age")
index.Order.Type = OrderTypeUnordered
// Read unordered by age
query := model.Equals("age", nil)
query.Order.Type = OrderTypeUnordered
// Read unordered by age where age = 20
query2 := model.Equals("age", 20)
query2.Order.Type = OrderTypeUnordered

Of course, maintaining this might be inconvenient, for this reason the ToQuery method was introduced, see below.

Creating a query out of an Index

index := model.Equality("age")
index.Order.Type = OrderTypeUnordered

db.Read(index.ToQuery(25))

Unordered listing without value

It's easy to see how listing things by unordered indexes on different fields should result in the same output: a randomly ordered list, ie:

ageIndex := model.Equality("age")
ageIndex.Order.Type = OrderTypeUnordered

emailIndex := model.Equality("email")
emailIndex.Order.Type = OrderTypeUnordered

result1 := []User{}
result2 := []User{}

db.Read(model.Equals("age"), &result1)
db.Read(model.Equals("email"), &result2)

// Both result1 and result2 will be an unordered listing without
// filtering on either the age or email fields.
// Could be thought of as a noop query despite not having an explicit "no query" listing.

Ordering by string fields

Ordering comes for "free" when dealing with numeric or boolean fields, but it involves in padding, inversing and order preserving base32 encoding of values to work for strings.

This can sometimes result in large keys saved, as the inverse of a small 1 byte character in a string is a 4 byte rune. Optionally adding base32 encoding on top to prevent exotic runes appearing in keys, strings blow up in size even more. If saving space is a requirement and ordering is not, ordering for strings should be turned off.

The matter is further complicated by the fact that the padding size must be specified ahead of time.

nameIndex := model.ByEquality("name")
nameIndex.StringOrderPadLength = 10

nameQuery := model.Equals("age", 22)
// `StringOrderPadLength` is not needed to be specified for the query

To turn off base32 encoding and keep the runes:

nameIndex.Base32Encode = false

Unique indexes

emailIndex := model.ByEquality("email")
emailIndex.Unique = true

Design

Restrictions

To maintain all indexes properly, all fields must be filled out when saving. This sometimes requires a Read, Modify, Write pattern. In other words, partial updates will break indexes.

This could be avoided later if model does the loading itself.

TODO

• Implement deletes
• Implement counters, for pattern inspiration see the tags service
• Test boolean indexes and its ordering
• There is a stuttering in the way id fields are being saved twice. ID fields since they are unique do not need id appended after them in the record keys.

Documentation

Overview

Package model implements convenience methods for managing indexes on top of the Store. See this doc for the general idea https://github.com/m3o/dev/blob/feature/storeindex/design/auto-indexes.md Prior art/Inspirations from github.com/gocassa/gocassa, which is a similar package on top an other KV store (Cassandra/gocql)

Index

• Constants
• Variables
• type Index
  • func ByEquality(fieldName string) Index
  • func (i Index) ToQuery(value interface{}) Query
• type Model
  • func New(instance interface{}, options *Options) Model
  • func NewModel(opts ...Option) Model
• type Option
  • func WithContext(ctx context.Context) Option
  • func WithDatabase(db string) Option
  • func WithDebug(d bool) Option
  • func WithIndexes(idx ...Index) Option
  • func WithKey(idField string) Option
  • func WithNamespace(ns string) Option
  • func WithStore(s store.Store) Option
  • func WithTable(t string) Option
• type Options
• type Order
• type OrderType
• type Query
  • func QueryAll() Query
  • func QueryByID(id string) Query
  • func QueryEquals(fieldName string, value interface{}) Query

Constants

const (
	OrderTypeUnordered = OrderType("unordered")
	OrderTypeAsc       = OrderType("ascending")
	OrderTypeDesc      = OrderType("descending")
)

Variables

var (
	ErrorNilInterface         = errors.New("interface is nil")
	ErrorNotFound             = errors.New("not found")
	ErrorMultipleRecordsFound = errors.New("multiple records found")
)

var (
	// DefaultKey is the default field for indexing
	DefaultKey = "ID"

	// DefaultIndex is the ID index
	DefaultIndex = newIndex("ID")

	// DefaultModel is the default model
	DefaultModel = NewModel()
)

Types

type Index

type Index struct {
	FieldName string
	// Type of index, eg. equality
	Type  string
	Order Order
	// Do not allow duplicate values of this field in the index.
	// Useful for emails, usernames, post slugs etc.
	Unique bool
	// Strings for ordering will be padded to a fix length
	// Not a useful property for Querying, please ignore this at query time.
	// Number is in bytes, not string characters. Choose a sufficiently big one.
	// Consider that each character might take 4 bytes given the
	// internals of reverse ordering. So a good rule of thumbs is expected
	// characters in a string X 4
	StringOrderPadLength int
	// True = base32 encode ordered strings for easier management
	// or false = keep 4 bytes long runes that might dispaly weirdly
	Base32Encode bool

	FloatFormat string
	Float64Max  float64
	Float32Max  float32
}

Index represents a data model index for fast access

func ByEquality

func ByEquality(fieldName string) Index

ByEquality constructs an equiality index on `fieldName`

func (Index) ToQuery

func (i Index) ToQuery(value interface{}) Query

type Model

type Model interface {
	// Context sets the context for the model returning a new copy
	Context(ctx context.Context) Model
	// Register a new model eg. User struct, Order struct
	Register(v interface{}) error
	// Create a new object. (Maintains indexes set up)
	Create(v interface{}) error
	// Update will take an existing object and update it.
	// TODO: Make use of "sync" interface to lock, read, write, unlock
	Update(v interface{}) error
	// Read accepts a pointer to a value and expects to fine one or more
	// elements. Read throws an error if a value is not found or we can't
	// find a matching index for a slice based query.
	Read(query Query, resultPointer interface{}) error
	// Deletes a record. Delete only support Equals("id", value) for now.
	// @todo Delete only supports string keys for now.
	Delete(query Query) error
}

Model represents a place where data can be saved to and queried from.

func New

func New(instance interface{}, options *Options) Model

New returns a new model with the given values

func NewModel

func NewModel(opts ...Option) Model

NewModel returns a new model with options or uses internal defaults

type Option

type Option func(*Options)

func WithContext

func WithContext(ctx context.Context) Option

WithContext sets the context for all queries

func WithDatabase

func WithDatabase(db string) Option

WithDatabase sets the default database for queries

func WithDebug

func WithDebug(d bool) Option

WithDebug enables debug logging

func WithIndexes

func WithIndexes(idx ...Index) Option

WithIndexes creates an option with the given indexes

func WithKey

func WithKey(idField string) Option

WithKey sets the Key

func WithNamespace

func WithNamespace(ns string) Option

WithNamespace sets the namespace to scope to

func WithStore

func WithStore(s store.Store) Option

WithStore create an option for setting the store

func WithTable

func WithTable(t string) Option

WithTable sets the default table for queries

type Options

type Options struct {
	// Database sets the default database
	Database string
	// Table sets the default table
	Table string
	// Enable debug logging
	Debug bool
	// The indexes to use for queries
	Indexes []Index
	// Namespace to scope to
	Namespace string
	// Store is the storage engine
	Store store.Store
	// Context is the context for all model queries
	Context context.Context
	// Key is the fiel name of the primary key
	Key string
}

type Order

type Order struct {
	FieldName string
	// Ordered or unordered keys. Ordered keys are padded.
	// Default is true. This option only exists for strings, where ordering
	// comes at the cost of having rather long padded keys.
	Type OrderType
}

Order is the order of the index

type OrderType

type OrderType string

type Query

type Query struct {
	Index
	Order  Order
	Value  interface{}
	Offset int64
	Limit  int64
}

func QueryAll

func QueryAll() Query

func QueryByID

func QueryByID(id string) Query

QueryByID is short hand for querying by the primary index

func QueryEquals

func QueryEquals(fieldName string, value interface{}) Query

Equals is an equality query by `fieldName` It filters records where `fieldName` equals to a value.