inventory

package module
v0.0.0-...-c2ecc7f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 14, 2024 License: MIT Imports: 6 Imported by: 0

README

Inventory

Build Status Go Report Card License

Inventory applies IoC (Inversion of Control) for application in-flight data. It shares some similarities with DI containers but instead of initializing once, application data containers are required to define the loading procedure of fresh data from any cold source. the application then enjoys an always-fresh, in-mem, indexed data as a dependency that can be passed through structs or funcs.

It was built while I worked @Cyolo to consolidate caching layer operations where the cache layer is the only access layer for access. No cold layer. Data is always prepared on the hot cache. It is rather inefficient in writes (compared to a kv store), but it's more than ok in reads.

The big advantage of this structure is that if all the data you need in your hot path fits in your memory, it will spare you from the frustrating mechanisms that meant for actively reading from the data center or in a centralized storage such as sql server, mongo db or etcd.

Components

DB

a primitive storage layer. it's best if it's shared among collections and this is why it is initialized independently.

how to init:

db := DB()
Extractor

a simple func that you implement in order to load a specific kind to the collection from the "cold" source.
here's an example of loading foo from an SQL db:

Extractor(func(add ...Item) {
    rows, err := db.Query("select id, name from foo")
    if err != nil {
        return
    }
    defer rows.Close()
	
    var foo foo
    for rows.Next() {
        err = rows.Scan(&foo)
        if err != nil {
            return
        }
		
        add(&foo)
    }
})
Collection

a high-level, typed, data access layer for mapping and querying the data by the application needs. the required DB instance is an interface and you can provide your implementation if needed. for example, you can provide an implementation that uses a disk if your dataset is too big.

how to init:

books := NewCollection[*book](db, "books", 
	Extractor(func(load func(in ...*book)) {
		rs := someDB.QueryAllBooks()
		for rs.Next() {
			var book *book
			err := rs.scan(book)
			load(book)
		}
		rs.Close()
	}), 
	PrimaryKey("id", func(book *book, val func(string)) { val(book.id) }),
)

how to use:

creating additional keys for unique properties will provide you with Getter by the provided key:

bookByName := books.AdditionalKey("name", func(book *book, keyVal func(string)) { val(book.name) }),
dune, ok := bookByName("Dune")

you can use the Getter as a dependency for some struct:

type bookService struct {
	bookByID inventory.Getter[*book]
}

func (bs *bookService) getBook(id string) (*book, bool) {
	return bs.bookByID(id)
}

you can also map the items by a key that will yield a list for a given value of the provided key:

bookByAuthor := books.MapBy("author", func(book *book, val func(string)) { val(book.author) }),
daBooks, err := bookByAuthor("Douglas Adams")

or simply iterating over all items in the collection, with the ability to stop whenever you are done:

books.Scan(func(book *book) bool {
	if whatINeeded(book) {
		// do something with it
		...
		// maybe stop?
		return false
	} 
	
	// proceed to next book
	return true
})

another useful gem is called Derivative - it is meant for creating objects based on hot-reloaded data - automatically and only once:

bookTags := inventory.Derive[*book, []string](books, "tags", func(book *book) ([]string, error) {
	text := loadText(book)
	return calculateTags(text)
})

so now you can call bookTags with a book and always get the tags relevant to the book at its latest state. this will always be invalidated as well and re-calculated when required but only once per reload of the original book.

Reload Data

reloading the data is performed as a reaction to invalidation of a collection. it deletes all related items from related collection and reloads all the relevant kinds (currently all data of a kind, not only the invalidated items).

collection.Invalidate()

the underlying db implements isolated transactions and therefore writes don't block reads. this means that the data in the db is stale until Invalidate returns.

Performance

performance is not a key objective of this solution. the idea is to manage fresh app data in-memory in a way that will be the most comfortable to work with - types, indexes, etc... for comparison, it is much faster than in-mem SQLite, but slower than in-mem kv dbs.
if performance is more important for you than readability then you should look for other solutions.

benchmark result on a MacBook Pro 2020 model

goos: darwin
goarch: amd64
pkg: github.com/avivklas/inventory
cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
Benchmark_collection
Benchmark_collection/get
Benchmark_collection/get-8                             3820045      296.1 ns/op
Benchmark_collection/query_one-to-one_relation
Benchmark_collection/query_one-to-one_relation-8       1074028	    1100 ns/op
Benchmark_collection/query_one-to-many_relation
Benchmark_collection/query_one-to-many_relation-8       797988      1504 ns/op

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Collection 

type Collection[T any] struct {
	// contains filtered or unexported fields
}

Collection is like a typed access layer to the db defined by a schema but because the purpose of this repository is IoC of the data, it also defines how the data is loaded from the cold source.

func NewCollection 

func NewCollection[T any](db DB, kind string, opts ...CollectionOpt[T]) (c *Collection[T])

NewCollection creates a Collection of T with the provided opts; PrimaryKey and Extractor are mandatory

func (*Collection[T]) AdditionalKey 

func (c *Collection[T]) AdditionalKey(key string, value indexFn[T]) Getter[T]

AdditionalKey creates an additional index on the collection using an indexFn

for example; c.AdditionalKey("name", func(f Foo, val func(string)) { val(f.name) })

func (*Collection[T]) GetBy 

func (c *Collection[T]) GetBy(key string) Getter[T]

GetBy creates a getter from existing index

func (*Collection[T]) Invalidate 

func (c *Collection[T]) Invalidate()

Invalidate reloads all data from the origin source, defined by the Extractor

func (*Collection[T]) Load 

func (c *Collection[T]) Load(writer DBWriter)

Load loads all data from the origin source, defined by the Extractor

func (*Collection[T]) MapBy 

func (c *Collection[T]) MapBy(key string, ref indexFn[T]) Query[T]

MapBy creates a Query from the provided key mapped by the provided indexFn, to be used for querying the collection by a non-unique attribute

func (*Collection[T]) PrimaryKey 

func (c *Collection[T]) PrimaryKey(key string, value indexFn[T]) Getter[T]

PrimaryKey creates a primary index on the collection using an indexFn

for example; c.PrimaryKey("id", func(f Foo, val func(string)) { val(f.id) })

func (*Collection[T]) Scalar 

func (c *Collection[T]) Scalar(key, value string) Scalar[T]

Scalar creates a "static" Getter that will require no key

func (*Collection[T]) Scan 

func (c *Collection[T]) Scan(consume func(T) bool, filters ...func(T) bool)

Scan iterates over all items in the collection, not sorted

func (*Collection[T]) With 

func (c *Collection[T]) With(opts ...CollectionOpt[T]) *Collection[T]

With instruments the collection with the provided opts

type CollectionOpt 

type CollectionOpt[T any] func(*Collection[T])

func AdditionalKey 

func AdditionalKey[T any](name string, value indexFn[T]) CollectionOpt[T]

AdditionalKey adds a secondary index of the collection

func Extractor 

func Extractor[T any](x extractFn[T]) CollectionOpt[T]

Extractor sets the extractFn of the collection. extractFn is a function that extracts the data from the origin source

func PrimaryKey 

func PrimaryKey[T any](name string, value indexFn[T]) CollectionOpt[T]

PrimaryKey sets the primary index of the collection

type DB 

type DB interface {
	DBViewer
	DBWriter

	// View provides atomic read-only access to the db for the scope of the
	// provided callback, isolated from other operations on the db.
	View(func(viewer DBViewer) error) error

	// Update provides atomic read-write access to the db for the scope of
	// the provided callback.
	Update(func(writer DBWriter) error) error

	// GetOrFill is a nice utility that wraps get, put if not exist and return
	// the value,
	GetOrFill(key string, fill func() (any, error), tags ...string) (val any, err error)

	// Invalidate deletes all keys related to the provided tags
	Invalidate(tags ...string) (deleted []string)
}

DB maintains items under keys indexed also by tags in order to be able to evict all items under a specific tag

func NewDB 

func NewDB() DB

type DBViewer 

type DBViewer interface {
	// Get safely retrieves a val identified by the provided key
	Get(key string) (val any, ok bool)

	// Iter retrieves all the keys under a tag and let you access each item in each key
	Iter(tag string, fn func(key string, getVal func() (any, bool)) (proceed bool))
}

DBViewer represents isolated read access handle to the db

type DBWriter 

type DBWriter interface {
	DBViewer

	// Put safely sets the provided val under the provided key indexed by the
	// provided tags
	Put(key string, val any)

	// Tag simply adds tag on a key
	Tag(key string, tags ...string)

	// Invalidate deletes all keys related to the provided tags
	Invalidate(tags ...string) (deleted []string)
}

DBWriter represents isolated write access handle to the db

type Derivative 

type Derivative[In any, Out any] func(in In) (Out, error)

Derivative is a fetch method that is lazily initiated from another item but only once

func Derive 

func Derive[In, Out any](collection *Collection[In], name string, fn func(in In) (out Out, err error)) Derivative[In, Out]

Derive creates a Derivative item fetcher that is stored under the provided subKey in relation to the provided item

type Getter 

type Getter[T any] func(val string) (T, bool)

Getter is a function for fetching 1 item of concrete type by a specific key

type InferredCollection 

type InferredCollection[T any] struct {
	*Collection[T]
	// contains filtered or unexported fields
}

func Infer 

func Infer[Base, Inferred any](baseCol *Collection[Base], mapBy string, mapFn mapFn[Base, Inferred]) *InferredCollection[Inferred]

Infer creates a "chained" collection in a way that for every loaded item on the base collection, the provided mapFn is called to load the inferred item

func (*InferredCollection[T]) Query 

func (i *InferredCollection[T]) Query(key string, filters ...func(T) bool) ([]T, error)

Query is the Query that created by the inferencee. the key is necessarily the primary key of the base collection.

func (*InferredCollection[T]) With 

func (i *InferredCollection[T]) With(opts ...CollectionOpt[T]) *InferredCollection[T]

With is just a proxy of the underlying Collection's method

type Query 

type Query[T any] func(key string, filters ...func(T) bool) ([]T, error)

Query is a function for fetching list of items of a concrete type by tags

type Scalar 

type Scalar[T any] func() (T, bool)

Scalar is a function for fetching singular item

type Scanner 

type Scanner[T any] func(consume func(T) bool, filters ...func(key string) bool)

Scanner is a function for iterating through items of a concrete type

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.