firestorm

README

go-firestorm

Go ORM (Object-relational mapping) for Google Cloud Firestore.

Goals

1. Easy to use
2. Non-intrusive
3. Non-exclusive
4. Fast

Features

• Basic CRUD operations
• Search
• Concurrent requests support (except when run in transactions)
• Transactions
• Nested transactions will reuse the first transaction (reads before writes as required by firestore)
• Configurable auto load of references
• Handles cyclic references
• Sub collections
• Supports embedded/anonymous structs
• Supports unexported fields
• Custom mappers between fields and types
• Caching (session + second level)
• Supports Google App Engine - 2. Gen (go version >= 1.11)

Getting Started

• Prerequisites
• Basic CRUD example
• Search
• Concurrent requests
• Transactions
• Cache
• Configurable auto load of references
• Customize data mapping
• Help

Prerequisites

This library only supports Firestore Native mode and not the old Datastore mode.

go get -u github.com/jschoedt/go-firestorm

Setup

1. Setup a Firestore client
2. Create a firestorm client and supply the names of the id and parent fields of your model structs. Parent is optional. The id field must be a string but can be called anything.

...
client, _ := app.Firestore(ctx)
fsc := firestorm.New(client, "ID", "")

1. Optional. For optimal caching to work consider adding the CacheHandler.

Basic CRUD example

Note: Recursive Create/Delete is not supported and must be called on every entity. So to create an A->B relation. Create B first so the B.ID has been created and then create A.

type Car struct {
	ID         string
	Make       string
	Year       time.Time
}

car := &Car{}
car.Make = "Toyota"
car.Year, _ = time.Parse(time.RFC3339, "2001-01-01T00:00:00.000Z")

// Create the entity
fsc.NewRequest().CreateEntities(ctx, car)()

if car.ID == "" {
    t.Errorf("car should have an auto generated ID")
}

// Read the entity by ID
otherCar := &Car{ID:car.ID}
fsc.NewRequest().GetEntities(ctx, otherCar)()
if otherCar.Make != "Toyota" {
    t.Errorf("car should have name: Toyota but was: %s", otherCar.Make)
}
if otherCar.Year != car.Year {
    t.Errorf("car should have same year: %s", otherCar.Year)
}

// Update the entity
car.Make = "Jeep"
fsc.NewRequest().UpdateEntities(ctx, car)()

otherCar := &Car{ID:car.ID}
fsc.NewRequest().GetEntities(ctx, otherCar)()
if otherCar.Make != "Jeep" {
    t.Errorf("car should have name: Jeep but was: %s", otherCar.Make)
}

// Delete the entity
fsc.NewRequest().DeleteEntities(ctx, car)()

otherCar = &Car{ID:car.ID}
if err := fsc.NewRequest().GetEntities(ctx, otherCar)(); err == nil {
    t.Errorf("We expect a NotFoundError")
}

More examples

Search

Create a query using the firebase client

car := &Car{}
car.ID = "testID"
car.Make = "Toyota"

fsc.NewRequest().CreateEntities(ctx, car)()

query := fsc.Client.Collection("Car").Where("make", "==", "Toyota")

result := make([]Car, 0)
if err := fsc.NewRequest().QueryEntities(ctx, query, &result)(); err != nil {
    t.Errorf("car was not found by search: %v", car)
}

if result[0].ID != car.ID || result[0].Make != car.Make {
    t.Errorf("entity did not match original entity : %v", result)
}

More examples

Concurrent requests

All CRUD operations are asynchronous and return a future func that when called will block until the operation is done.

NOTE: the state of the entities is undefined until the future func returns.

car := &Car{Make:"Toyota"}

// Create the entity which returns a future func
future := fsc.NewRequest().CreateEntities(ctx, car)

// ID is not set
if car.ID != "" {
	t.Errorf("car ID should not have been set yet")
}

// do some more work

// blocks and waits for the database to finish
future()

// now the car has been saved and the ID has been set
if car.ID == "" {
    t.Errorf("car should have an auto generated ID now")
}

More examples

Transactions

Transactions are simply done in a function using the transaction context

car := &Car{Make: "Toyota"}

fsc.DoInTransaction(ctx, func(transCtx context.Context) error {

    // Create the entity in the transaction using the transCtx
    fsc.NewRequest().CreateEntities(transCtx, car)()

    // Using the transCtx we can load the entity as it is saved in the session context
    otherCar := &Car{ID:car.ID}
    fsc.NewRequest().GetEntities(transCtx, otherCar)()
    if otherCar.Make != car.Make {
        t.Errorf("The car should have been saved in the transaction context")
    }

    // Loading using an other context (request) will fail as the car is not created until the func returns successfully
    if err := fsc.NewRequest().GetEntities(ctx, &Car{ID:car.ID})(); err == nil {
        t.Errorf("We expect a NotFoundError")
    }
})

// Now we can load the car as the transaction has been committed
otherCar := &Car{ID:car.ID}
fsc.NewRequest().GetEntities(ctx, otherCar)()
if otherCar.Make != "Toyota" {
    t.Errorf("car should have name: Toyota but was: %s", otherCar.Make)
}

More examples

Cache

Firestorm supports adding a session cache to the context. The session cache only caches entities that are loaded within the same request.

# add it to a single handler:
http.HandleFunc("/", firestorm.CacheHandler(otherHandler))
# or add it to the routing chain (for gorilla/mux, go-chi etc.):
r.Use(firestorm.CacheMiddleware)

To add a second level cache (such as Redis or memcache) the Cache interface needs to be implemented and added to the client:

fsc.SetCache(c)

Firestore will first try to fetch an entity from the session cache. If it is not found it will try the second level cache.

Configurable auto load of references

Use the req.SetLoadPaths("fieldName") to auto load a particular field or req.SetLoadPaths(firestorm.AllEntities) to load all fields.

Load an entity path by adding multiple paths eg.: path->to->field

fsc.NewRequest().SetLoadPaths("path", "path.to", "path.to.field").GetEntities(ctx, car)()

More examples

Customize data mapping

This library uses go-structmapper for mapping values between Firestore and structs. The mapping can be customized by setting the mappers:

fsc.MapToDB = mapper.New()
fsc.MapFromDB = mapper.New()

Help

Help is provided in the go-firestorm User Group

Documentation

Index

• Constants
• Variables
• func CacheHandler(next http.HandlerFunc) http.HandlerFunc
• func CacheMiddleware(next http.Handler) http.Handler
• type Cache
• type EntityMap
  • func (entity EntityMap) Copy() EntityMap
• type FSClient
  • func New(client *firestore.Client, id, parent string) *FSClient
  • func (fsc *FSClient) DefaultFromDBMapperFunc(inKey string, inVal interface{}) (mt mapper.MappingType, outKey string, outVal interface{})
  • func (fsc *FSClient) DefaultToDBMapperFunc(inKey string, inVal interface{}) (mt mapper.MappingType, outKey string, outVal interface{})
  • func (fsc *FSClient) DoInTransaction(ctx context.Context, f func(tctx context.Context) error) error
  • func (fsc *FSClient) NewRequest() *Request
  • func (fsc *FSClient) SetCache(cache Cache)
• type FutureFunc
• type NotFoundError
  • func (e NotFoundError) Error() string
• type Request
  • func (req *Request) CreateEntities(ctx context.Context, entities interface{}) FutureFunc
  • func (req *Request) DeleteEntities(ctx context.Context, entities interface{}) FutureFunc
  • func (req *Request) GetEntities(ctx context.Context, entities interface{}) func() ([]interface{}, error)
  • func (req *Request) GetID(entity interface{}) string
  • func (req *Request) GetParent(entity interface{}) interface{}
  • func (req *Request) QueryEntities(ctx context.Context, query firestore.Query, toSlicePtr interface{}) FutureFunc
  • func (req *Request) SetID(entity interface{}, id string)
  • func (req *Request) SetLoadPaths(paths ...string) *Request
  • func (req *Request) SetMapperFunc(mapperFunc mapperFunc) *Request
  • func (req *Request) ToCollection(entity interface{}) *firestore.CollectionRef
  • func (req *Request) ToRef(entity interface{}) *firestore.DocumentRef
  • func (req *Request) UpdateEntities(ctx context.Context, entities interface{}) FutureFunc

Constants

const AllEntities = "ALL"

AllEntities loads all paths on the struct see: SetLoadPaths

Variables

var (
	// SessionCacheKey is the key for the session map in the context
	SessionCacheKey = contextKey("sessionCache")
	// ErrCacheMiss returned on a cache miss
	ErrCacheMiss = errors.New("not found in cache")
)

Functions

func CacheHandler

func CacheHandler(next http.HandlerFunc) http.HandlerFunc

CacheHandler should be used on the mux chain to support session cache. So getting the same entity several times will only generate on DB hit

func CacheMiddleware

func CacheMiddleware(next http.Handler) http.Handler

CacheMiddleware can be used as a CacheHandler middleware for popular routing frameworks e.g. gorilla/mux, go-chi, ... r.Use(firestorm.CacheMiddleware)

Types

type Cache

type Cache interface {
	Get(ctx context.Context, key string) (EntityMap, error)
	GetMulti(ctx context.Context, keys []string) (map[string]EntityMap, error)
	Set(ctx context.Context, key string, item EntityMap) error
	SetMulti(ctx context.Context, items map[string]EntityMap) error
	Delete(ctx context.Context, key string) error
	DeleteMulti(ctx context.Context, keys []string) error
}

Cache can be used to implement custom caching

type EntityMap

type EntityMap map[string]interface{}

func (EntityMap) Copy

func (entity EntityMap) Copy() EntityMap

Copy returns a 'shallow' copy of the map.

type FSClient

type FSClient struct {
	Client           *firestore.Client
	MapToDB          *mapper.Mapper
	MapFromDB        *mapper.Mapper
	IDKey, ParentKey string
	Cache            *cacheWrapper
	IsEntity         func(i interface{}) bool
}

FSClient is the client used to perform the CRUD actions

func New

func New(client *firestore.Client, id, parent string) *FSClient

New creates a firestorm client. Supply the names of the id and parent fields of your model structs Leave parent blank if sub-collections are not used.

func (*FSClient) DefaultFromDBMapperFunc

func (fsc *FSClient) DefaultFromDBMapperFunc(inKey string, inVal interface{}) (mt mapper.MappingType, outKey string, outVal interface{})

DefaultFromDBMapperFunc default mapper that maps firestore fields and values to entity fields and values

func (*FSClient) DefaultToDBMapperFunc

func (fsc *FSClient) DefaultToDBMapperFunc(inKey string, inVal interface{}) (mt mapper.MappingType, outKey string, outVal interface{})

DefaultToDBMapperFunc default mapper that maps entity fields and values to be firestore fields and values

func (*FSClient) DoInTransaction

func (fsc *FSClient) DoInTransaction(ctx context.Context, f func(tctx context.Context) error) error

DoInTransaction wraps any updates that needs to run in a transaction. Use the transaction context tctx for any calls that need to be part of the transaction. Do reads before writes as required by firestore

func (*FSClient) NewRequest

func (fsc *FSClient) NewRequest() *Request

NewRequest creates a new CRUD Request to firestore

func (*FSClient) SetCache

func (fsc *FSClient) SetCache(cache Cache)

SetCache sets a second level cache besides the session cache. Use it for eg. memcache or redis

type FutureFunc

type FutureFunc func() error

FutureFunc is a function that when called blocks until the result is ready

type NotFoundError

type NotFoundError struct {
	// Refs contains the references not found
	Refs map[string]*firestore.DocumentRef
}

NotFoundError is returned when any of the entities are not found in firestore The error can be ignored if dangling references is not a problem

func (NotFoundError) Error

func (e NotFoundError) Error() string

type Request

type Request struct {
	FSC *FSClient
	// contains filtered or unexported fields
}

Request a request builder for querying firestore

func (*Request) CreateEntities

func (req *Request) CreateEntities(ctx context.Context, entities interface{}) FutureFunc

CreateEntities creates the entities and auto creates the id if left empty. Supply either a struct or a slice as value or reference.

func (*Request) DeleteEntities

func (req *Request) DeleteEntities(ctx context.Context, entities interface{}) FutureFunc

DeleteEntities deletes the entities. Supply either a struct or a slice as value or reference.

func (*Request) GetEntities

func (req *Request) GetEntities(ctx context.Context, entities interface{}) func() ([]interface{}, error)

GetEntities reads the entities from the database by their id. Supply either a pointer to a struct or pointer to a slice. Returns a slice containing the found entities and an error if some entities are not found.

func (*Request) GetID

func (req *Request) GetID(entity interface{}) string

GetID gets the id of the entity. It panics if the entity does not have an ID field.

func (*Request) GetParent

func (req *Request) GetParent(entity interface{}) interface{}

GetParent gets the patent of the entity

func (*Request) QueryEntities

func (req *Request) QueryEntities(ctx context.Context, query firestore.Query, toSlicePtr interface{}) FutureFunc

QueryEntities query for entities. Supply a reference to a slice for the result

func (*Request) SetID

func (req *Request) SetID(entity interface{}, id string)

SetID sets the id field to the given id

func (*Request) SetLoadPaths

func (req *Request) SetLoadPaths(paths ...string) *Request

SetLoadPaths adds the paths (refs) to load for the entity. Eg. to load a users grandmother: 'mother.mother' To load all refs on the struct use firestorm.AllEntities See examples: https://github.com/jschoedt/go-firestorm/blob/master/tests/integration_test.go

func (*Request) SetMapperFunc

func (req *Request) SetMapperFunc(mapperFunc mapperFunc) *Request

SetMapperFunc is called before the map is saved to firestore. This can be used to modify the map before it is saved

func (*Request) ToCollection

func (req *Request) ToCollection(entity interface{}) *firestore.CollectionRef

ToCollection creates a firestore CollectionRef to the entity

func (*Request) ToRef

func (req *Request) ToRef(entity interface{}) *firestore.DocumentRef

ToRef creates a firestore DocumentRef for the entity

func (*Request) UpdateEntities

func (req *Request) UpdateEntities(ctx context.Context, entities interface{}) FutureFunc

UpdateEntities updates the entities. Supply either a struct or a slice as value or reference.