firestore

Documentation

Overview

Package firestore implements storage.DB and storage.VectorDB using Google Cloud's Firestore service.

A brief introduction to Firestore: it is a document DB with hierarchical keys. A key is a string of the form "coll1/id1/coll2/id2/.../collN/idN", where the colls are called "collections" and the values are called "documents". Each document is a set of key-value pairs. In Go, a document is represented by a map[string]any, or a struct with exported fields: the Go Firestore client provides convenience functions for converting documents to and from structs.

The two database implementations in this package use three collections:

• The "locks" collection holds documents representing the locks used by DB.Lock and DB.Unlock.
• The "values" collection holds the key-value pairs for DB. Keys are byte slices but document names are strings. We hex-encode the keys to obtain strings with the same ordering.
• The "vectors" collection holds embeddding vectors for VectorDB.

Index

• type DB
  • func NewDB(ctx context.Context, lg *slog.Logger, projectID, database string, ...) (*DB, error)
  • func (db *DB) Batch() storage.Batch
  • func (f DB) Close()
  • func (db *DB) Delete(key []byte)
  • func (db *DB) DeleteRange(start, end []byte)
  • func (f DB) Flush()
  • func (db *DB) Get(key []byte) ([]byte, bool)
  • func (db *DB) Lock(name string)
  • func (f DB) Panic(msg string, args ...any)
  • func (db *DB) Scan(start, end []byte) iter.Seq2[[]byte, func() []byte]
  • func (db *DB) Set(key, val []byte)
  • func (db *DB) Unlock(name string)
• type VectorDB
  • func NewVectorDB(ctx context.Context, lg *slog.Logger, projectID, database, namespace string, ...) (*VectorDB, error)
  • func (db *VectorDB) All() iter.Seq2[string, func() llm.Vector]
  • func (db *VectorDB) Batch() storage.VectorBatch
  • func (db *VectorDB) Close()
  • func (db *VectorDB) Delete(id string)
  • func (db *VectorDB) Flush()
  • func (db *VectorDB) Get(id string) (llm.Vector, bool)
  • func (db *VectorDB) Search(vec llm.Vector, n int) []storage.VectorResult
  • func (db *VectorDB) Set(id string, vec llm.Vector)

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is a connection to a Firestore database. It implements storage.DB.

func NewDB

func NewDB(ctx context.Context, lg *slog.Logger, projectID, database string, opts ...option.ClientOption) (*DB, error)

NewDB constructs a DB with the given GCP logger, project ID, Firestore database, and client options. The projectID must not be empty. If the database is empty, the default database will be used.

Key-value pairs are stored in the collection "values". Locks are stored in the collection "locks".

func (*DB) Batch

func (db *DB) Batch() storage.Batch

Batch implements storage.DB.Batch.

func (DB) Close

func (f DB) Close()

func (*DB) Delete

func (db *DB) Delete(key []byte)

Delete implements storage.DB.Delete.

func (*DB) DeleteRange

func (db *DB) DeleteRange(start, end []byte)

DeleteRange implements storage.DB.DeleteRange.

func (DB) Flush

func (f DB) Flush()

func (*DB) Get

func (db *DB) Get(key []byte) ([]byte, bool)

Get implements storage.DB.Get.

func (*DB) Lock

func (db *DB) Lock(name string)

Lock implements storage.DB.Lock. Locks are stored as documents in the "locks" collection of the database. If the document exists its content is the UID of the DB that holds it, as set in NewDB. It is an error for a DB to unlock a lock that it doesn't own. If the calling process fails, the lock times out after two minutes. Otherwise, locks are renewed every minute if Unlock is not called. There is no way to change the timeout or renew value.

func (DB) Panic

func (f DB) Panic(msg string, args ...any)

func (*DB) Scan

func (db *DB) Scan(start, end []byte) iter.Seq2[[]byte, func() []byte]

Scan implements storage.DB.Scan.

func (*DB) Set

func (db *DB) Set(key, val []byte)

Set implements storage.DB.Set.

func (*DB) Unlock

func (db *DB) Unlock(name string)

Unlock releases the lock. It panics if the lock isn't locked by this DB.

type VectorDB

type VectorDB struct {
	// contains filtered or unexported fields
}

A VectorDB is a storage.VectorDB using Firestore.

func NewVectorDB

func NewVectorDB(ctx context.Context, lg *slog.Logger, projectID, database, namespace string, opts ...option.ClientOption) (*VectorDB, error)

NewVectorDB creates a VectorDB with the given logger, GCP project ID, Firestore database, namespace and client options. The projectID must not be empty. If the database is empty, the default database will be used. The namespace must be a valid Firestore collection ID. Namespaces allow multiple vector DBs to be stored in the same Firestore DB.

Vectors in a VectorDB with namespace NS are stored in the Firestore collection "vectorDBs/NS/vectors".

func (*VectorDB) All

func (db *VectorDB) All() iter.Seq2[string, func() llm.Vector]

All implements storage.VectorDB.All.

func (*VectorDB) Batch

func (db *VectorDB) Batch() storage.VectorBatch

Batch implements storage.VectorDB.Batch.

func (*VectorDB) Close

func (db *VectorDB) Close()

Close closes the VectorDB, releasing its resources.

func (*VectorDB) Delete

func (db *VectorDB) Delete(id string)

Delete implements storage.VectorDB.Delete.

func (*VectorDB) Flush

func (db *VectorDB) Flush()

Flush implements storage.VectorDB.Flush. It is a no-op.

func (*VectorDB) Get

func (db *VectorDB) Get(id string) (llm.Vector, bool)

Get implements storage.VectorDB.Get.

func (*VectorDB) Search

func (db *VectorDB) Search(vec llm.Vector, n int) []storage.VectorResult

Search implements storage.VectorDB.Search.

func (*VectorDB) Set

func (db *VectorDB) Set(id string, vec llm.Vector)

Set implements storage.VectorDB.Set.