pomdb

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 20, 2024 License: BSD-3-Clause Imports: 20 Imported by: 0

README


pomdb-go


PomDB is a NoSQL object database that leverages the robust storage capabilities of Amazon S3 to store and retrieve data. PomDB is entirely client-driven and enforces an opinionated structure for consistency, compatibility, and speed. PomDB is in early development and may change significantly before reaching a stable release. See the issues page and roadmap to contribute or follow along.

Table of Contents

Object Databases

An object database is a type of NoSQL database that stores data as discrete objects, rather than rows and columns. Objects are self-contained units of data that can contain multiple fields, including nested objects and arrays. Object databases are schemaless, meaning that objects can have different fields and data types, and can be updated without changing the database schema.

Feature Highlights

  • Serverless client-driven architecture
  • S3-backed durability and consistency
  • Strongly-typed and schemaless data storage
  • Lexicographically sortable ULID identifiers
  • Real-time change data capture via S3 events
  • Soft-deletes for reversible data management
  • Inverted indexes for fast and efficient querying
  • Paging for huge data sets and high throughput
  • Have a feature request? Let us know

Use Cases

  • 📱 Web and Mobile Applications: user data, sessions, application state
  • 🤖 IoT and Edge Computing: large volumes of sensor data
  • 📝 Content Management Systems: metadata, versioning, content retrieval
  • 🤷♂ What are you building? Let us know

Installation

go get github.com/pomdb/pomdb-go

Quick start

package main

import (
  "log"

  "github.com/pomdb/pomdb-go"
)

type User struct {
  pomdb.Model
  FullName string `json:"full_name" pomdb:"index"`
  Email    string `json:"email" pomdb:"index,unique"`
}

var client = pomdb.Client{
  Bucket: "pomdb",
  Region: "us-east-1",
}

func main() {
  if err := client.Connect(); err != nil {
    log.Fatal(err)
  }

  user := User{
    FullName: "John Pip",
    Email:    "john.pip@zip.com",
  }

  res, err := client.Create(&user)
  if err != nil {
    log.Fatal(err)
  }

  log.Printf("Created user %s at %d", user.ID, user.CreatedAt)
}

Creating a Client

The client is used to manage the location and structure of the database. PomDB requires a dedicated bucket to store data, and the bucket must exist before the client is created.

import (
  "log"

  "github.com/pomdb/pomdb-go"
)

var client = pomdb.Client{
  Bucket: "pomdb",
  Region: "us-east-1",
}

func main() {
  if err := client.Connect(); err != nil {
    log.Fatal(err)
  }

  // ...
}

Creating a Model

Models are used to manage the structure of objects stored in collections. Models are defined using structs, with json tags to serialize the data. When embedding the pomdb.Model struct, its fields are automatically added to your model. You can choose to omit these fields, or define them manually. If you choose to define them manually, they must use the same names, types, and tags as the fields defined by PomDB:

embedding pomdb.Model

type User struct {
  pomdb.Model
  FullName  string `json:"full_name" pomdb:"index"`
  Email     string `json:"email" pomdb:"index,unique"`
}

or, defining fields manually

type User struct {
  ID        pomdb.ULID      `json:"id" pomdb:"id"`
  CreatedAt pomdb.Timestamp `json:"created_at" pomdb:"created_at"`
  UpdatedAt pomdb.Timestamp `json:"updated_at" pomdb:"updated_at"`
  DeletedAt pomdb.Timestamp `json:"deleted_at" pomdb:"deleted_at"`
  FullName  string          `json:"full_name" pomdb:"index"`
  Email     string          `json:"email" pomdb:"index,unique"`
}

Object Identifiers

PomDB automatically generates a Universally Unique Lexicographically Sortable Identifer (ULID) for each object stored in the database. IDs are stored in the ID field of the struct, and serialized to the id attribute in the json output. Models must embed the pomdb.Model struct, or define an ID field of type pomdb.ULID:

embedding pomdb.Model

type User struct {
  pomdb.Model
  FirstName string `json:"first_name" pomdb:"index"`
  LastName  string `json:"last_name" pomdb:"index"`
  Email     string `json:"email" pomdb:"index,unique"`
}

or, defining ID field manually

type User struct {
  ID        pomdb.ULID `json:"id" pomdb:"id"`
  FirstName string     `json:"first_name" pomdb:"index"`
  LastName  string     `json:"last_name" pomdb:"index"`
  Email     string     `json:"email" pomdb:"index,unique"`
  //...
}

serializes to:

{
  "id": "01HS8Q7MVGA8CVCVVFYEH1VY2T",
  "first_name": "John",
  "last_name": "Pip",
  "email": "john.pip@zip.com",
  "created_at": 1711210960,
  "updated_at": 1711210960,
  "deleted_at": 0
}

Object Timestamps

Timestamps are used to track when objects are created, updated, and deleted. The native time.Time type is used to represent timestamps, and is automatically converted to and from Unix time. Fields with the created_at, updated_at, and deleted_at tags are automatically updated by PomDB:

embedding pomdb.Model

type User struct {
  pomdb.Model
  FirstName string `json:"first_name" pomdb:"index"`
  LastName  string `json:"last_name" pomdb:"index"`
  Email     string `json:"email" pomdb:"index,unique"`
}

or, defining timestamps manually

type User struct {
  ID        pomdb.ULID      `json:"id" pomdb:"id"`
  CreatedAt pomdb.Timestamp `json:"created_at" pomdb:"created_at"`
  UpdatedAt pomdb.Timestamp `json:"updated_at" pomdb:"updated_at"`
  DeletedAt pomdb.Timestamp `json:"deleted_at" pomdb:"deleted_at"`
  FirstName string          `json:"first_name" pomdb:"index"`
  LastName  string          `json:"last_name" pomdb:"index"`
  Email     string          `json:"email" pomdb:"index,unique"`
  //...
}

serializes to:

{
  "id": "01HS8Q7MVGA8CVCVVFYEH1VY2T",
  "first_name": "John",
  "last_name": "Pip",
  "email": "john.pip@zip.com",
  "created_at": 1711210960,
  "updated_at": 1711210960,
  "deleted_at": 0
}

Working with Objects

Objects are stored in collections, and represent a single record in the database. Objects can be found in S3 under the following path:

{{$bucket}}/{{$collection}}/{{$ulid}}

Marshalling strategy

PomDB will convert the model name to snake case and pluralize it for the collection name. For example, the User model will be stored in the users collection. Fields are serialized using the json tag, and must be exported. Fields that are not exported will be ignored.

Query methods

pomdb/pomdb-go#1 Ideas for improved and expanded query features 💡

Create(model interface{})

This method is used to create a new object in the database. model must be a pointer to an interface that embeds the pomdb.Model struct, or defines an ID field of type pomdb.ULID, e.g.:

Equivalent to INSERT INTO users (id, full_name, email) VALUES (...)

user := User{
  FirstName: "John",
  LastName:  "Pip",
  Email:     "john.pip@zip.com",
}

if err := client.Create(&user); err != nil {
  log.Fatal(err)
}
Update(model interface{})

This method is used to update an existing object in the database. model must be a pointer to an interface that embeds the pomdb.Model struct, or defines an ID field of type pomdb.ULID, e.g.:

Equivalent to UPDATE users SET email = 'jane.pip@zip.com' WHERE id = '...'

user.Email = "john.pip@zap.com"

if err := client.Update(&user); err != nil {
  log.Fatal(err)
}
Delete(model interface{})

This method is used to delete an existing object in the database. model must be a pointer to an interface that embeds the pomdb.Model struct, or defines an ID field of type pomdb.ULID, e.g.:

Equivalent to DELETE FROM users WHERE id = '...'

if err := client.Delete(&user); err != nil {
  log.Fatal(err)
}
FindOne(query pomdb.Query)

This method is used to find a single object in the database using an index. The query must include the model, field name, and field value, e.g.:

Equivalent to SELECT * FROM users WHERE email = 'jane.pip@zip.com'

query := pomdb.Query{
  Model: User{},
  Field: "email",
  Value: "john.pip@zip.com",
}

res, err := client.FindOne(query)
if err != nil {
  log.Fatal(err)
}

user := res.(*User)
FindMany(query pomdb.Query)

This method is used to find multiple objects in the database using an index. The query must include the model, field name, field value, and filter, e.g.:

Equivalent to SELECT * FROM users WHERE age < 40

query := pomdb.Query{
  Model:  User{},
  Field:  "age",
  Filter: pomdb.QueryLessThan,
  Value:  40,
}

res, err := client.FindMany(query)
if err != nil {
  log.Fatal(err)
}

users := make([]User, len(res.Contents))
for i, user := range res.Contents {
  users[i] = user.(User)
}
FindAll(query pomdb.Query)

This method is used to find all objects in the database. The model must be included in the query, e.g.:

Equivalent to SELECT * FROM users

query := pomdb.Query{
  Model: User{},
}

res, err := client.FindAll(query)
if err != nil {
  log.Fatal(err)
}

users := make([]User, len(res.Contents))
for i, user := range res.Contents {
  users[i] = user.(User)
}

// ...

Query filters

PomDB provides a basic set of comparison operators for the Filter field of the query. If no filter is provided, the query will default to pomdb.QueryEqual. Filters may only be used with the FindMany method. Filters passed to other query methods will be ignored:

pomdb.QueryEqual

Equivalent to SELECT * FROM users WHERE age = 40

query := pomdb.Query{
  Model:  User{},
  Field:  "age",
  Filter: pomdb.QueryEqual,
  Value:  40,
}
pomdb.QueryLessThan

Equivalent to SELECT * FROM users WHERE age < 40

query := pomdb.Query{
  Model:  User{},
  Field:  "age",
  Filter: pomdb.QueryLessThan,
  Value:  40,
}
pomdb.QueryGreaterThan

Equivalent to SELECT * FROM users WHERE age > 40

query := pomdb.Query{
  Model:  User{},
  Field:  "age",
  Filter: pomdb.QueryGreaterThan,
  Value:  40,
}
pomdb.QueryBetween

Equivalent to SELECT * FROM users WHERE age BETWEEN 30 AND 40

query := pomdb.Query{
  Model:  User{},
  Field:  "age",
  Filter: pomdb.QueryBetween,
  Value:  []int{30, 40},
}
pomdb.QueryIn

Equivalent to SELECT * FROM users WHERE age IN (30, 40, 50)

query := pomdb.Query{
  Model:  User{},
  Field:  "age",
  Filter: pomdb.QueryIn,
  Value:  []int{30, 40, 50},
}

Soft-deletes

PomDB supports soft-deletes, allowing objects to be marked as deleted without actually removing them from the database. Soft-deleted objects are stored in the database with a non-zero DeletedAt object tag, and are automatically excluded from queries. Soft-deleted objects can be restored or purged using the Restore and Purge methods, respectively. To enable soft-deletes, set the SoftDeletes field of the client to true:

var client = pomdb.Client{
  Bucket:      "pomdb",
  Region:      "us-east-1",
  SoftDeletes: true,
}
Restore(model interface{})

This method is used to restore a soft-deleted object in the database. model must be a pointer to an interface that embeds the pomdb.Model struct, or defines an ID field of type pomdb.ULID, e.g.:

if err := client.Restore(&user); err != nil {
  log.Fatal(err)
}
Purge(model interface{})

This method is used to permanently delete a soft-deleted object and its indexes from the database. model must be a pointer to an interface that embeds the pomdb.Model struct, or defines an ID field of type pomdb.ULID, e.g.:

if err := client.Purge(&user); err != nil {
  log.Fatal(err)
}

Working with Indexes

Indexes are used to optimize queries. PomDB supports the following index types, and automatically maintains them when objects are created, updated, or deleted:

Index types

unique

Enforces uniqueness of the field's value across the collection. In the example, any Email field in User structs will be indexed uniquely. PomDB ensures no two User records have the same email.

type User struct {
  Email string `pomdb:"index,unique"` // Unique index on Email
  // ...
}

S3: /{{$col}}/indexes/unique/{{$fld}}/{{$val}}/{{$ulid}}

shared

Allows multiple records to share the same value for the indexed field. In the example, Category is indexed non-uniquely, allowing aggregation and querying of 'Product' records by shared categories.

type Product struct {
  Category string `pomdb:"index"` // Shared index on Category
  // ...
}

S3: /{{$col}}/indexes/shared/{{$fld}}/{{$val}}/{{$[]ulid}}

ranged

Facilitates queries within a range of values, like dates or numbers. In the example, Date is indexed for ranged queries, allowing for queries like events happening within a certain time frame.

type Event struct {
  Birthday pomdb.Timstamp `pomdb:"index,range"` // Range index on Date
  // ...
}

S3: /{{$col}}/indexes/ranged/{{$fld}}/{{$val}}/{{$[]ulid}}

Composite indexes

Composite indexes are used to optimize queries that involve multiple fields. In the example, IPAddress and UserAgent are indexed together as IPAddressUserAgent, allowing for queries that involve both fields. Composite indexes can be unique or shared, and are created by concatenating the field values with a delimiter, e.g. IPAddress#UserAgent:

type Log struct {
  IPAddress          string `pomdb:"index,unique"`
  UserAgent          string `pomdb:"index"`
  IPAddressUserAgent string `pomdb:"index,unique"`
  // ...
}

log := Log{
  IPAddress: "172.40.53.24",
  UserAgent: "Mozilla/5.0",
}

log.IPAddressUserAgent = log.IPAddress +"#"+ log.UserAgent

if err := client.Create(&log); err != nil {
  log.Fatal(err)
}

Encoding strategy

PomDB uses base64 encoding to store index values. This allows for a consistent and predictable way to store and retrieve objects, and ensures that the index keys are valid S3 object keys. The length of the index key is limited to 1024 bytes. If the encoded index key exceeds this limit, PomDB will return an error.

Pagination

PomDB supports pagination using the Limit and NextToken fields of the query. The Limit field is used to specify the maximum number of objects to return per page, and the NextToken field is used to specify the starting point for the next page. If there are more objects to return, PomDB will set the NextToken field of the response. If there are no more objects to return, NextToken will be an empty string:

query := pomdb.Query{
  Model: User{},
  Limit: 10,
}

res, err := client.FindAll(query)
if err != nil {
  log.Fatal(err)
}

for res.NextToken != "" {
  for _, user := range res.Contents {
    // ...
  }

  query.NextToken = res.NextToken
  res, err = client.FindAll(query)
  if err != nil {
    log.Fatal(err)
  }
}

// process the last page
for _, user := range res.Contents {
  // ...
}

Roadmap

You can view the roadmap and feature requests on the GitHub project page.

Documentation

Index

Constants

View Source 
const (
	QueryLimitDefault int = 100
)

Variables

View Source 
var ErrInvalidHex = errors.New("[Error] Model: the provided hex string is not a valid ULID")

ErrInvalidHex indicates that a hex string cannot be converted to an ObjectID.

Functions

This section is empty.

Types

type Client 

type Client struct {
	Service     *s3.Client
	Bucket      string
	Region      string
	SoftDeletes bool
	Pessimistic bool
	Optimistic  bool
}

func (*Client) CheckBucket 

func (c *Client) CheckBucket() error

func (*Client) CheckIndexExists 

func (c *Client) CheckIndexExists(ca *ModelCache) error

CheckIndexExists checks if an index item exists in the given collection.

func (*Client) Connect 

func (c *Client) Connect() error

func (*Client) Create 

func (c *Client) Create(i interface{}) (*string, error)

Create creates a record in the database

func (*Client) CreateIndexItems 

func (c *Client) CreateIndexItems(ca *ModelCache) error

CreateIndexItems creates an index item in the given collection.

func (*Client) Delete 

func (c *Client) Delete(i interface{}) (*string, error)

Delete deletes a record and its indexes from the database.

func (*Client) DeleteIndexItems 

func (c *Client) DeleteIndexItems(ca *ModelCache) error

DeleteIndexItems deletes index items in the given collection.

func (*Client) FindAll 

func (c *Client) FindAll(q Query) (*FindAllResult, error)

FindAll returns all objects of a given collection.

func (*Client) FindMany 

func (c *Client) FindMany(q Query) (*FindManyResult, error)

FindMany retrieves multiple objects of a given index.

func (*Client) FindOne 

func (c *Client) FindOne(q Query) (interface{}, error)

FindOne retrieves a single object of a given collection or index.

func (*Client) Purge 

func (c *Client) Purge(i interface{}) (*string, error)

Purge permanently removes a soft-deleted record and its indexes from the database.

func (*Client) Restore 

func (c *Client) Restore(i interface{}) (*string, error)

Restore restores soft-deleted records and indexes in the database.

func (*Client) SoftDelete 

func (c *Client) SoftDelete(i interface{}) (*string, error)

SoftDelete soft deletes a record and its indexes from the database.

func (*Client) Update 

func (c *Client) Update(i interface{}) (*string, error)

Update updates a record in the database.

func (*Client) UpdateIndexItems 

func (c *Client) UpdateIndexItems(ca *ModelCache) error

UpdateIndexItems updates index items in the given collection.

type FindAllResult 

type FindAllResult struct {
	Docs      []interface{}
	NextToken string
}

type FindManyResult 

type FindManyResult struct {
	Docs      []interface{}
	NextToken string
}

type IndexField 

type IndexField struct {
	FieldName     string
	FieldType     reflect.Type
	CurrentValue  string
	PreviousValue string
	IndexType     IndexType
}

type IndexType 

type IndexType int
const (
	UniqueIndex IndexType = iota
	SharedIndex
	RangedIndex
)

type Model 

type Model struct {
	ID        ULID      `json:"id" pomdb:"id"`
	CreatedAt Timestamp `json:"created_at" pomdb:"created_at"`
	UpdatedAt Timestamp `json:"updated_at" pomdb:"updated_at"`
	DeletedAt Timestamp `json:"deleted_at" pomdb:"deleted_at"`
}

type ModelCache 

type ModelCache struct {
	ModelID     *reflect.Value
	IndexFields []IndexField
	CreatedAt   *reflect.Value
	UpdatedAt   *reflect.Value
	DeletedAt   *reflect.Value
	Collection  string
	Reference   interface{}
}

func NewModelCache 

func NewModelCache(rv reflect.Value) *ModelCache

func (*ModelCache) CompareIndexFields 

func (mc *ModelCache) CompareIndexFields(model interface{}) bool

CompareIndexFields compares the index fields in the cache to the input.

func (*ModelCache) GetModelID 

func (mc *ModelCache) GetModelID() string

GetModelID returns the model ID from the cache.

func (*ModelCache) SetDeletedAt 

func (mc *ModelCache) SetDeletedAt()

SetDeletedAt sets the DeletedAt field in the cache.

func (*ModelCache) SetManagedFields 

func (mc *ModelCache) SetManagedFields()

SetManagedFields sets the managed fields in the cache.

func (*ModelCache) SetUpdatedAt 

func (mc *ModelCache) SetUpdatedAt()

SetUpdatedAt sets the UpdatedAt field in the cache.

type Query 

type Query struct {
	Model     interface{}
	Field     string
	Value     any
	Filter    QueryFilter
	Limit     int
	NextToken string
}

func (*Query) Compare 

func (q *Query) Compare(obj types.Object, idx *IndexField) (bool, error)

FilterResults filters the results of a query based on the query filter.

type QueryFilter 

type QueryFilter int
const (
	QueryEqual QueryFilter = iota
	QueryGreaterThan
	QueryLessThan
	QueryBetween
)

type Timestamp 

type Timestamp time.Time

func NewTimestamp 

func NewTimestamp() Timestamp

func NilTimestamp 

func NilTimestamp() Timestamp

func (Timestamp) After 

func (t Timestamp) After(u Timestamp) bool

func (Timestamp) Before 

func (t Timestamp) Before(u Timestamp) bool

func (Timestamp) Equal 

func (t Timestamp) Equal(u Timestamp) bool

func (Timestamp) IsNil 

func (t Timestamp) IsNil() bool

IsNil returns true if the Timestamp is the zero value.

func (Timestamp) MarshalJSON 

func (ts Timestamp) MarshalJSON() ([]byte, error)

MarshalJSON customizes the JSON representation of Timestamp.

func (Timestamp) String 

func (t Timestamp) String() string

String returns the string representation of the Timestamp.

func (*Timestamp) UnmarshalJSON 

func (ts *Timestamp) UnmarshalJSON(b []byte) error

UnmarshalJSON populates the Timestamp from a JSON representation.

func (*Timestamp) UnmarshalText 

func (ts *Timestamp) UnmarshalText(b []byte) error

UnmarshalText populates the Timestamp from a text representation.

type ULID 

type ULID ulid.ULID

ObjectID is the BSON ObjectID type represented as a 12-byte array.

func NewULID 

func NewULID() ULID

NewULID generates a new ObjectID.

func (ULID) MarshalJSON 

func (id ULID) MarshalJSON() ([]byte, error)

MarshalJSON customizes the JSON representation of ULID.

func (ULID) String 

func (id ULID) String() string

func (*ULID) UnmarshalJSON 

func (id *ULID) UnmarshalJSON(b []byte) error

UnmarshalJSON populates the ULID from a JSON representation.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.