genji

README

Genji

Document-oriented, embedded, SQL database

Table of contents

• Introduction
• Features
• Installation
• Usage
• Engines
• Genji shell
• Contributing

Introduction

Genji is a schemaless database that allows running SQL queries on documents.

Checkout the SQL documentation, the Go doc and the usage example in the README to get started quickly.

⚠ Genji's API is still unstable: Database compatibility is not guaranteed before reaching v1.0.0

Features

• Optional schemas: Genji tables are schemaless, but it is possible to add constraints on any field to ensure the coherence of data within a table.
• Multiple Storage Engines: It is possible to store data on disk or in ram, but also to choose between B-Trees and LSM trees. Genji relies on BoltDB and Badger to manage data.
• Transaction support: Read-only and read/write transactions are supported by default.
• SQL and Documents: Genji mixes the best of both worlds by combining powerful SQL commands with JSON dot notation.
• Easy to use, easy to learn: Genji was designed for simplicity in mind. It is really easy to insert and read documents of any shape.
• Compatible with the database/sql package

Installation

Install the Genji database

go get github.com/genjidb/genji

Usage

There are two ways of using Genji, either by using Genji's API or by using the database/sql package.

Using Genji's API

// Create a database instance, here we'll store everything on-disk using the BoltDB engine
db, err := genji.Open("my.db")
if err != nil {
    log.Fatal(err)
}
// Don't forget to close the database when you're done
defer db.Close()

// Create a table. Genji tables are schemaless, you don't need to specify a schema if not needed.
err = db.Exec("CREATE TABLE user")

// Create an index.
err = db.Exec("CREATE INDEX idx_user_name ON test (name)")

// Insert some data
err = db.Exec("INSERT INTO user (id, name, age) VALUES (?, ?, ?)", 10, "Foo1", 15)

// Supported values can go from simple integers to richer data types like lists or documents
err = db.Exec(`
    INSERT INTO user (id, name, age, address, friends)
    VALUES (
        11,
        'Foo2',
        20,
        {"city": "Lyon", "zipcode": "69001"},
        ["foo", "bar", "baz"]
    )`)

// It is also possible to insert values using document notation, which is a JSON-like notation with support for expressions.
err = db.Exec(`
    INSERT INTO user
    VALUES {
        id: 11,
        name: 'Foo2',
        "age": 20,
        "address": {"city": "Lyon", "zipcode": "69001"},
        "friends": ["foo", "bar", "baz"],
        single: 1 AND 1
    }`)

// Or even to use structures
type User struct {
    ID              uint
    Name            []byte
    TheAgeOfTheUser float64 `genji:"age"`
    Address         struct {
        City    string
        ZipCode string
    }
}

// Let's create a user
u := User{
    ID: 20,
    Name: "foo",
    TheAgeOfTheUser: 40,
}
u.Address.City = "Lyon"
u.Address.ZipCode = "69001"

err := db.Exec(`INSERT INTO user VALUES ?`, &u)

// Use a transaction
tx, err := db.Begin(true)
defer tx.Rollback()
err = tx.Exec("INSERT INTO user (id, name, age) VALUES (?, ?, ?)", 12, "Foo3", 25)
...
err = tx.Commit()

// Query some documents
res, err := db.Query("SELECT id, name, age, address FROM user WHERE age >= ?", 18)
// always close the result when you're done with it
defer res.Close()

// Iterate over the results
err = res.Iterate(func(d document.Document) error {
    // When querying an explicit list of fields, you can use the Scan function to scan them
    // in order. Note that the types don't have to match exactly the types stored in the table
    // as long as they are compatible.
    var id int
    var name string
    var age int32
    var address struct{
        City string
        ZipCode string
    }

    err = document.Scan(d, &id, &name, &age, &address)
    if err != nil {
        return err
    }

    fmt.Println(id, name, age, address)

    // It is also possible to scan the results into a structure
    var u User
    err = document.StructScan(d, &user)
    if err != nil {
        return err
    }

    fmt.Println(u)

    // Or scan into a map
    var m map[string]interface{}
    err = document.MapScan(d, &m)
    if err != nil {
        return err
    }

    fmt.Println(m)
    return nil
})

// Count results
count, err := res.Count()

// Get first document from the results using the First method of the stream
d, err := res.First()

// Apply some transformations
err = res.
    // Filter all even ids
    Filter(func(d document.Document) (bool, error) {
        f, err := d.GetByField("id")
        ...
        id, err := f.DecodeToInt()
        ...
        return id % 2 == 0, nil
    }).
    // Enrich the documents with a new field
    Map(func(d document.Document) (document.Document, error) {
        var fb document.FieldBuffer

        err := fb.ScanDocument(r)
        ...
        fb.Add(document.NewTextValue("group", "admin"))
        return &fb, nil
    }).
    // Iterate on them
    Iterate(func(d document.Document) error {
        ...
    })

Using database/sql

// import Genji as a blank import
import _ "github.com/genjidb/genji/sql/driver"

// Create a sql/database DB instance
db, err := sql.Open("genji", "my.db")
if err != nil {
    log.Fatal(err)
}
defer db.Close()

// Then use db as usual
res, err := db.ExecContext(...)
res, err := db.Query(...)
res, err := db.QueryRow(...)

Engines

Genji currently supports storing data in BoltDB, Badger and in-memory.

Using the BoltDB engine

import (
    "log"

    "github.com/genjidb/genji"
)

func main() {
    db, err := genji.Open("my.db")
    defer db.Close()
}

Using the memory engine

import (
    "log"

    "github.com/genjidb/genji"
)

func main() {
    db, err := genji.Open(":memory:")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
}

Using the Badger engine

import (
    "log"

    "github.com/genjidb/genji"
    "github.com/genjidb/genji/engine/badgerengine"
    "github.com/dgraph-io/badger/v2"
)

func main() {
    // Create a badger engine
    ng, err := badgerengine.NewEngine(badger.DefaultOptions("mydb"))
    if err != nil {
        log.Fatal(err)
    }

    // Pass it to genji
    db, err := genji.New(ng)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
}

Genji shell

The genji command line provides an SQL shell that can be used to create, modify and consult Genji databases.

Make sure the Genji command line is installed:

go get github.com/genjidb/genji/cmd/genji

Example:

# Opening an in-memory database:
genji

# Opening a BoltDB database:
genji my.db

# Opening a Badger database:
genji --badger pathToData

Contributing

Contributions are welcome!

• Feedback
• Feature ideas
• Bug reports
• Pull Requests
• Anything that comes to mind to improve Genji!

If you have any doubt, join the Gophers Slack channel or open an issue.

Documentation

Overview

Package genji implements a document-oriented, embedded SQL database. Genji supports various engines that write data on-disk, like BoltDB or Badger, and in memory.

Example

package main

import (
	"encoding/json"
	"fmt"
	"os"

	"github.com/genjidb/genji"
	"github.com/genjidb/genji/document"
)

type User struct {
	ID      int64
	Name    string
	Age     uint32
	Address struct {
		City    string
		ZipCode string
	}
}

func main() {
	// Create a database instance, here we'll store everything in memory
	db, err := genji.Open(":memory:")
	if err != nil {
		panic(err)
	}
	defer db.Close()

	// Create a table. Genji tables are schemaless by default, you don't need to specify a schema.
	err = db.Exec("CREATE TABLE user")
	if err != nil {
		panic(err)
	}

	// Create an index.
	err = db.Exec("CREATE INDEX idx_user_name ON user (name)")
	if err != nil {
		panic(err)
	}

	// Insert some data
	err = db.Exec("INSERT INTO user (id, name, age) VALUES (?, ?, ?)", 10, "foo", 15)
	if err != nil {
		panic(err)
	}

	// Insert some data using document notation
	err = db.Exec(`INSERT INTO user VALUES {id: 12, "name": "bar", age: ?, address: {city: "Lyon", zipcode: "69001"}}`, 16)
	if err != nil {
		panic(err)
	}

	// Structs can be used to describe a document
	err = db.Exec("INSERT INTO user VALUES ?, ?", &User{ID: 1, Name: "baz", Age: 100}, &User{ID: 2, Name: "bat"})
	if err != nil {
		panic(err)
	}

	// Query some documents
	stream, err := db.Query("SELECT * FROM user WHERE id > ?", 1)
	if err != nil {
		panic(err)
	}
	// always close the result when you're done with it
	defer stream.Close()

	// Iterate over the results
	err = stream.Iterate(func(d document.Document) error {
		var u User

		err = document.StructScan(d, &u)
		if err != nil {
			return err
		}

		fmt.Println(u)
		return nil
	})
	if err != nil {
		panic(err)
	}

	// Count results
	count, err := stream.Count()
	if err != nil {
		panic(err)
	}
	fmt.Println("Count:", count)

	// Get first document from the results
	d, err := stream.First()
	if err != nil {
		panic(err)
	}

	// Scan into a struct
	var u User
	err = document.StructScan(d, &u)
	if err != nil {
		panic(err)
	}

	enc := json.NewEncoder(os.Stdout)

	// Apply some manual transformations
	err = stream.
		// Filter all even ids
		Filter(func(d document.Document) (bool, error) {
			v, err := d.GetByField("id")
			if err != nil {
				return false, err
			}
			return v.V.(int64)%2 == 0, err
		}).
		// Enrich the documents with a new field
		Map(func(d document.Document) (document.Document, error) {
			var fb document.FieldBuffer

			err := fb.ScanDocument(d)
			if err != nil {
				return nil, err
			}

			fb.Add("group", document.NewTextValue("admin"))
			return &fb, nil
		}).
		// Iterate on them
		Iterate(func(d document.Document) error {
			return enc.Encode(d)
		})

	if err != nil {
		panic(err)
	}

}

Output:

{10 foo 15 { }}
{12 bar 16 {Lyon 69001}}
{2 bat 0 { }}
Count: 3
{"id":10,"name":"foo","age":15,"group":"admin"}
{"id":12,"name":"bar","age":16,"address":{"city":"Lyon","zipcode":"69001"},"group":"admin"}
{"id":2,"name":"bat","age":0,"address":{"city":"","zipcode":""},"group":"admin"}

Index

• type DB
  • func New(ng engine.Engine) (*DB, error)
  • func Open(path string) (*DB, error)
  • func (db *DB) Begin(writable bool) (*Tx, error)
  • func (db *DB) Close() error
  • func (db *DB) Exec(q string, args ...interface{}) error
  • func (db *DB) Query(q string, args ...interface{}) (*query.Result, error)
  • func (db *DB) QueryDocument(q string, args ...interface{}) (document.Document, error)
  • func (db *DB) Update(fn func(tx *Tx) error) error
  • func (db *DB) View(fn func(tx *Tx) error) error
• type Tx
  • func (tx *Tx) Exec(q string, args ...interface{}) error
  • func (tx *Tx) Query(q string, args ...interface{}) (*query.Result, error)
  • func (tx *Tx) QueryDocument(q string, args ...interface{}) (document.Document, error)

Examples

• Package
• Tx

Types

type DB

type DB struct {
	DB *database.Database
}

DB represents a collection of tables stored in the underlying engine.

func New

func New(ng engine.Engine) (*DB, error)

New initializes the DB using the given engine.

func Open

func Open(path string) (*DB, error)

Open creates a Genji database at the given path. If path is equal to ":memory:" it will open an in memory database, otherwise it will create an on-disk database using the BoltDB engine.

func (*DB) Begin

func (db *DB) Begin(writable bool) (*Tx, error)

Begin starts a new transaction. The returned transaction must be closed either by calling Rollback or Commit.

func (*DB) Close

func (db *DB) Close() error

Close the database.

func (*DB) Exec

func (db *DB) Exec(q string, args ...interface{}) error

Exec a query against the database without returning the result.

func (*DB) Query

func (db *DB) Query(q string, args ...interface{}) (*query.Result, error)

Query the database and return the result. The returned result must always be closed after usage.

func (*DB) QueryDocument

func (db *DB) QueryDocument(q string, args ...interface{}) (document.Document, error)

QueryDocument runs the query and returns the first document. If the query returns no error, QueryDocument returns database.ErrDocumentNotFound.

func (*DB) Update

func (db *DB) Update(fn func(tx *Tx) error) error

Update starts a read-write transaction, runs fn and automatically commits it.

func (*DB) View

func (db *DB) View(fn func(tx *Tx) error) error

View starts a read only transaction, runs fn and automatically rolls it back.

type Tx

type Tx struct {
	*database.Transaction
}

Tx represents a database transaction. It provides methods for managing the collection of tables and the transaction itself. Tx is either read-only or read/write. Read-only can be used to read tables and read/write can be used to read, create, delete and modify tables.

Example

db, err := genji.Open(":memory:")
if err != nil {
	log.Fatal(err)
}
defer db.Close()

tx, err := db.Begin(true)
if err != nil {
	panic(err)
}
defer tx.Rollback()

err = tx.Exec("CREATE TABLE IF NOT EXISTS user")
if err != nil {
	log.Fatal(err)
}

err = tx.Exec("INSERT INTO user (id, name, age) VALUES (?, ?, ?)", 10, "foo", 15)
if err != nil {
	log.Fatal(err)
}

d, err := tx.QueryDocument("SELECT id, name, age FROM user WHERE name = ?", "foo")
if err != nil {
	panic(err)
}

var u User
err = document.StructScan(d, &u)
if err != nil {
	panic(err)
}

fmt.Println(u)

var id uint64
var name string
var age uint8

err = document.Scan(d, &id, &name, &age)
if err != nil {
	panic(err)
}

fmt.Println(id, name, age)

err = tx.Commit()
if err != nil {
	panic(err)
}

Output:

{10 foo 15 { }}
10 foo 15

func (*Tx) Exec

func (tx *Tx) Exec(q string, args ...interface{}) error

Exec a query against the database within tx and without returning the result.

func (*Tx) Query

func (tx *Tx) Query(q string, args ...interface{}) (*query.Result, error)

Query the database withing the transaction and returns the result. Closing the returned result after usage is not mandatory.

func (*Tx) QueryDocument

func (tx *Tx) QueryDocument(q string, args ...interface{}) (document.Document, error)

QueryDocument runs the query and returns the first document. If the query returns no error, QueryDocument returns database.ErrDocumentNotFound.