gitdb

package module
v2.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 3, 2020 License: MIT Imports: 30 Imported by: 0

README

GitDB

Go Report Card Coverage Build Status Travis Godoc Releases LICENSE

What is GitDB?

GitDB is not a binary. It’s a library!

GitDB is a decentralized document database written in Go. It provides database-like functionalities via strictly defined interfaces.

GitDB allows developers to create Models of objects in their application which implement a Model Interface that can access it's persistence features. This allows GitDB to work with these objects in database operations.

Why GitDB - motivation behind project?

  • A need for a database that was quick and simple to set up
  • A need for a database that was decentralized and each participating client in a system can store their data independent of other clients.

Features

  • Decentralized
  • Document store
  • Embedded into your go application
  • Encryption (encrypt on write, decrypt on read)
  • Record locking.
  • Simple Indexing System
  • Transactions
  • Web UI

Project versioning

GitDB uses semantic versioning. API should not change between patch and minor releases. New minor versions may add additional features to the API.

Table of Contents

Getting Started

Installing

To start using GitDB, install Go and run go get:

$ go get github.com/fobilow/gitdb/v2
Configuration

Below are configuration options provided by GitDB

Name Description Type Required Default
DbPath Path on computer where you want GitDB to create/clone your database string Y N/A
ConnectionName Unique name for gitdb connection. Use this when opening multiple GitDB connections string N "default"
OnlineRemote URL for remote git server you want GitDB to sync with e.g git@github.com:user/db.git or https://github.com/user/db.git.

Note: GitDB automatically generates ssh keys for every database and will automatically use this key to sync with the OnlineRemote, therefore ensure that the generated keys are added to this git server. The ssh keys can be found at Config.DbPath/.gitdb/.ssh

 string N ""
SyncInterval This controls how often you want GitDB to sync with the online remote time.Duration. N 5s
EncryptionKey 16,24, or 32 byte string used to provide AES encryption for Models that implement ShouldEncrypt string N ""
User This specifies the user connected to the Gitdb and will be used to commit all changes to the database gitdb.User N ghost <ghost@local>
UIPort Use this option to change the default port which GitDB uses to serve it's web user interface int N 4120
Factory For backward compatibity with v1. In v1 GitDB needed a factory method to be able construct concrete Model for certain database operations. This has now been dropped in v2 func(dataset string) gitdb.Model N nil

You can configure GitDB either using the constructor or constructing it yourself

cfg := gitdb.NewConfig(path)
//or
cfg := gitdb.Config{
  DbPath: path
}
Importing GitDB

To use GitDB as an embedded document store, import as:

import "github.com/fobilow/gitdb/v2"

cfg := gitdb.NewConfig(path)
db, err := gitdb.Open(cfg)
if err != nil {
  log.Fatal(err)
}
defer db.Close()
Opening a database
package main

import (
  "log"
  "github.com/fobilow/gitdb/v2"
)

func main() {
  
  cfg := gitdb.NewConfig("/tmp/data")
  // Open will create or clone down a git repo 
  // in configured path if it does not exist.
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  ...
}
Models

A Model is a struct that represents a record in GitDB. GitDB only works with models that implement the gidb.Model interface

gitdb.TimeStampedModel is a simple struct that allows you to easily add CreatedAt and UpdatedAt to all the Models in your application and will automatically time stamp them before persisting to GitDB. You can write your own base Models to embed common fields across your application Models

type BankAccount struct {
  //TimeStampedModel will add CreatedAt and UpdatedAt fields this Model
  gitdb.TimeStampedModel 
  AccountType         string
  AccountNo           string
  Currency            string
  Name                string
}

func (b *BankAccount) GetSchema() *gitdb.Schema {
  //Dataset Name
  name := "Accounts"
  //Block ID
  block := b.CreatedAt.Format("200601")
  //Record ID
  record := b.AccountNo

  //Indexes speed up searching
  indexes := make(map[string]interface{})
  indexes["AccountType"] = b.AccountType
 
  return gitdb.NewSchema(name, block, record, indexes)
}

func (b *BankAccount) Validate() error            { return nil }
func (b *BankAccount) IsLockable() bool           { return false }
func (b *BankAccount) ShouldEncrypt() bool        { return false }
func (b *BankAccount) GetLockFileNames() []string { return []string{} }

...
Inserting/Updating a record
package main

import (
  "log"
  "github.com/fobilow/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  //populate model
  account := &BankAccount()
  account.AccountNo = "0123456789"
  account.AccountType = "Savings"
  account.Currency = "GBP"
  account.Name = "Foo Bar"

  err = db.Insert(account)
  if err != nil {
    log.Println(err)
  }

  //get account id
  log.Println(gitdb.Id(account))

  //update account name
  account.Name = "Bar Foo"
  err = db.Insert(account)
  if err != nil {
    log.Println(err)
  }
}
Fetching a single record
package main
import (
  "log"
  "github.com/fobilow/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  //model to passed to Get to store result 
  var account BankAccount()
  err = db.Get("Accounts/202003/0123456789", &account)
  if err != nil {
    log.Println(err)
  }
}
Fetching all records in a dataset
package main

import (
  "fmt"
  "log"
  "github.com/fobilow/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  records, err := db.Fetch("Accounts")
  if err != nil {
    log.Print(err)
    return
  }

  accounts := []*BankAccount{}
  for _, r := range records {
    b := &BankAccount{}
    r.Hydrate(b)
    accounts = append(accounts, b)
    log.Print(fmt.Sprintf("%s-%s", gitdb.ID(b), b.AccountNo))
  }
}
Deleting a record
package main

import (
  "log"
  "github.com/fobilow/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  err := db.Delete("Accounts/202003/0123456789")
  if err != nil {
    log.Print(err)
  }
}
Search for records
package main

import (
  "fmt"
  "log"
  "github.com/fobilow/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  //Find all records that have savings account type
  searchParam := &db.SearchParam{Index: "AccountType", Value: "Savings"}
  records, err := dbconn.Search("Accounts", []*db.SearchParam{searchParam}, gitdb.SearchEquals)
  if err != nil {
    log.Println(err.Error())
    return
  } 

  accounts := []*BankAccount{}
  for _, r := range records {
    b := &BankAccount{}
    r.Hydrate(b)
    accounts = append(accounts, b)
    log.Print(fmt.Sprintf("%s-%s", b.ID, b.CreatedAt))
  }
}
Transactions
package main

import (
  "log"
  "github.com/fobilow/gitdb/v2"
)

func main() {
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  func accountUpgradeFuncOne() error { println("accountUpgradeFuncOne..."); return nil }
  func accountUpgradeFuncTwo() error { println("accountUpgradeFuncTwo..."); return errors.New("accountUpgradeFuncTwo failed") }
  func accountUpgradeFuncThree() error { println("accountUpgradeFuncThree"); return nil }

  tx := db.StartTransaction("AccountUpgrade")
  tx.AddOperation(accountUpgradeFuncOne)
  tx.AddOperation(accountUpgradeFuncTwo)
  tx.AddOperation(accountUpgradeFuncThree)
  terr := tx.Commit()
  if terr != nil {
    log.Print(terr)
  }
}
Encryption

GitDB suppports AES encryption and is done on a Model level, which means you can have a database with different Models where some are encrypted and others are not. To encrypt your data, your Model must implement ShouldEncrypt() to return true and you must set gitdb.Config.EncryptionKey. For maximum security set this key to a 32 byte string to select AES-256

package main

import (
  "log"
  "github.com/fobilow/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  cfg.EncryptionKey = "a_32_bytes_string_for_AES-256"
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  //populate model
  account := &BankAccount()
  account.AccountNo = "0123456789"
  account.AccountType = "Savings"
  account.Currency = "GBP"
  account.Name = "Foo Bar"

  //Insert will encrypt the account
  err = db.Insert(account)
  if err != nil {
    log.Println(err)
  }

  //Get will automatically decrypt account
  var account BankAccount()
  err = db.Get("Accounts/202003/0123456789", &account)
  if err != nil {
    log.Println(err)
  }
}

Resources

For more information on getting started with Gitdb, check out the following articles:

Caveats & Limitations

It's important to pick the right tool for the job and GitDB is no exception. Here are a few things to note when evaluating and using GitDB:

  • GitDB is good for systems where data producers are indpendent.
  • GitDB currently depends on the git binary to work

Reading the Source

GitDB is a relatively small code base (<5KLOC) for an embedded, distributed, document database so it can be a good starting point for people interested in how databases work.

The best places to start are the main entry points into GitDB:

  • Open() - Initializes the reference to the database. It's responsible for creating the database if it doesn't exist and pulling down existing database if an online remote is specified.

If you have additional notes that could be helpful for others, please submit them via pull request.

Documentation

Index

Constants

View Source 
const RecVersion = "v2"

RecVersion of gitdb

Variables

This section is empty.

Functions

func AutoBlock 

func AutoBlock(dbPath string, m Model, method BlockMethod, n int64) string

AutoBlock automatically generates block id for a given Model depending on a BlockMethod

func ID 

func ID(m Model) string

ID returns the id of a given Model

func Indexes 

func Indexes(m Model) map[string]interface{}

Indexes returns the index map of a given Model

func ParseID 

func ParseID(id string) (dataset string, block string, record string, err error)

ParseID parses a record id and returns it's metadata

func SetLogLevel 

func SetLogLevel(l LogLevel)

SetLogLevel sets log level

func SetLogger 

func SetLogger(l *golog.Logger)

SetLogger sets Logger

Types

type BlockMethod 

type BlockMethod string

BlockMethod type of method to use with AutoBlock 

var (
	//BlockBySize generates a new block when current block has reached a specified size
	BlockBySize BlockMethod = "size"
	//BlockByCount generates a new block when the number of records has reached a specified count
	BlockByCount BlockMethod = "count"
)

type Config 

type Config struct {
	ConnectionName string
	DbPath         string
	OnlineRemote   string
	EncryptionKey  string
	SyncInterval   time.Duration
	User           *User
	Factory        func(string) Model
	UIPort         int
}

Config represents configuration options for GitDB

func NewConfig 

func NewConfig(dbPath string) *Config

NewConfig constructs a *Config

func (*Config) Validate 

func (c *Config) Validate() error

Validate returns an error is *Config.DbPath is not set

type GUI added in v2.0.1 

type GUI interface {
	Embed(name, src string)
	// contains filtered or unexported methods
}

GUI interface

func UI added in v2.0.1 

func UI() GUI

UI provides an API to run UI server from outside this package

type GitDb 

type GitDb interface {
	Close() error
	Insert(m Model) error
	InsertMany(m []Model) error
	Get(id string, m Model) error
	Exists(id string) error
	Fetch(dataset string) ([]*record, error)
	Search(dataDir string, searchParams []*SearchParam, searchMode SearchMode) ([]*record, error)
	Delete(id string) error
	DeleteOrFail(id string) error
	Lock(m Model) error
	Unlock(m Model) error
	Migrate(from Model, to Model) error
	GetMails() []*mail
	StartTransaction(name string) *transaction
	GetLastCommitTime() (time.Time, error)
	SetUser(user *User) error
	Config() Config
}

GitDb interface defines all export funcs an implementation must have

func Conn 

func Conn() GitDb

Conn returns the last connection started by Open(*Config) if you opened more than one connection use GetConn(name) instead

func GetConn 

func GetConn(name string) GitDb

GetConn returns a specific gitdb connection by name

func Open 

func Open(config *Config) (GitDb, error)

Open opens a connection to GitDB

type LogLevel 

type LogLevel int

LogLevel is used to set verbosity of GitDB 

const (
	//LogLevelNone - log nothing
	LogLevelNone LogLevel = 0
	//LogLevelError - logs only errors
	LogLevelError LogLevel = 1
	//LogLevelWarning  - logs warning and errors
	LogLevelWarning LogLevel = 2
	//LogLevelTest - logs only debug messages
	LogLevelTest LogLevel = 3
	//LogLevelInfo - logs info, warining and errors
	LogLevelInfo LogLevel = 4
)

type Model 

type Model interface {
	GetSchema() *Schema
	//Validate validates a Model
	Validate() error
	//IsLockable informs GitDb if a Model support locking
	IsLockable() bool
	//GetLockFileNames informs GitDb of files a Models using for locking
	GetLockFileNames() []string
	//ShouldEncrypt informs GitDb if a Model support encryption
	ShouldEncrypt() bool
	//BeforeInsert is called by gitdb before insert
	BeforeInsert() error
}

Model interface describes methods GitDB supports

type Schema 

type Schema struct {
	// contains filtered or unexported fields
}

Schema holds functions for generating a model id

func NewSchema 

func NewSchema(name, block, record string, indexes map[string]interface{}) *Schema

NewSchema constructs a *Schema

func (*Schema) Validate 

func (a *Schema) Validate() error

Validate ensures *Schema is valid

type SearchMode 

type SearchMode int

SearchMode defines how gitdb should search with SearchParam 

const (
	//SearchEquals will search index for records whose values equal SearchParam.Value
	SearchEquals SearchMode = 1
	//SearchContains will search index for records whose values contain SearchParam.Value
	SearchContains SearchMode = 2
	//SearchStartsWith will search index for records whose values start with SearchParam.Value
	SearchStartsWith SearchMode = 3
	//SearchEndsWith will search index for records whose values ends with SearchParam.Value
	SearchEndsWith SearchMode = 4
)

type SearchParam 

type SearchParam struct {
	Index string
	Value string
}

SearchParam represents search parameters against GitDB index

type TimeStampedModel 

type TimeStampedModel struct {
	CreatedAt time.Time
	UpdatedAt time.Time
}

TimeStampedModel provides time stamp fields

func (*TimeStampedModel) BeforeInsert added in v2.0.1 

func (m *TimeStampedModel) BeforeInsert() error

BeforeInsert implements Model.BeforeInsert

type User added in v2.0.1 

type User struct {
	Name  string
	Email string
}

User represents the user currently connected to the database and will be used to identify who made changes to it

func NewUser 

func NewUser(name string, email string) *User

NewUser constructs a *DbUser

func (*User) AuthorName added in v2.0.1 

func (u *User) AuthorName() string

AuthorName return commit author git style

func (*User) String added in v2.0.1 

func (u *User) String() string

String is an alias for AuthorName

Source Files

View all Source files

Directories

Path Synopsis
cmd
gitdb
booking

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.