gitdb

package module
v2.0.0-...-d8030fd Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 22, 2022 License: MIT Imports: 33 Imported by: 0

README

GitDB

Go Report Card Coverage Build Status Travis Godoc Releases LICENSE

What is GitDB?

GitDB is not a binary. It’s a library!

GitDB is a decentralized document database written in Go that uses Git under the hood to provide database-like functionalities via strictly defined interfaces.

GitDB allows developers to create Models of objects in their application which implement a Model Interface that can access it's persistence features. This allows GitDB to work with these objects in database operations.

Why GitDB - motivation behind project?

  • A need for a database that was quick and simple to set up
  • A need for a database that was decentralized and each participating client in a system can store their data independent of other clients.

Features

  • Decentralized
  • Document store
  • Embedded into your go application
  • Encryption (encrypt on write, decrypt on read)
  • Record locking.
  • Simple Indexing System
  • Transactions
  • Web UI

Project versioning

GitDB uses semantic versioning. API should not change between patch and minor releases. New minor versions may add additional features to the API.

Table of Contents

Getting Started

Installing

To start using GitDB, install Go and run go get:

$ go get github.com/gogitdb/gitdb/v2
Configuration

Below are configuration options provided by GitDB

Name Description Type Required Default
DbPath Path on your machine where you want GitDB to create/clone your database string Y N/A
ConnectionName Unique name for gitdb connection. Use this when opening multiple GitDB connections string N "default"
OnlineRemote URL for remote git server you want GitDB to sync with e.g git@github.com:user/db.git or https://github.com/user/db.git.

Note: The first time GitDB runs, it will automatically generate ssh keys and will automatically attempt to use this key to sync with the OnlineRemote, therefore ensure that the generated keys are added to this git server. The ssh keys can be found at Config.DbPath/.gitdb/ssh

 string N ""
SyncInterval This controls how often you want GitDB to sync with the online remote time.Duration. N 5s
EncryptionKey 16,24 or 32 byte string used to provide AES encryption for Models that implement ShouldEncrypt string N ""
User This specifies the user connected to the Gitdb and will be used to commit all changes to the database gitdb.User N ghost <ghost@gitdb.local>
EnableUI Use this option to enable GitDB web user interface bool N false
UIPort Use this option to change the default port which GitDB uses to serve it's web user interface int N 4120
Factory For backward compatibity with v1. In v1 GitDB needed a factory method to be able construct concrete Model for certain database operations. This has now been dropped in v2 func(dataset string) gitdb.Model N nil
Mock Flag used for testing apps. If true, will return a mock GitDB connection bool N false

You can configure GitDB either using the constructor or constructing it yourself

cfg := gitdb.NewConfig(path)
//or
cfg := gitdb.Config{
  DbPath: path
}
Importing GitDB

To use GitDB as an embedded document store, import as:

import "github.com/gogitdb/gitdb/v2"

cfg := gitdb.NewConfig(path)
db, err := gitdb.Open(cfg)
if err != nil {
  log.Fatal(err)
}
defer db.Close()
Opening a database
package main

import (
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main() {
  
  cfg := gitdb.NewConfig("/tmp/data")
  // Open will create or clone down a git repo 
  // in configured path if it does not exist.
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  ...
}
Models

A Model is a struct that represents a record in GitDB. GitDB only works with models that implement the gidb.Model interface

gitdb.TimeStampedModel is a simple struct that allows you to easily add CreatedAt and UpdatedAt to all the Models in your application and will automatically time stamp them before persisting to GitDB. You can write your own base Models to embed common fields across your application Models

type BankAccount struct {
  //TimeStampedModel will add CreatedAt and UpdatedAt fields this Model
  gitdb.TimeStampedModel 
  AccountType         string
  AccountNo           string
  Currency            string
  Name                string
}

func (b *BankAccount) GetSchema() *gitdb.Schema {
  //Dataset Name
  name := "Accounts"
  //Block ID
  block := b.CreatedAt.Format("200601")
  //Record ID
  record := b.AccountNo

  //Indexes speed up searching
  indexes := make(map[string]interface{})
  indexes["AccountType"] = b.AccountType
 
  return gitdb.NewSchema(name, block, record, indexes)
}

func (b *BankAccount) Validate() error            { return nil }
func (b *BankAccount) IsLockable() bool           { return false }
func (b *BankAccount) ShouldEncrypt() bool        { return false }
func (b *BankAccount) GetLockFileNames() []string { return []string{} }

...
Inserting/Updating a record
package main

import (
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  //populate model
  account := &BankAccount{}
  account.AccountNo = "0123456789"
  account.AccountType = "Savings"
  account.Currency = "GBP"
  account.Name = "Foo Bar"

  err = db.Insert(account)
  if err != nil {
    log.Println(err)
  }

  //get account id
  log.Println(gitdb.Id(account))

  //update account name
  account.Name = "Bar Foo"
  err = db.Insert(account)
  if err != nil {
    log.Println(err)
  }
}
Fetching a single record
package main
import (
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  //model to passed to Get to store result 
  var account BankAccount
  if err := db.Get("Accounts/202003/0123456789", &account); err != nil {
    log.Println(err)
  }
}
Fetching all records in a dataset
package main

import (
  "fmt"
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  records, err := db.Fetch("Accounts")
  if err != nil {
    log.Print(err)
    return
  }

  accounts := []*BankAccount{}
  for _, r := range records {
    b := &BankAccount{}
    r.Hydrate(b)
    accounts = append(accounts, b)
    log.Print(fmt.Sprintf("%s-%s", gitdb.ID(b), b.AccountNo))
  }
}
Fetching all records from specific block in a dataset
package main

import (
  "fmt"
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  records, err := db.Fetch("Accounts", "b0", "b1")
  if err != nil {
    log.Print(err)
    return
  }

  var accounts []*BankAccount
  for _, r := range records {
    b := &BankAccount{}
    r.Hydrate(b)
    accounts = append(accounts, b)
    log.Print(fmt.Sprintf("%s-%s", gitdb.ID(b), b.AccountNo))
  }
}
Deleting a record
package main

import (
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  if err := db.Delete("Accounts/202003/0123456789"); err != nil {
    log.Print(err)
  }
}
Search for records
package main

import (
  "fmt"
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  //Find all records that have savings account type
  searchParam := &db.SearchParam{Index: "AccountType", Value: "Savings"}
  records, err := dbconn.Search("Accounts", []*db.SearchParam{searchParam}, gitdb.SearchEquals)
  if err != nil {
    log.Println(err.Error())
    return
  } 

  accounts := []*BankAccount{}
  for _, r := range records {
    b := &BankAccount{}
    r.Hydrate(b)
    accounts = append(accounts, b)
    log.Print(fmt.Sprintf("%s-%s", b.ID, b.CreatedAt))
  }
}
Transactions
package main

import (
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main() {
  cfg := gitdb.NewConfig("/tmp/data")
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  func accountUpgradeFuncOne() error { println("accountUpgradeFuncOne..."); return nil }
  func accountUpgradeFuncTwo() error { println("accountUpgradeFuncTwo..."); return errors.New("accountUpgradeFuncTwo failed") }
  func accountUpgradeFuncThree() error { println("accountUpgradeFuncThree"); return nil }

  tx := db.StartTransaction("AccountUpgrade")
  tx.AddOperation(accountUpgradeFuncOne)
  tx.AddOperation(accountUpgradeFuncTwo)
  tx.AddOperation(accountUpgradeFuncThree)
  terr := tx.Commit()
  if terr != nil {
    log.Print(terr)
  }
}
Encryption

GitDB suppports AES encryption and is done on a Model level, which means you can have a database with different Models where some are encrypted and others are not. To encrypt your data, your Model must implement ShouldEncrypt() to return true and you must set gitdb.Config.EncryptionKey. For maximum security set this key to a 32 byte string to select AES-256

package main

import (
  "log"
  "github.com/gogitdb/gitdb/v2"
)

func main(){
  cfg := gitdb.NewConfig("/tmp/data")
  cfg.EncryptionKey = "a_32_bytes_string_for_AES-256"
  db, err := gitdb.Open(cfg)
  if err != nil {
    log.Fatal(err)
  }
  defer db.Close()

  //populate model
  account := &BankAccount{}
  account.AccountNo = "0123456789"
  account.AccountType = "Savings"
  account.Currency = "GBP"
  account.Name = "Foo Bar"

  //Insert will encrypt the account
  err = db.Insert(account)
  if err != nil {
    log.Println(err)
  }

  //Get will automatically decrypt account
  var account BankAccount
  err = db.Get("Accounts/202003/0123456789", &account)
  if err != nil {
    log.Println(err)
  }
}

Resources

For more information on getting started with Gitdb, check out the following articles:

Caveats & Limitations

It's important to pick the right tool for the job and GitDB is no exception. Here are a few things to note when evaluating and using GitDB:

  • GitDB is good for systems where data producers are indpendent.
  • GitDB currently depends on the git binary to work

Reading the Source

GitDB is a relatively small code base (<5KLOC) for an embedded, distributed, document database so it can be a good starting point for people interested in how databases work.

The best places to start are the main entry points into GitDB:

  • Open() - Initializes the reference to the database. It's responsible for creating the database if it doesn't exist and pulling down existing database if an online remote is specified.

If you have additional notes that could be helpful for others, please submit them via pull request.

Documentation

Index

Constants

View Source 
const RecVersion = "v2"

RecVersion of gitdb

Variables

View Source 
var (
	ErrNoRecords       = errors.ErrNoRecords
	ErrRecordNotFound  = errors.ErrRecordNotFound
	ErrInvalidRecordID = errors.ErrInvalidRecordID
	ErrDBSyncFailed    = errors.ErrDBSyncFailed
	ErrLowBattery      = errors.ErrLowBattery
	ErrNoOnlineRemote  = errors.ErrNoOnlineRemote
	ErrAccessDenied    = errors.ErrAccessDenied
	ErrInvalidDataset  = errors.ErrInvalidDataset
)

Functions

func AutoBlock 

func AutoBlock(dbPath string, m Model, method BlockMethod, n int64) string

AutoBlock automatically generates block id for a given Model depending on a BlockMethod

func ID 

func ID(m Model) string

ID returns the id of a given Model

func Indexes 

func Indexes(m Model) map[string]interface{}

Indexes returns the index map of a given Model

func ParseID 

func ParseID(id string) (dataset string, block string, record string, err error)

ParseID parses a record id and returns it's metadata

func SetLogLevel 

func SetLogLevel(l LogLevel)

SetLogLevel sets log level

func SetLogger 

func SetLogger(l *golog.Logger)

SetLogger sets Logger

Types

type BlockMethod 

type BlockMethod string

BlockMethod type of method to use with AutoBlock 

var (
	//BlockBySize generates a new block when current block has reached a specified size
	BlockBySize BlockMethod = "size"
	//BlockByCount generates a new block when the number of records has reached a specified count
	BlockByCount BlockMethod = "count"
)

type Config 

type Config struct {
	ConnectionName string
	DBPath         string
	OnlineRemote   string
	EncryptionKey  string
	SyncInterval   time.Duration
	User           *User
	Factory        func(string) Model
	EnableUI       bool
	UIPort         int
	// Mock is a hook for testing apps. If true will return a Mock DB connection
	Mock   bool
	Driver dbDriver
}

Config represents configuration options for GitDB

func NewConfig 

func NewConfig(dbPath string) *Config

NewConfig constructs a *Config

func NewConfigWithLocalDriver 

func NewConfigWithLocalDriver(dbPath string) *Config

NewConfigWithLocalDriver constructs a *Config

func (*Config) Validate 

func (c *Config) Validate() error

Validate returns an error is *Config.DBPath is not set

type Error 

type Error struct {
	// contains filtered or unexported fields
}

func ErrorWithResolution 

func ErrorWithResolution(e error, resolution string) Error

func (Error) Error 

func (e Error) Error() string

func (Error) Resolution 

func (e Error) Resolution() string

type GitDb 

type GitDb interface {
	Close() error
	Insert(m Model) error
	InsertMany(m []Model) error
	Get(id string, m Model) error
	Exists(id string) error
	Fetch(dataset string, block ...string) ([]*db.Record, error)
	Search(dataDir string, searchParams []*SearchParam, searchMode SearchMode) ([]*db.Record, error)
	Delete(id string) error
	DeleteOrFail(id string) error
	Lock(m Model) error
	Unlock(m Model) error
	Upload() *Upload
	Migrate(from Model, to Model) error
	GetMails() []*mail
	StartTransaction(name string) Transaction
	GetLastCommitTime() (time.Time, error)
	SetUser(user *User) error
	Config() Config
	Sync() error
	RegisterModel(dataset string, m Model) bool
}

GitDb interface defines all exported funcs an implementation must have

func Conn 

func Conn() GitDb

Conn returns the last connection started by Open(*Config) if you opened more than one connection use GetConn(name) instead

func GetConn 

func GetConn(name string) GitDb

GetConn returns a specific gitdb connection by name

func Open 

func Open(config *Config) (GitDb, error)

Open opens a connection to GitDB

type LockableModel 

type LockableModel interface {
	//GetLockFileNames informs GitDb of files a Models using for locking
	GetLockFileNames() []string
}

type LogLevel 

type LogLevel int

LogLevel is used to set verbosity of GitDB 

const (
	//LogLevelNone - log nothing
	LogLevelNone LogLevel = iota
	//LogLevelError - logs only errors
	LogLevelError
	//LogLevelWarning  - logs warning and errors
	LogLevelWarning
	//LogLevelTest - logs only debug messages
	LogLevelTest
	//LogLevelInfo - logs info, warining and errors
	LogLevelInfo
)

type Model 

type Model interface {
	GetSchema() *Schema
	//Validate validates a Model
	Validate() error
	//ShouldEncrypt informs GitDb if a Model support encryption
	ShouldEncrypt() bool
	//BeforeInsert is called by gitdb before insert
	BeforeInsert() error
}

Model interface describes methods GitDB supports

type ResolvableError 

type ResolvableError interface {
	Resolution() string
}

type Schema 

type Schema struct {
	// contains filtered or unexported fields
}

Schema holds functions for generating a model id

func NewSchema 

func NewSchema(name, block, record string, indexes map[string]interface{}) *Schema

NewSchema constructs a *Schema

func (*Schema) Validate 

func (a *Schema) Validate() error

Validate ensures *Schema is valid

type SearchMode 

type SearchMode int

SearchMode defines how gitdb should search with SearchParam 

const (
	// SearchEquals will search index for records whose values equal SearchParam.Value
	SearchEquals SearchMode = 1
	// SearchContains will search index for records whose values contain SearchParam.Value
	SearchContains SearchMode = 2
	// SearchStartsWith will search index for records whose values start with SearchParam.Value
	SearchStartsWith SearchMode = 3
	// SearchEndsWith will search index for records whose values ends with SearchParam.Value
	SearchEndsWith SearchMode = 4
)

type SearchParam 

type SearchParam struct {
	Index string
	Value string
}

SearchParam represents search parameters against GitDB index

type TimeStampedModel 

type TimeStampedModel struct {
	CreatedAt time.Time
	UpdatedAt time.Time
}

TimeStampedModel provides time stamp fields

func (*TimeStampedModel) BeforeInsert 

func (m *TimeStampedModel) BeforeInsert() error

BeforeInsert implements Model.BeforeInsert

type Transaction 

type Transaction interface {
	Commit() error
	AddOperation(o operation)
}

Transaction represents a db transaction

type Upload 

type Upload struct {
	// contains filtered or unexported fields
}

Upload provides API for managing file uploads

func (*Upload) Delete 

func (u *Upload) Delete(id string) error

Delete an upload by id

func (*Upload) Get 

func (u *Upload) Get(id string, result *UploadModel) error

Get returns an upload by id

func (*Upload) New 

func (u *Upload) New(bucket, file string) error

New uploads specified file into bucket

func (*Upload) Replace 

func (u *Upload) Replace(bucket, file string) error

Replace overrides a specified file in bucket

type UploadModel 

type UploadModel struct {
	Bucket     string
	File       string
	Path       string
	UploadedBy string
	TimeStampedModel
}

UploadModel represents a file upload

func (*UploadModel) GetSchema 

func (u *UploadModel) GetSchema() *Schema

GetSchema implements Model.GetSchema

func (*UploadModel) ShouldEncrypt 

func (U *UploadModel) ShouldEncrypt() bool

func (*UploadModel) Validate 

func (u *UploadModel) Validate() error

Validate implements Model.Validate

type User 

type User struct {
	Name  string
	Email string
}

User represents the user currently connected to the database and will be used to identify who made changes to it

func NewUser 

func NewUser(name string, email string) *User

NewUser constructs a *DbUser

func (*User) AuthorName 

func (u *User) AuthorName() string

AuthorName return commit author git style

func (*User) String 

func (u *User) String() string

String is an alias for AuthorName

Source Files

View all Source files

Directories

Path Synopsis
cmd
gitdb
booking
internal
crypto
db
digital
errors

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.