melopepondb

package module
v1.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 6, 2022 License: MIT Imports: 16 Imported by: 0

README

MelopeponDB

release-1-1-4 continuous-integration-testing golang-1-19 go-report-card

A small pure Golang database package that uses Gob, Json, or other backends to serialize data to and from disk.

Install
go get github.com/benduckwdesign/melopepondb

Features

  • Extendable -- Easy to edit code and add custom backends
  • Small database -- Smaller than most database setups
  • Simple -- Easy to understand and use
  • Standardized -- Uses standard Go encoding packages
  • Tables -- Supports tables, although entirely optional
  • MIT license
v1.1.4
  • Finishing touches to SGobDB backend
  • New SJsonDB backend
  • Old Gob and Json backends deprecated
  • Can now set number of backups to keep (zero to disable, five is default)
  • Can set store mode to per table to use folders for tables
  • Safeguards for valid file path if database is not in memory only
  • Ability to "batch" transactions (disables updating / saving to disk until transactions complete)
  • Loose key validations for table names and key names when querying
  • Ability to set maximum number of goroutines used by MelopeponDB
  • Better and more standardized error returns for all functions
  • New method of providing options on database open (NewSGobOptions())
  • New method for fetching list of tablenames AllTables
v1.1.3
  • Added SuperGob (SGobDB) backend, a Gob backend which is goroutine-safe and free of data race conditions (can be used concurrently)
  • Internally caches as any type instead of []byte, so it is comparatively as fast as the newer Json backend (slower for lookups, faster for updates)
  • Not production ready, as it does not have good error handling, rollbacks in the case of read failure, or documentation, however, it passes testing so it is functional
  • Eventually, these changes will be brought over to Json backend once SGobDB reaches feature parity, and old Gob backend will be deprecated
  • Fixed error message typo in gob.go
v1.0.8
  • API change: renamed DBFilter to DBQuery for clarity
  • Added Count() method to return total number of keys in database
  • Ability to disable debug messages for security
  • Fixed potential debug message error when Get pointer is nil
  • Added performance benchmarks (now part of GitHub workflow)
  • Additional retries when LoadFromDisk fails due to busy IO
  • More consistent debug messages
v1.0.7
  • Locking for IO operations and tempfile when saving
  • Rollback from backup if database doesn't exist
  • Concurrent synchronous saving and loading using goroutines
v1.0.6
  • JSON backend

Roadmap

  • Rework of JsonDB with concurrency improvements from SGobDB
  • Replacement of GobDB with SGobDB

These features are not guaranteed to be implemented, but they are certainly welcome:

  • Option to pay serialization cost upfront for InMem databases
  • Ability to toggle FileSync on or off independently of InMem
  • Fetch from rollback if file is unreadable or undecodable
  • Periodic backup scheduling
  • Encryption
  • Better key indexing (worse performance for in-memory databases, see better-index-dev branch)
  • Relations with lazy resolvable queries
  • Copy / duplicate one in-memory database to another

Features for out-of-memory database manipulation:

  • Split database map into subfiles so that InMem: false will read from and save to disk
  • Implement caching behavior for InMem false to keep the last X read values in memory
  • "Transaction Locking" when FileSync is on (or batched queries before file sync is triggered)
  • Automatic backup recovery from expanded subfiles to single file or vice versa
  • Ability to swap storage modes on close or open (ability to recompact or expand data)
  • Ability to rebuild index on command / Automatic backup recovery for index if missing
  • Ability to choose file per database, per table, or per item in certain scenarios (storage mode)

How to use?

Using MelopeponDB is simple. Here is a basic example for the Gob backend. To use the JSON backend, simply call OpenJsonDB instead. Multiple backend types can be used at the same time within the same project.

Warning

Do not use the same labels for different databases with differing backends. Additionally, it is recommended to only use ASCII for database keys. (This is due to how key filtering is performed.)


package main

import (
    melo "github.com/benduckwdesign/melopepondb"
)

var pdb *melo.DBQuery = melo.Query().DB("DatabaseOfPeople").Table("family")

type familyPerson struct {
    Name string
    Age *int
}

func main() {

	defer melo.CloseDB("DatabaseOfPeople")

	melo.OpenSGobDB("DatabaseOfPeople", "people_db", nil)

	grandma := familyPerson{
		Name: "Ruby",
		Age:  nil,
	}

	pdb.Update("grandma", &grandma)

	var grandpa familyPerson

	pdb.Get("grandma", &grandpa)

	grandpa.Name = "Johnny"

	pdb.Update("grandpa", &grandpa)

}

License

MelopeponDB is licensed under the MIT License.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ByteToString added in v1.1.4 

func ByteToString(b []byte) string

Converts a byte slice to string with zero allocation.

func Close 

func Close()

Alias for CloseAll().

func CloseAll 

func CloseAll() *[]error

Closes all databases.

func CloseDB 

func CloseDB(name string) error

Closes the database with the given name.

func CreateTempFile added in v1.0.7 

func CreateTempFile(name string) (string, *os.File, error)

func DisableDebugMessages added in v1.0.8 

func DisableDebugMessages()

This will disable all debug messages for security (information will not be leaked.) Due to its secure nature, debug messages cannot be re-enabled unless the application is restarted and this function is not called.

func DoesExist added in v1.0.7 

func DoesExist(dest string) bool

Checks if a file does exist.

func DoesNotExist 

func DoesNotExist(dest string) bool

Checks if a file does not exist.

func KeynameValidator added in v1.1.4 

func KeynameValidator(keyname *string) (bool, error)

Returns true if the keyname is valid and false if it is not. Because tablenames are just key prefixes, this can also be used to validate tablenames.

func OpenSGobDB added in v1.0.9 

func OpenSGobDB(label string, loc string, opts *SGobOptions) error

Opens a database with the Super Gob backend. label : Nickname for the database, only used for queries. loc : Filepath where database will be stored. opts : Use NewSGobOptions() to set available options for this backend type. The database is only rotated on close or when calling SaveDB directly to prevent excessive IO.

func OpenSJsonDB added in v1.1.4 

func OpenSJsonDB(label string, loc string, opts *SJsonOptions) error

Opens a database with the Super Gob backend. label : Nickname for the database, only used for queries. loc : Filepath where database will be stored. opts : Use NewSJsonOptions() to set available options for this backend type. The database is only rotated on close or when calling SaveDB directly to prevent excessive IO.

func SGobOptionBackupsToKeep added in v1.1.4 

func SGobOptionBackupsToKeep(num uint32) func(opts *SGobOptions) error

Sets how many backups to keep. Default is five. Set to zero to disable.

func SGobOptionCacheAfterGet added in v1.1.4 

func SGobOptionCacheAfterGet() func(opts *SGobOptions) error

Will save values in cache after they are fetched from disk. Don't use this if you have a database that is too large to fit in memory.

func SGobOptionInMemoryOnly added in v1.1.4 

func SGobOptionInMemoryOnly() func(opts *SGobOptions) error

Will not persist to disk even if location is set. Keeps all values cached in memory.

func SGobOptionStoreModePerItem added in v1.1.4 

func SGobOptionStoreModePerItem() func(opts *SGobOptions) error

Database stores invididual items as their own "files".

func SGobOptionStoreModePerTable added in v1.1.4 

func SGobOptionStoreModePerTable() func(opts *SGobOptions) error

Database stores tables as their own "files".

func SGobOptionStoreModeSingleFile added in v1.1.4 

func SGobOptionStoreModeSingleFile() func(opts *SGobOptions) error

Database stores itself as a single file. Currently not supported.

func SGobOptionSyncAfterWrite added in v1.1.4 

func SGobOptionSyncAfterWrite() func(opts *SGobOptions) error

Will trigger a save to disk after every Update or Delete.

func SGobOptionUncacheAfterSave added in v1.1.4 

func SGobOptionUncacheAfterSave() func(opts *SGobOptions) error

Will remove values from cache after they are persisted to disk so they must be fetched from disk again.

func SJsonOptionBackupsToKeep added in v1.1.4 

func SJsonOptionBackupsToKeep(num uint32) func(opts *SJsonOptions) error

Sets how many backups to keep. Default is five. Set to zero to disable.

func SJsonOptionCacheAfterGet added in v1.1.4 

func SJsonOptionCacheAfterGet() func(opts *SJsonOptions) error

Will save values in cache after they are fetched from disk. Don't use this if you have a database that is too large to fit in memory.

func SJsonOptionInMemoryOnly added in v1.1.4 

func SJsonOptionInMemoryOnly() func(opts *SJsonOptions) error

Will not persist to disk even if location is set. Keeps all values cached in memory.

func SJsonOptionStoreModePerItem added in v1.1.4 

func SJsonOptionStoreModePerItem() func(opts *SJsonOptions) error

Database stores invididual items as their own "files".

func SJsonOptionStoreModePerTable added in v1.1.4 

func SJsonOptionStoreModePerTable() func(opts *SJsonOptions) error

Database stores tables as their own "files".

func SJsonOptionStoreModeSingleFile added in v1.1.4 

func SJsonOptionStoreModeSingleFile() func(opts *SGobOptions) error

Database stores itself as a single file. Currently not supported.

func SJsonOptionSyncAfterWrite added in v1.1.4 

func SJsonOptionSyncAfterWrite() func(opts *SJsonOptions) error

Will trigger a save to disk after every Update or Delete.

func SJsonOptionUncacheAfterSave added in v1.1.4 

func SJsonOptionUncacheAfterSave() func(opts *SJsonOptions) error

Will remove values from cache after they are persisted to disk so they must be fetched from disk again.

func SaveDB added in v1.0.9 

func SaveDB(name string) error

Forces a database to save to disk.

func SetActiveWorkersLimit added in v1.1.4 

func SetActiveWorkersLimit(limit int64)

Limits the number of goroutines used by IO calls for all databases. Default is 2000. Set this to whatever your system, filesystem, or disk can handle smoothly.

func Wait added in v1.1.4 

func Wait()

Waits until all database operations have completed.

Types

type DBQuery added in v1.0.8 

type DBQuery struct {
	GetDBName    *string
	GetTableName *string
	// contains filtered or unexported fields
}

A database query.

func Query 

func Query() *DBQuery

Creates a new database query.

func (*DBQuery) Count added in v1.0.8 

func (f *DBQuery) Count() (int64, error)

Returns how many keys are in the entire database.

func (*DBQuery) DB added in v1.0.8 

func (f *DBQuery) DB(db_name string) *DBQuery

Chooses a specific database for the query.

func (*DBQuery) Delete added in v1.0.8 

func (f *DBQuery) Delete(key string) error

Deletes a key-value pair from the database.

func (*DBQuery) Get added in v1.0.8 

func (f *DBQuery) Get(key string, v any) error

Fetches a value for the key. v is a pointer to the object the value will be set to. v will be nil if the value cannot be found or is nil. v will be decoded internally from a byte slice.

func (*DBQuery) GetKeys added in v1.0.8 

func (f *DBQuery) GetKeys() (*[]string, error)

Fetches a list of keys from the database. If a table is provided in the query, only keys for the table will be returned. The resulting keys will have the tablename stripped.

func (*DBQuery) GetModificationTime added in v1.1.4 

func (f *DBQuery) GetModificationTime(key string) (int64, error)

Returns modification time of the item in microseconds.

func (DBQuery) GetSGobDBInstance added in v1.0.9 

func (f DBQuery) GetSGobDBInstance() *SGobDatabase

Fetches an SGobDatabase instance from its label if there is one currently open. Returns nil if the database is closed or no such database exists.

func (DBQuery) GetSJsonDBInstance added in v1.1.4 

func (f DBQuery) GetSJsonDBInstance() *SJsonDatabase

Fetches an SJsonDatabase instance from its label if there is one currently open. Returns nil if the database is closed or no such database exists.

func (DBQuery) ResolveKeyname added in v1.0.8 

func (f DBQuery) ResolveKeyname(key *string) *string

Resolves a query to a fullname key.

func (*DBQuery) Table added in v1.0.8 

func (f *DBQuery) Table(table_name string) *DBQuery

Chooses a specific table for the query.

func (*DBQuery) TransactionEnd added in v1.1.4 

func (f *DBQuery) TransactionEnd(rotate bool) error

rotate: Specifies whether to rotate backups (true) or not (false). For SGob databases, resumes syncing writes after TransactionStart was called. Will cause a save to disk operation.

func (*DBQuery) TransactionStart added in v1.1.4 

func (f *DBQuery) TransactionStart() error

For SGob databases, pauses syncing writes until TransactionEnd is called. TransactionEnd will cause a save to disk operation.

func (*DBQuery) Update added in v1.0.8 

func (f *DBQuery) Update(key string, v any) error

Updates a value for a key. v can be the object itself or a pointer to the object that the value will be read from. v will be encoded internally to a byte slice when using gob backend.

type SGobDatabase added in v1.0.9 

type SGobDatabase struct {
	Label           string
	Location        string
	DBName          string
	LastSaved       int64
	Mem             *map[string]*SGobItem
	MutexMem        *map[string]*sync.RWMutex
	MutexAllocGuard *sync.RWMutex
	KeyCount        *atomic.Int64
	// ActiveOperations *atomic.Int64
	Closing         *atomic.Bool
	Opening         *atomic.Bool
	Saving          *atomic.Bool
	SyncWritePaused *atomic.Bool
	Options         *SGobOptions
}

A database with the SGob backend.

func GetSGobDBInstance added in v1.0.9 

func GetSGobDBInstance(label string) *SGobDatabase

Fetches an SGobDatabase instance from its label if there is one currently open. Returns nil if the database is closed or no such database exists.

func (*SGobDatabase) AllKeys added in v1.1.4 

func (d *SGobDatabase) AllKeys() (*[]string, error)

Fetches a list of all keys for the database. Will include table prefixes.

func (*SGobDatabase) AllTables added in v1.1.4 

func (d *SGobDatabase) AllTables() (*[]string, error)

Fetches a list of all table key prefixes for the database. Will not include specific fully named keys.

func (*SGobDatabase) Close added in v1.0.9 

func (d *SGobDatabase) Close() error

Closes the database.

func (*SGobDatabase) Count added in v1.0.9 

func (d *SGobDatabase) Count() (int64, error)

func (*SGobDatabase) Delete added in v1.0.9 

func (d *SGobDatabase) Delete(Key *string) error

func (*SGobDatabase) FilterKeys added in v1.1.4 

func (d *SGobDatabase) FilterKeys(prefix string) (*[]string, error)

Fetches a list of keys for the database. Will not include table prefix in key for filter matches.

func (*SGobDatabase) Get added in v1.0.9 

func (d *SGobDatabase) Get(Key *string, v any) error

func (*SGobDatabase) GetModificationTime added in v1.1.4 

func (d *SGobDatabase) GetModificationTime(Key *string) (int64, error)

Returns modification time of the item in microseconds.

func (*SGobDatabase) LoadFromDisk added in v1.0.9 

func (d *SGobDatabase) LoadFromDisk() error

Loads the database from its disk location.

func (*SGobDatabase) SaveToDisk added in v1.0.9 

func (d *SGobDatabase) SaveToDisk(rotate bool) error

Saves the database to disk.

func (*SGobDatabase) TransactionEnd added in v1.1.4 

func (d *SGobDatabase) TransactionEnd(rotate bool) error

rotate: Specifies whether to rotate previous database backups (true) or not (false). Re-enables disk persistence and triggers a save to disk operation. Useful for batch operations.

func (*SGobDatabase) TransactionStart added in v1.1.4 

func (d *SGobDatabase) TransactionStart() error

Disables disk persistence temporarily. Useful for batch operations.

func (*SGobDatabase) Update added in v1.0.9 

func (d *SGobDatabase) Update(Key *string, Data any) error

type SGobIndex added in v1.0.9 

type SGobIndex struct {
	Keys []string
}

An index of keys for the SGob backend.

type SGobItem added in v1.0.9 

type SGobItem struct {
	Cache        any
	Deleted      bool
	LastModified int64
}

A database item for the SGob backend.

type SGobOptions added in v1.1.4 

type SGobOptions struct {
	InMemoryOnly        bool
	SyncAfterWrite      bool
	UncacheAfterSave    bool
	CacheAfterGet       bool
	StoreModePerItem    bool
	StoreModeSingleFile bool
	StoreModePerTable   bool
	BackupsToKeep       uint32
}

A struct with options for an SGob database.

func NewSGobOptions added in v1.1.4 

func NewSGobOptions(options ...func(*SGobOptions) error) *SGobOptions

Creates a new SGobOptions struct. Chain .Option methods to set options. Will panic if an option has somehow been misconfigured.

type SJsonDatabase added in v1.1.4 

type SJsonDatabase struct {
	Label           string
	Location        string
	DBName          string
	LastSaved       int64
	Mem             *map[string]*SJsonItem
	MutexMem        *map[string]*sync.RWMutex
	MutexAllocGuard *sync.RWMutex
	KeyCount        *atomic.Int64
	// ActiveOperations *atomic.Int64
	Closing         *atomic.Bool
	Opening         *atomic.Bool
	Saving          *atomic.Bool
	SyncWritePaused *atomic.Bool
	Options         *SJsonOptions
}

A database with the SJson backend.

func GetSJsonDBInstance added in v1.1.4 

func GetSJsonDBInstance(label string) *SJsonDatabase

Fetches an SJsonDatabase instance from its label if there is one currently open. Returns nil if the database is closed or no such database exists.

func (*SJsonDatabase) AllKeys added in v1.1.4 

func (d *SJsonDatabase) AllKeys() (*[]string, error)

Fetches a list of all keys for the database. Will include table prefixes.

func (*SJsonDatabase) AllTables added in v1.1.4 

func (d *SJsonDatabase) AllTables() (*[]string, error)

Fetches a list of all table key prefixes for the database. Will not include specific fully named keys.

func (*SJsonDatabase) Close added in v1.1.4 

func (d *SJsonDatabase) Close() error

Closes the database.

func (*SJsonDatabase) Count added in v1.1.4 

func (d *SJsonDatabase) Count() (int64, error)

func (*SJsonDatabase) Delete added in v1.1.4 

func (d *SJsonDatabase) Delete(Key *string) error

func (*SJsonDatabase) FilterKeys added in v1.1.4 

func (d *SJsonDatabase) FilterKeys(prefix string) (*[]string, error)

Fetches a list of keys for the database. Will not include table prefix in key for filter matches.

func (*SJsonDatabase) Get added in v1.1.4 

func (d *SJsonDatabase) Get(Key *string, v any) error

func (*SJsonDatabase) GetModificationTime added in v1.1.4 

func (d *SJsonDatabase) GetModificationTime(Key *string) (int64, error)

Returns modification time of the item in microseconds.

func (*SJsonDatabase) LoadFromDisk added in v1.1.4 

func (d *SJsonDatabase) LoadFromDisk() error

Loads the database from its disk location.

func (*SJsonDatabase) SaveToDisk added in v1.1.4 

func (d *SJsonDatabase) SaveToDisk(rotate bool) error

Saves the database to disk.

func (*SJsonDatabase) TransactionEnd added in v1.1.4 

func (d *SJsonDatabase) TransactionEnd(rotate bool) error

rotate: Specifies whether to rotate previous database backups (true) or not (false). Re-enables disk persistence and triggers a save to disk operation. Useful for batch operations.

func (*SJsonDatabase) TransactionStart added in v1.1.4 

func (d *SJsonDatabase) TransactionStart() error

Disables disk persistence temporarily. Useful for batch operations.

func (*SJsonDatabase) Update added in v1.1.4 

func (d *SJsonDatabase) Update(Key *string, Data any) error

type SJsonIndex added in v1.1.4 

type SJsonIndex struct {
	Keys []string
}

An index of keys for the SJson backend.

type SJsonItem added in v1.1.4 

type SJsonItem struct {
	Cache        any
	Deleted      bool
	LastModified int64
}

A database item for the SJson backend.

type SJsonOptions added in v1.1.4 

type SJsonOptions struct {
	InMemoryOnly        bool
	SyncAfterWrite      bool
	UncacheAfterSave    bool
	CacheAfterGet       bool
	StoreModePerItem    bool
	StoreModeSingleFile bool
	StoreModePerTable   bool
	BackupsToKeep       uint32
}

A struct with options for an SJson database.

func NewSJsonOptions added in v1.1.4 

func NewSJsonOptions(options ...func(*SJsonOptions) error) *SJsonOptions

Creates a new SJsonOptions struct. Chain .Option methods to set options. Will panic if an option has somehow been misconfigured.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.