pudge

package module
v1.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 4, 2021 License: MIT Imports: 14 Imported by: 1

README

Documentation Go Report Card Build Status Mentioned in Awesome Go

Table of Contents

Local Modifications

This fork of pudge improves the efficiency of deletions for databases with large numbers of keys. It also provides a compaction API call to remove deleted keys and values, and also 'orphaned' entries in the value file that are no longer used.

Description

Package pudge is a fast and simple key/value store written using Go's standard library.

It presents the following:

  • Supporting very efficient lookup, insertions and deletions
  • Performance is comparable to hash tables
  • Ability to get the data in sorted order, which enables additional operations like range scan
  • Select with limit/offset/from key, with ordering or by prefix
  • Safe for use in goroutines
  • Space efficient
  • Very short and simple codebase
  • Well tested, used in production

pudge

Usage

package main

import (
	"log"

	"github.com/recoilme/pudge"
)

func main() {
	// Close all database on exit
	defer pudge.CloseAll()

	// Set (directories will be created)
	pudge.Set("../test/test", "Hello", "World")

	// Get (lazy open db if needed)
	output := ""
	pudge.Get("../test/test", "Hello", &output)
	log.Println("Output:", output)

	ExampleSelect()
}


//ExampleSelect
func ExampleSelect() {
	cfg := &pudge.Config{
		SyncInterval: 1} // every second fsync
	db, err := pudge.Open("../test/db", cfg)
	if err != nil {
		log.Panic(err)
	}
	defer db.DeleteFile()
	type Point struct {
		X int
		Y int
	}
	for i := 100; i >= 0; i-- {
		p := &Point{X: i, Y: i}
		db.Set(i, p)
	}
	var point Point
	db.Get(8, &point)
	log.Println(point)
	// Output: {8 8}
	// Select 2 keys, from 7 in ascending order
	keys, _ := db.Keys(7, 2, 0, true)
	for _, key := range keys {
		var p Point
		db.Get(key, &p)
		log.Println(p)
	}
	// Output: {8 8}
	// Output: {9 9}
}

Cookbook

  • Store data of any type. Pudge uses Gob encoder/decoder internally. No limits on keys/values size.
pudge.Set("strings", "Hello", "World")
pudge.Set("numbers", 1, 42)

type User struct {
	Id int
	Name string
}
u := &User{Id: 1, Name: "name"}
pudge.Set("users", u.Id, u)

  • Pudge is stateless and safe for use in goroutines. You don't need to create/open files before use. Just write data to pudge, don't worry about state. web server example

  • Pudge is parallel. Readers don't block readers, but a writer - does, but by the stateless nature of pudge it's safe to use multiples files for storages.

Illustration from slowpoke (based on pudge)

  • Default store system: like memcache + file storage. Pudge uses in-memory hashmap for keys, and writes values to files (no value data stored in memory). But you may use inmemory mode for values, with custom config:
cfg = pudge.DefaultConfig()
cfg.StoreMode = 2
db, err := pudge.Open(dbPrefix+"/"+group, cfg)
...
db.Counter(key, val)

In that case, all data is stored in memory and will be stored on disk only on Close.

Example server for highload, with http api

  • You may use pudge as an engine for creating databases.

Example database

  • Don't forget to close all opened databases on shutdown/kill.
 	// Wait for interrupt signal to gracefully shutdown the server 
	quit := make(chan os.Signal)
	signal.Notify(quit, os.Interrupt, os.Kill)
	<-quit
	log.Println("Shutdown Server ...")
	if err := pudge.CloseAll(); err != nil {
		log.Println("Pudge Shutdown err:", err)
	}

example recovery function for gin framework

  • Pudge has a primitive select/query engine.
// Select 2 keys, from 7 in ascending order
   keys, _ := db.Keys(7, 2, 0, true)
// select keys from db where key>7 order by keys asc limit 2 offset 0
  • Pudge will work well on SSD or spined disks. Pudge doesn't eat memory or storage or your sandwich. No hidden compaction/rebalancing/resizing and so on tasks. No LSM Tree. No MMap. It's a very simple database with less than 500 LOC. It's good for simple social network or highload system

Disadvantages

  • No transaction system. All operations are isolated, but you don't may batching them with automatic rollback.
  • Keys function (select/query engine) may be slow. Speed of query may vary from 10ms to 1sec per million keys. Pudge don't use BTree/Skiplist or Adaptive radix tree for store keys in ordered way on every insert. Ordering operation is "lazy" and run only if needed.
  • If you need storage or database for hundreds of millions keys - take a look at Sniper or b52. They are optimized for highload (pudge - not).
  • No fsync on every insert. Most of database fsync data by the timer too
  • Deleted data don't remove from physically (but upsert will try to reuse space). You may shrink database only with backup right now
pudge.BackupAll("backup")
  • Keys automatically convert to binary and ordered with binary comparator. It's simple for use, but ordering will not work correctly for negative numbers for example
  • Author of project don't work at Google or Facebook and his name not Howard Chu or Brad Fitzpatrick. But I'm open for issue or contributions.

Motivation

Some databases very well for writing. Some of the databases very well for reading. But pudge is well balanced for both types of operations. It has small cute api, and don't have hidden graveyards. It's just hashmap where values written in files. And you may use one database for in-memory/persistent storage in a stateless stressfree way

Benchmarks

All tests here

Some tests, MacBook Pro (Retina, 13-inch, Early 2015)

Test 1

Number of keys: 1000000 Minimum key size: 16, maximum key size: 64 Minimum value size: 128, maximum value size: 512 Concurrency: 2

pogreb goleveldb bolt badgerdb pudge slowpoke pudge(mem)
1M (Put+Get), seconds 187 38 126 34 23 23 2
1M Put, ops/sec 5336 34743 8054 33539 47298 46789 439581
1M Get, ops/sec 1782423 98406 499871 220597 499172 445783 1652069
FileSize,Mb 568 357 552 487 358 358 358
Test 4

Number of keys: 10000000 Key size: 8 Value size: 16 Concurrency: 100

goleveldb badgerdb pudge
10M (Put+Get), seconds 165 120 243
10M Put, ops/sec 122933 135709 43843
10M Get, ops/sec 118722 214981 666067
FileSize,Mb 312 1370 381

Documentation

Overview

Package pudge implements a low-level key/value store in pure Go. Keys stored in memory, Value stored on disk

Usage 

package main

import (
	"log"

	"github.com/recoilme/pudge"
)

func main() {
	cfg := pudge.DefaultConfig()
	cfg.SyncInterval = 0 //disable every second fsync
	db, err := pudge.Open("../test/db", cfg)
	if err != nil {
		log.Panic(err)
	}
	defer db.DeleteFile()
	type Point struct {
		X int
		Y int
	}
	for i := 100; i >= 0; i-- {
		p := &Point{X: i, Y: i}
		db.Set(i, p)
	}
	var point Point
	db.Get(8, &point)
	log.Println(point)
	// Output: {8 8}
}

Index

Examples

Constants

This section is empty.

Variables

View Source 
var DefaultConfig = &Config{
	FileMode:     0644,
	DirMode:      0755,
	SyncInterval: 0,
	StoreMode:    0}

DefaultConfig is default config

View Source 
var (

	// ErrKeyNotFound - key not found
	ErrKeyNotFound = errors.New("Error: key not found")
)

Functions

func BackupAll 

func BackupAll(dir string) (err error)

BackupAll - backup all opened Db if dir not set it will be backup delete old backup file before run ignore all errors

func Close 

func Close(f string) error

Close - sync & close files. Return error if any.

func CloseAll 

func CloseAll() (err error)

CloseAll - close all opened Db

func Count 

func Count(f string) (int, error)

Count returns the number of items in the Db.

func Counter 

func Counter(f string, key interface{}, incr int) (int64, error)

Counter return int64 incremented on incr with lazy open

func Delete 

func Delete(f string, key interface{}) error

Delete remove key Returns error if key not found

func DeleteFile 

func DeleteFile(file string) error

DeleteFile close db and delete file

func Get 

func Get(f string, key, value interface{}) error

Get return value by key with opening if needed Return error if any.

Example 
Set("test/test", "Hello", "World")
output := ""
Get("test/test", "Hello", &output)
defer CloseAll()
fmt.Println(output)

Output:

World

func Gets added in v1.0.4 

func Gets(file string, keys []interface{}) (result [][]byte)

Gets return key/value pairs in random order result contains key and value Gets not return error if key not found If no keys found return empty result

func Has 

func Has(f string, key interface{}) (bool, error)

Has return true if key exists. Return error if any.

func KeyToBinary 

func KeyToBinary(v interface{}) ([]byte, error)

KeyToBinary return key in bytes

func Keys 

func Keys(f string, from interface{}, limit, offset int, asc bool) ([][]byte, error)

Keys return keys in ascending or descending order (false - descending,true - ascending) if limit == 0 return all keys if offset > 0 - skip offset records If from not nil - return keys after from (from not included)

func Set 

func Set(f string, key, value interface{}) error

Set store any key value to db with opening if needed

Example 
Set("test/test", "Hello", "World")
defer CloseAll()

Output:

func Sets added in v1.0.4 

func Sets(file string, pairs []interface{}) (err error)

Sets store vals and keys Use it for mass insertion every pair must contain key and value

func ValToBinary 

func ValToBinary(v interface{}) ([]byte, error)

ValToBinary return value in bytes

Types

type Cmd 

type Cmd struct {
	Seek    uint32
	Size    uint32
	KeySeek uint32
	Val     []byte
}

Cmd represent keys and vals addresses

type Config 

type Config struct {
	FileMode     int // 0644
	DirMode      int // 0755
	SyncInterval int // in seconds
	StoreMode    int // 0 - file first, 2 - memory first(with persist on close), 2 - with empty file - memory without persist
}

Config fo db Default FileMode = 0644 Default DirMode = 0755 Default SyncInterval = 0 sec, 0 - disable sync (os will sync, typically 30 sec or so) If StroreMode==2 && file == "" - pure inmemory mode

type Db 

type Db struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Db represent database

func Open 

func Open(f string, cfg *Config) (*Db, error)

Open return db object if it opened. Create new db if not exist. Read db to obj if exist. Or error if any. Default Config (if nil): &Config{FileMode: 0644, DirMode: 0755, SyncInterval: 0}

Example 
cfg := &Config{
	SyncInterval: 0} //disable every second fsync
db, err := Open("test/db", cfg)
if err != nil {
	log.Panic(err)
}
defer db.DeleteFile()
type Point struct {
	X int
	Y int
}
for i := 100; i >= 0; i-- {
	p := &Point{X: i, Y: i}
	db.Set(i, p)
}
var point Point
db.Get(8, &point)
fmt.Println(point)

Output:

{8 8}

func (*Db) Close 

func (db *Db) Close() error

Close - sync & close files. Return error if any.

func (*Db) CompactAndClose added in v1.0.4 

func (db *Db) CompactAndClose() error

func (*Db) Count 

func (db *Db) Count() (int, error)

Count returns the number of items in the Db.

func (*Db) Counter 

func (db *Db) Counter(key interface{}, incr int) (int64, error)

Counter return int64 incremented on incr

func (*Db) Delete 

func (db *Db) Delete(key interface{}) error

Delete remove key Returns error if key not found

func (*Db) DeleteFile 

func (db *Db) DeleteFile() error

DeleteFile close and delete file

func (*Db) DeleteKeys added in v1.0.4 

func (db *Db) DeleteKeys(keys []interface{}) (int, error)

DeleteKeys remove keys more efficiently than by doing so a key at a time. Returns error if any of the keys are not found

func (*Db) FileSize 

func (db *Db) FileSize() (int64, error)

FileSize returns the total size of the disk storage used by the DB.

func (*Db) Get 

func (db *Db) Get(key, value interface{}) error

Get return value by key Return error if any.

func (*Db) Has 

func (db *Db) Has(key interface{}) (bool, error)

Has return true if key exists. Return error if any.

func (*Db) Keys 

func (db *Db) Keys(from interface{}, limit, offset int, asc bool) ([][]byte, error)

Keys return keys in ascending or descending order (false - descending,true - ascending) if limit == 0 return all keys if offset > 0 - skip offset records If from not nil - return keys after from (from not included)

func (*Db) KeysByPrefix 

func (db *Db) KeysByPrefix(prefix []byte, limit, offset int, asc bool) ([][]byte, error)

KeysByPrefix return keys with prefix in ascending or descending order (false - descending,true - ascending) if limit == 0 return all keys if offset > 0 - skip offset records If from not nil - return keys after from (from not included)

func (*Db) Set 

func (db *Db) Set(key, value interface{}) error

Set store any key value to db

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.