rotom

package module
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 14, 2024 License: MIT Imports: 16 Imported by: 0

README ¶

Rotom

Go Report Card Go Reference codecov Test and coverage

English | 中文 | doc

📃Introduction

Welcome to rotom, an embedded high-performance key-value in-memory database written in Go, has many built-in data types, support for persistence and data recovery.

Currently features:

  1. Built-in data types string, map, set, list, zset, and bitmap.
  2. Second level ttl supported for each key-value pair.
  3. Based on GigaCache, which is managing GB-level data, saving 50% memory compared to stdmap, with better performance and reduced GC overhead.
  4. Internal encoding/decoding lib that more effective than protobuf.
  5. Persistent log support, and can recover database from logs.

If you want to know more technical details, check out doc.

🚚Usage

Before using, please install rotom into your project first:

go get github.com/xgzlucario/rotom

And install the gofakeit library for generating some random data:

go get github.com/brianvoe/gofakeit/v6

Run the sample program:

package main

import (
	"fmt"
	"time"

	"github.com/brianvoe/gofakeit/v6"
	"github.com/xgzlucario/rotom"
)

func main() {
	db, err := rotom.Open(rotom.DefaultOptions)
	if err != nil {
		panic(err)
	}
	defer db.Close()

	// Set
	for i := 0; i < 10000; i++ {
		phone := gofakeit.Phone()
        user := []byte(gofakeit.Username())
		// Set bytes
		db.Set(phone, user)
		// Or set with ttl
		db.SetEx(phone, user, time.Minute)
		// Or set with deadline
		db.SetTx(phone, user, time.Now().Add(time.Minute).UnixNano())
	}
    
	// Get
	key := gofakeit.Phone()
	user, ttl, ok := db.Get(key)
	// ...
}

🚀Performance

Rotom has super multi-threading performance. The following is the bench test data.

Test Environment
goos: linux
goarch: amd64
pkg: github.com/xgzlucario/GigaCache
cpu: 13th Gen Intel(R) Core(TM) i5-13600KF
Benchmark
========== Set ==========
size: 100*10000 enties
desc: key 10 bytes, value 10 bytes
cost: 1.120466957s
qps: 892467.60
50th: 990 ns
90th: 1107 ns
99th: 1724 ns

========== SetEx ==========
size: 100*10000 enties
desc: key 10 bytes, value 10 bytes, ttl 1min
cost: 1.125406833s
qps: 888551.77
50th: 986 ns
90th: 1092 ns
99th: 2110 ns

========== Get ==========
size: 100*10000 enties
desc: key 10 bytes, value 10 bytes
cost: 255.023737ms
qps: 3920845.99
50th: 212 ns
90th: 267 ns
99th: 532 ns

========== Get 8 parallel ==========
size: 100*10000 enties
desc: key 10 bytes, value 10 bytes
cost: 138.089874ms
qps: 7240728.26
50th: 173 ns
90th: 261 ns
99th: 501 ns

========== RPush ==========
size: 100*10000 enties
desc: value 10 bytes
cost: 1.033844185s
qps: 967248.36
50th: 936 ns
90th: 1011 ns
99th: 1609 ns

========== HSet ==========
size: 100*10000 enties
desc: field 10 bytes, value 10 bytes
cost: 1.182939928s
qps: 845337.51
50th: 987 ns
90th: 1117 ns
99th: 1850 ns

========== HGet ==========
size: 100*10000 enties
desc: field 10 bytes, value 10 bytes
cost: 292.57105ms
qps: 3417634.25
50th: 220 ns
90th: 306 ns
99th: 575 ns

========== BitSet ==========
size: 100*10000 enties
desc: offset uint32
cost: 916.172391ms
qps: 1091477.02
50th: 833 ns
90th: 880 ns
99th: 1219 ns

========== ZSet ==========
size: 100*10000 enties
desc: field 10 bytes, incr float64
cost: 1.321257444s
qps: 756843.75
50th: 1121 ns
90th: 1248 ns
99th: 2025 ns

Documentation ¶

Overview ¶

Package rotom provides an in-memory key-value database.

Index ¶

Constants ¶

This section is empty.

Variables ¶

View Source
var (
	ErrKeyNotFound = errors.New("key not found")

	ErrFieldNotFound = errors.New("field not found")

	ErrTypeAssert = errors.New("type assert error")

	ErrOutOfBounds = errors.New("index out of bounds")

	ErrWrongType = errors.New("wrong data type")

	ErrWrongBitValue = errors.New("wrong bit value")

	ErrUnSupportDataType = errors.New("unsupport data type")

	ErrUnknownOperationType = errors.New("unknown operation type")

	ErrNotString = errors.New("value is not string")

	ErrNotNumberic = errors.New("value is not numberic")

	ErrInvalidArgs = errors.New("invalid args")

	ErrInvalidResponse = errors.New("invalid response")

	ErrDatabaseClosed = errors.New("database closed")

	ErrUnSupportOperation = errors.New("unsupport operation")

	ErrIndexOutOfRange = errors.New("index out of range")

	ErrEmptyList = errors.New("list is empty")

	ErrDatabaseIsUsing = errors.New("database is using")

	ErrShrinkRunning = errors.New("shrink is running")
)
View Source
var (
	DefaultOptions = Options{
		DirPath:          "rotom",
		ShardCount:       1024,
		SyncPolicy:       EverySecond,
		ShrinkCronExpr:   "0 0 0/1 * * ?",
		RunSkipLoadError: true,
	}
)

Functions ¶

This section is empty.

Types ¶

type BitMap ¶

type BitMap = *structx.Bitmap

Type aliases for built-in types.

type Cmd ¶

type Cmd struct {
	// contains filtered or unexported fields
}

type DB ¶ added in v1.7.3

type DB struct {
	// contains filtered or unexported fields
}

DB represents a rotom database.

func Open ¶

func Open(options Options) (*DB, error)

Open create a new db instance by options.

func (*DB) BitAnd ¶ added in v1.7.3

func (db *DB) BitAnd(dst string, src ...string) error

BitAnd

func (*DB) BitArray ¶ added in v1.7.3

func (db *DB) BitArray(key string) ([]uint32, error)

BitArray

func (*DB) BitCount ¶ added in v1.7.3

func (db *DB) BitCount(key string) (uint64, error)

BitCount

func (*DB) BitFlip ¶ added in v1.7.3

func (db *DB) BitFlip(key string, start, end uint32) error

BitFlip

func (*DB) BitOr ¶ added in v1.7.3

func (db *DB) BitOr(dst string, src ...string) error

BitOr

func (*DB) BitSet ¶ added in v1.7.3

func (db *DB) BitSet(key string, val bool, offsets ...uint32) (int, error)

BitSet

func (*DB) BitTest ¶ added in v1.7.3

func (db *DB) BitTest(key string, offset uint32) (bool, error)

BitTest

func (*DB) BitXor ¶ added in v1.7.3

func (db *DB) BitXor(dst string, src ...string) error

BitXor

func (*DB) Close ¶ added in v1.7.3

func (db *DB) Close() error

Close the database, close all data files and release file lock.

func (*DB) GC ¶ added in v1.7.3

func (db *DB) GC()

GC triggers the garbage collection to evict expired kv datas.

func (*DB) Get ¶ added in v1.7.3

func (db *DB) Get(key string) ([]byte, int64, error)

Get

func (*DB) GetOptions ¶ added in v1.8.0

func (db *DB) GetOptions() Options

GetOptions

func (*DB) HGet ¶ added in v1.7.3

func (db *DB) HGet(key, field string) ([]byte, error)

HGet

func (*DB) HKeys ¶ added in v1.7.3

func (db *DB) HKeys(key string) ([]string, error)

HKeys

func (*DB) HLen ¶ added in v1.7.3

func (db *DB) HLen(key string) (int, error)

HLen

func (*DB) HRemove ¶ added in v1.7.3

func (db *DB) HRemove(key string, fields ...string) (n int, err error)

HRemove

func (*DB) HSet ¶ added in v1.7.3

func (db *DB) HSet(key, field string, val []byte) error

HSet

func (*DB) Incr ¶ added in v1.9.0

func (db *DB) Incr(key string, incr int64) (n int64, err error)

Incr increase number to key.

func (*DB) LIndex ¶ added in v1.7.3

func (db *DB) LIndex(key string, i int) (string, error)

LIndex

func (*DB) LLen ¶ added in v1.7.3

func (db *DB) LLen(key string) (int, error)

LLen

func (*DB) LPop ¶ added in v1.9.0

func (db *DB) LPop(key string) (string, error)

LPop

func (*DB) LPush ¶ added in v1.9.0

func (db *DB) LPush(key string, items ...string) error

LPush

func (*DB) LRange ¶ added in v1.9.0

func (db *DB) LRange(key string, start, end int, f func(string) (stop bool)) error

LRange

func (*DB) LSet ¶ added in v1.9.0

func (db *DB) LSet(key string, index int, item string) (bool, error)

LSet

func (*DB) Len ¶ added in v1.7.3

func (db *DB) Len() int

Len

func (*DB) RPop ¶ added in v1.9.0

func (db *DB) RPop(key string) (string, error)

RPop

func (*DB) RPush ¶ added in v1.9.0

func (db *DB) RPush(key string, items ...string) error

RPush

func (*DB) Remove ¶ added in v1.7.3

func (db *DB) Remove(keys ...string) (n int)

Remove

func (*DB) SAdd ¶ added in v1.7.3

func (db *DB) SAdd(key string, items ...string) (int, error)

SAdd

func (*DB) SCard ¶ added in v1.7.3

func (db *DB) SCard(key string) (int, error)

SCard

func (*DB) SDiff ¶ added in v1.7.3

func (db *DB) SDiff(dst string, src ...string) error

SDiff

func (*DB) SHas ¶ added in v1.7.3

func (db *DB) SHas(key string, items ...string) (bool, error)

SHas returns whether the given items are all in the set.

func (*DB) SInter ¶ added in v1.7.3

func (db *DB) SInter(dst string, src ...string) error

SInter

func (*DB) SMembers ¶ added in v1.7.3

func (db *DB) SMembers(key string) ([]string, error)

SMembers

func (*DB) SRemove ¶ added in v1.7.3

func (db *DB) SRemove(key string, items ...string) error

SRemove

func (*DB) SUnion ¶ added in v1.7.3

func (db *DB) SUnion(dst string, src ...string) error

SUnion

func (*DB) Scan ¶ added in v1.7.3

func (db *DB) Scan(f func([]byte, []byte, int64) bool)

Scan

func (*DB) Set ¶ added in v1.7.3

func (db *DB) Set(key string, val []byte)

Set store key-value pair.

func (*DB) SetEx ¶ added in v1.7.3

func (db *DB) SetEx(key string, val []byte, ttl time.Duration)

SetEx store key-value pair with ttl.

func (*DB) SetTTL ¶ added in v1.9.0

func (db *DB) SetTTL(key string, ts int64) bool

SetTTL set expired time of key-value.

func (*DB) SetTx ¶ added in v1.7.3

func (db *DB) SetTx(key string, val []byte, ts int64)

SetTx store key-value pair with deadline.

func (*DB) Shrink ¶ added in v1.7.3

func (db *DB) Shrink() error

Shrink rewrite db file.

func (*DB) Sync ¶ added in v1.8.0

func (db *DB) Sync() error

Sync

func (*DB) ZAdd ¶ added in v1.7.3

func (db *DB) ZAdd(zset, key string, score float64) error

ZAdd

func (*DB) ZCard ¶ added in v1.7.3

func (db *DB) ZCard(zset string) (int, error)

ZCard

func (*DB) ZGet ¶ added in v1.7.3

func (db *DB) ZGet(zset, key string) (float64, error)

ZGet

func (*DB) ZIncr ¶ added in v1.7.3

func (db *DB) ZIncr(zset, key string, incr float64) (float64, error)

ZIncr

func (*DB) ZIter ¶ added in v1.7.3

func (db *DB) ZIter(zset string, f func(string, float64) bool) error

ZIter

func (*DB) ZRemove ¶ added in v1.7.3

func (db *DB) ZRemove(zset string, key string) error

ZRemove

type List ¶

type List = *structx.List

Type aliases for built-in types.

type Map ¶

type Map = *structx.SyncMap

Type aliases for built-in types.

type Operation ¶

type Operation byte

Operation needs redo.

const (
	OpSetTx Operation = iota
	OpSetTTL
	OpRemove
	// map
	OpHSet
	OpHRemove
	// set
	OpSAdd
	OpSRemove
	OpSMerge // union, inter, diff
	// list
	OpLPush
	OpRPush
	OpLPop
	OpRPop
	OpLSet
	// bitmap
	OpBitSet
	OpBitFlip
	OpBitMerge // or, and, xor
	// zset
	OpZAdd
	OpZIncr
	OpZRemove
)

type Options ¶ added in v1.7.3

type Options struct {
	// Dir path if the db storage path.
	DirPath string

	// ShardCount is the shard numbers for underlying hashmap.
	ShardCount uint32

	// SyncPolicy is whether to synchronize writes to disk.
	// Setting `Sync` is required for durability of a single write operation, but also results in slower writes.
	// Setting `EverySecond` is much faster, but less durable as it relies on the OS to flush the writes to disk.
	SyncPolicy SyncPolicy

	// ShrinkCronExpr sauto shrink will be triggered when cron expr is satisfied.
	// cron expression follows the standard cron expression.
	// e.g. "0 0 * * *" means merge at 00:00:00 every day.
	// Setting empty string "" will disable auto shrink.
	ShrinkCronExpr string

	// Skip error when loading db file when startup.
	RunSkipLoadError bool
}

Options represents the configuration for rotom.

type Set ¶

type Set = *structx.Set

Type aliases for built-in types.

type String ¶

type String = []byte

Type aliases for built-in types.

type SyncPolicy ¶ added in v1.7.3

type SyncPolicy byte

SyncPolicy represents how often data is synced to disk.

const (
	EverySecond SyncPolicy = iota
	Sync
)

type Type ¶ added in v1.6.0

type Type = int64

Type is the data type for Rotom.

const (
	TypeString Type = iota + 1
	TypeMap
	TypeSet
	TypeList
	TypeZSet
	TypeBitmap
)

type ZSet ¶

type ZSet = *structx.ZSet[string, float64]

Type aliases for built-in types.

Directories ¶

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL