README
¶
sniper
A simple and efficient thread-safe key/value store for Go.
Getting Started
Features
- Store hundreds of millions of entries
- Fast. High concurrent. Thread-safe. Scales on multi-core CPUs
- Extremly low memory usage
- Zero GC overhead
- Simple, pure Go implementation
Installing
To start using sniper, install Go and run go get:
$ go get -u github.com/recoilme/sniper
This will retrieve the library.
Usage
The Sniper includes this methods:
Set, Get, Incr, Decr, Delete, Count, Open, Close, FileSize, Backup.
s, _ := sniper.Open(sniper.Dir("1"))
s.Set([]byte("hello"), []byte("go"))
res, _ = s.Get([]byte("hello"))
fmt.Println(res)
s.Close()
// Output:
// go
Performance
MacBook Pro 2019 (Quad-Core Intel Core i7 2,8 GHz, 16 ГБ, APPLE SSD AP0512M)
go version go1.14 darwin/amd64
number of cpus: 8
number of keys: 10000000
keysize: 10
random seed: 1570109110136449000
-- sniper --
set: 10,000,000 ops over 8 threads in 63159ms, 158,331/sec, 6315 ns/op, 644.3 MB, 67 bytes/op
get: 10,000,000 ops over 8 threads in 4455ms, 2,244,629/sec, 445 ns/op, 305.5 MB, 32 bytes/op
del: 10,000,000 ops over 8 threads in 37568ms, 266,182/sec, 3756 ns/op, 122.8 MB, 12 bytes/op
With fsync
set: 10,000,000 ops over 8 threads in 85088ms, 117,524/sec, 8508 ns/op, 644.4 MB, 67 bytes/op
get: 10,000,000 ops over 8 threads in 5623ms, 1,778,268/sec, 562 ns/op, 305.5 MB, 32 bytes/op
How it is done
- Sniper database is sharded on 250+ chunks. Each chunk has its own lock (RW), so it supports high concurrent access on multi-core CPUs.
- Each chunk store
hash(key) -> (value addr, value size), map. - Hash is very short, and has collisions. Sniper has collisions resolver.
- Efficient space reuse alghorithm. Every packet has power of 2 size, for inplace rewrite on value update and map of deleted entrys, for reusing space.
Limitations
- 512 Kb - maximum entry size
len(key) + len(value) - ~1 Tb - maximum database size
Mac OS tip
How to Change Open Files Limit on OS X and macOS
Contact
Vadim Kulibaba @recoilme
License
sniper source code is available under the MIT License.
Documentation
¶
Index ¶
- Constants
- Variables
- func DeleteStore(dir string) error
- func NextPowerOf2(v uint32) (power byte, val uint32)
- type Header
- type OptStore
- type Store
- func (s *Store) Backup(w io.Writer) (err error)
- func (s *Store) BackupGZ(w io.Writer) (err error)
- func (s *Store) Bucket(name string) (*sortedset.BucketStore, error)
- func (s *Store) Close() (err error)
- func (s *Store) Count() (cnt int)
- func (s *Store) Decr(k []byte, v uint64) (uint64, error)
- func (s *Store) Delete(k []byte) (isDeleted bool, err error)
- func (s *Store) Expire() (err error)
- func (s *Store) FileSize() (fs int64, err error)
- func (s *Store) Get(k []byte) (v []byte, err error)
- func (s *Store) Incr(k []byte, v uint64) (uint64, error)
- func (s *Store) Keys(bucket *sortedset.BucketStore, limit, offset int) []string
- func (s *Store) Put(bucket *sortedset.BucketStore, k, v []byte) (err error)
- func (s *Store) Restore(r io.Reader) (err error)
- func (s *Store) RestoreGZ(r io.Reader) (err error)
- func (s *Store) Set(k, v []byte, expire uint32) (err error)
- func (s *Store) Touch(k []byte, expire uint32) (err error)
Constants ¶
const Version = "0.4.1"
Variables ¶
var ErrCollision = errors.New("Error, hash collision")
ErrCollision - must not happen
var ErrFormat = errors.New("Error, unexpected file format")
ErrFormat unexpected file format
var ErrNotFound = errors.New("Error, key not found")
ErrNotFound key not found error
Functions ¶
func NextPowerOf2 ¶
NextPowerOf2 return next power of 2 for v and it's value return maxuint32 in case of overflow
Types ¶
type OptStore ¶ added in v0.2.0
OptStore is a store options
func ChunksCollision ¶ added in v0.3.0
ChunksCollision - number chunks for collisions resolving, default is 4 (>1_000_000_000 of 8 bytes alphabet keys without collision errors) different keys may has same hash collision chunks needed for resolving this, without collisions errors if ChunkColCnt - zero, ErrCollision will return in case of collision
func ChunksPrefix ¶ added in v0.4.0
ChunksPrefix - prefix for a chunks filename
func ChunksTotal ¶ added in v0.3.0
ChunksTotal - total chunks/shards, default 256 Must be more then collision chunks
func ExpireInterval ¶ added in v0.3.0
ExpireInterval - how often run key expiration process expire only one chunk
func SyncInterval ¶ added in v0.1.8
SyncInterval - how often fsync do, default 0 - OS will do it
type Store ¶
Store struct data in store sharded by chunks
func Open ¶
Open return new store It will create 256 shards Each shard store keys and val size and address in map[uint32]uint32
options, see https://gist.github.com/travisjeffery/8265ca411735f638db80e2e34bdbd3ae#gistcomment-3171484 usage - Open(Dir("1"), SyncInterval(1*time.Second))
func (*Store) Bucket ¶ added in v0.3.0
func (s *Store) Bucket(name string) (*sortedset.BucketStore, error)
Bucket - create new bucket for storing keys with same prefix in memory index
func (*Store) Keys ¶ added in v0.3.0
func (s *Store) Keys(bucket *sortedset.BucketStore, limit, offset int) []string
Keys will return keys stored with Put method Params: key prefix ("" - return all keys) Limit - 0, all Offset - 0, zero offset Keys will be without prefix and in descending order
func (*Store) Put ¶ added in v0.3.0
func (s *Store) Put(bucket *sortedset.BucketStore, k, v []byte) (err error)
Put - store key and val with Set And add key in index (backed by sortedset)
Source Files
Directories