hakjdb

README

Overview

HakjDB is a simple in-memory key-value data store that was built as an educative hobby project. It is written in the Go programming language.

HakjDB allows you to store key-value pairs of different data types in namespaces called databases. The data is stored in the server's memory.

HakjDB uses a simple client-server model, and has a well-defined and documented gRPC API. It can be used as a temporary database, session storage or cache. It may not be suitable for advanced needs, and does not offer data persistence on disk.

Data is stored at keys of different types. Each data type allows you to store different kind of data such as string values or objects.

Instances are easily configurable with command line flags, environment variables and a simple YAML file.

Components:

• hakjserver - The server process
• hakjctl - CLI tool to control and interact with the server
• hakjdb-gui - Cross-platform GUI desktop application to visually interact with the server

This is not meant for production use, as it lacks a lot of features, and is not as professional as some popular key-value stores.

How it works

The following diagram shows the architecture

Documentation

• Using the server
• Using the CLI
• Configuring the server
• Databases
• Data types
• API

Releases

Release notes are available here.

Prebuilt binaries are available for download. Multiple platforms supported. You can download them here.

However, it is recommended to build the binaries from source or install them with Go.

Releases are managed with GoReleaser.

NOTE: This project went by name kvdb before v1.0.0. That's why name kvdb appears in older releases.

Install binaries

The binaries can be installed with Go.

Install the server binary

go install github.com/hollowdll/hakjdb/cmd/hakjserver@latest

Install the CLI binary

go install github.com/hollowdll/hakjdb/cmd/hakjctl@latest

These will install the binaries to your Go bin directory. It is recommened to create separate directories for the binaries and then place the binaries in these directories because the binaries create some files relative to their parent directory.

For example on Linux

mkdir -p ~/hakjdb/hakjserver
mkdir -p ~/hakjdb/hakjctl
mv hakjserver ~/hakjdb/hakjserver
mv hakjctl ~/hakjdb/hakjctl

Build binaries

To build the binaries from source, you first need to install Go. Minimum version required is go1.22.

Instructions for installing Go can be found here.

You may also need tools to work with gRPC and Protocol Buffers in Go. This is needed if you want to compile .proto files and generate Go code.

• Protocol Buffer compiler
• Quickstart

If you just want to compile the binaries then installing only Go is enough.

After you have successfully installed go, clone this repository.

Cloning with git:

git clone https://github.com/hollowdll/hakjdb.git

Note: You can also download the source code for a specific release here.

Change directory to the project root:

cd hakjdb

Get the dependencies:

go mod tidy

Build the server:

go build -o ./bin/hakjserver/ ./cmd/hakjserver/

Build the CLI:

go build -o ./bin/hakjctl/ ./cmd/hakjctl/

These will build the binaries to bin/ directory in the project root. You can change the output directory and binary names to whatever you like by modifying the path with -o flag.

For more advanced build, use go help build to see more build options.

You can also use the scripts build_hakjctl.sh and build_hakjserver.sh to build the binaries.

Docker

Images are available in Docker Hub with multiple tags. Links below.

• Repository
• Old repository

Pull the server image

docker pull hakj/hakjdb

Build the server image

Make sure to be in the project root

cd hakjdb

Latest tag

docker build -f "./Dockerfile.bookworm" -t hakjdb:latest .

Debian based image

docker build -f "./Dockerfile.bookworm" -t hakjdb:bookworm .

Alpine Linux based image

docker build -f "./Dockerfile.alpine" -t hakjdbdb:alpine .

These commands build the image only for a single architecture. If you want to build multi-arch images for other platforms, read this.

Start a container

Example of starting a container

docker run -p 12345:12345 --rm -it hakjdb

This binds the host's port 12345 to the container's port 12345 so you can access the server outside the container.

Running tests

Change directory to the project root:

cd hakjdb

Run all tests:

go test ./...

Run only integration tests:

go test ./tests/integration

Show verbose test result output:

go test -v ./...

Integration tests

Running the integration tests starts test servers both with and without TLS. The servers are stopped after the tests have finished.

The operating system assigns random TCP ports for the test servers. You can output the assigned ports to the console by running the integration tests with -v flag.

For example:

go test -v ./tests/integration

Benchmarks

Benchmarks are useful for testing the average performance of the server. They can measure the average number of requests that can be performed in a second.

Run benchmarks:

cd tests/benchmark
go test -bench=. -v

Results show that the best performance is achieved when not using authentication or TLS. They reduce the throughput a little bit. Benchmarks were done for String key writes and reads. They have about the same average performance.

Benchmarks were run with Intel Ultra 7 155H CPU and requests were handled in parallel using goroutines.

Average results are shown in the table below

Auth enabled	TLS enabled	Average requests per second
No	No	~33000
Yes	No	~24000
Yes	Yes	~22000

License

This project is licensed under MIT license. It is free and open source software.

Documentation

Index

• Constants
• type DB
  • func NewDB(name string, desc string, cfg DBConfig) *DB
  • func (db *DB) ChangeDescription(newDescription string)
  • func (db *DB) ChangeName(newName string)
  • func (db *DB) CreatedAt() time.Time
  • func (db *DB) DeleteAllKeys()
  • func (db *DB) DeleteHashMapFields(key string, fields []string) (uint32, bool)
  • func (db *DB) DeleteKeys(keys []string) uint32
  • func (db *DB) Description() string
  • func (db *DB) GetAllKeys() []string
  • func (db *DB) GetEstimatedStorageSizeBytes() uint64
  • func (db *DB) GetHashMapFieldValues(key string, fields []string) (map[string]*HashMapFieldValueResult, bool)
  • func (db *DB) GetHashMapKey(key string) (HashMapKey, bool)
  • func (db *DB) GetKeyCount() int
  • func (db *DB) GetKeyType(key string) (DBKeyType, bool)
  • func (db *DB) GetStringKey(key string) (StringKey, bool)
  • func (db *DB) Name() string
  • func (db *DB) SetHashMap(key string, fields map[string][]byte) uint32
  • func (db *DB) SetString(key string, value []byte)
  • func (db *DB) UpdatedAt() time.Time
• type DBConfig
• type DBKey
• type DBKeyType
  • func (t DBKeyType) String() string
• type DefaultLogger
  • func DisabledLogger() *DefaultLogger
  • func NewDefaultLogger() *DefaultLogger
  • func (l *DefaultLogger) CloseLogFile() error
  • func (l *DefaultLogger) Debug(v ...any)
  • func (l *DefaultLogger) Debugf(format string, v ...any)
  • func (l *DefaultLogger) Disable()
  • func (l *DefaultLogger) EnableLogFile(filepath string) error
  • func (l *DefaultLogger) Error(v ...any)
  • func (l *DefaultLogger) Errorf(format string, v ...any)
  • func (l *DefaultLogger) Fatal(v ...any)
  • func (l *DefaultLogger) Fatalf(format string, v ...any)
  • func (l *DefaultLogger) Info(v ...any)
  • func (l *DefaultLogger) Infof(format string, v ...any)
  • func (l *DefaultLogger) LogLevel() (LogLevel, string)
  • func (l *DefaultLogger) SetLogLevel(level LogLevel)
  • func (l *DefaultLogger) Warning(v ...any)
  • func (l *DefaultLogger) Warningf(format string, v ...any)
• type HashMapField
• type HashMapFieldValueResult
• type HashMapKey
• type HashMapValue
• type LogLevel
  • func GetLogLevelFromStr(levelStr string) (LogLevel, string, bool)
• type Logger
• type StringKey
• type StringValue

Constants

const (
	LogLevelDebug   LogLevel = 0
	LogLevelInfo    LogLevel = 1
	LogLevelWarning LogLevel = 2
	LogLevelError   LogLevel = 3
	LogLevelFatal   LogLevel = 4

	LogLevelDebugStr   string = "debug"
	LogLevelInfoStr    string = "info"
	LogLevelWarningStr string = "warning"
	LogLevelErrorStr   string = "error"
	LogLevelFatalStr   string = "fatal"

	DefaultLogLevel    LogLevel = LogLevelInfo
	DefaultLogLevelStr string   = LogLevelInfoStr
)

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is a database used as a namespace for storing key-value pairs.

func NewDB

func NewDB(name string, desc string, cfg DBConfig) *DB

func (*DB) ChangeDescription

func (db *DB) ChangeDescription(newDescription string)

func (*DB) ChangeName

func (db *DB) ChangeName(newName string)

func (*DB) CreatedAt

func (db *DB) CreatedAt() time.Time

func (*DB) DeleteAllKeys

func (db *DB) DeleteAllKeys()

DeleteAllKeys deletes all the keys.

func (*DB) DeleteHashMapFields

func (db *DB) DeleteHashMapFields(key string, fields []string) (uint32, bool)

DeleteHashMapFields removes the specified fields from a HashMap key value. Returns the number of removed fields. The returned bool is true if the key exists and holds a HashMap.

func (*DB) DeleteKeys

func (db *DB) DeleteKeys(keys []string) uint32

DeleteKeys deletes the specified keys. Returns the number of keys that were deleted.

func (*DB) Description

func (db *DB) Description() string

func (*DB) GetAllKeys

func (db *DB) GetAllKeys() []string

GetAllKeys returns all the keys.

func (*DB) GetEstimatedStorageSizeBytes

func (db *DB) GetEstimatedStorageSizeBytes() uint64

GetEstimatedStorageSizeBytes returns the estimated size of stored data in bytes.

func (*DB) GetHashMapFieldValues

func (db *DB) GetHashMapFieldValues(key string, fields []string) (map[string]*HashMapFieldValueResult, bool)

GetHashMapFieldValues returns a HashMap key value's field values. The returned bool is true if the key exists, or false if the key doesn't exist.

func (*DB) GetHashMapKey

func (db *DB) GetHashMapKey(key string) (HashMapKey, bool)

GetHashMapKey retrieves a HashMap key value. The returned map is empty if the key doesn't exist. The returned bool is true if the key exists and holds a HashMap.

func (*DB) GetKeyCount

func (db *DB) GetKeyCount() int

GetKeyCount returns the number of keys in the database.

func (*DB) GetKeyType

func (db *DB) GetKeyType(key string) (DBKeyType, bool)

GetKeyType returns the data type of the key if it exists. The returned bool is true if the key exists and false if it doesn't.

func (*DB) GetStringKey

func (db *DB) GetStringKey(key string) (StringKey, bool)

GetStringKey retrieves a String key value. The returned bool is true if the key exists and holds a String.

func (*DB) Name

func (db *DB) Name() string

func (*DB) SetHashMap

func (db *DB) SetHashMap(key string, fields map[string][]byte) uint32

SetHashMap sets the specified fields in a HashMap key value, overwriting previous fields. Creates the key if it doesn't exist. Returns the number of added fields.

func (*DB) SetString

func (db *DB) SetString(key string, value []byte)

SetString sets a String key value, overwriting previous key. Creates the key if it doesn't exist.

func (*DB) UpdatedAt

func (db *DB) UpdatedAt() time.Time

type DBConfig

type DBConfig struct {
	// The maximum number of fields a HashMap key value can hold.
	MaxHashMapFields uint32
}

DBConfig contains fields to configure a database.

type DBKey

type DBKey string

DBKey represents a database key.

type DBKeyType

type DBKeyType string

DBKeyType represents the name of a database key data type.

const (
	StringKeyTypeName  DBKeyType = "String"
	HashMapKeyTypeName DBKeyType = "HashMap"
)

func (DBKeyType) String

func (t DBKeyType) String() string

type DefaultLogger

type DefaultLogger struct {
	// contains filtered or unexported fields
}

DefaultLogger is a default implementation of the Logger interface. Log output defaults to standard error stream. Debug logs are disabled by default. Call EnableDebug to enable them.

func DisabledLogger

func DisabledLogger() *DefaultLogger

func NewDefaultLogger

func NewDefaultLogger() *DefaultLogger

func (*DefaultLogger) CloseLogFile

func (l *DefaultLogger) CloseLogFile() error

func (*DefaultLogger) Debug

func (l *DefaultLogger) Debug(v ...any)

func (*DefaultLogger) Debugf

func (l *DefaultLogger) Debugf(format string, v ...any)

func (*DefaultLogger) Disable

func (l *DefaultLogger) Disable()

func (*DefaultLogger) EnableLogFile

func (l *DefaultLogger) EnableLogFile(filepath string) error

func (*DefaultLogger) Error

func (l *DefaultLogger) Error(v ...any)

func (*DefaultLogger) Errorf

func (l *DefaultLogger) Errorf(format string, v ...any)

func (*DefaultLogger) Fatal

func (l *DefaultLogger) Fatal(v ...any)

func (*DefaultLogger) Fatalf

func (l *DefaultLogger) Fatalf(format string, v ...any)

func (*DefaultLogger) Info

func (l *DefaultLogger) Info(v ...any)

func (*DefaultLogger) Infof

func (l *DefaultLogger) Infof(format string, v ...any)

func (*DefaultLogger) LogLevel

func (l *DefaultLogger) LogLevel() (LogLevel, string)

func (*DefaultLogger) SetLogLevel

func (l *DefaultLogger) SetLogLevel(level LogLevel)

func (*DefaultLogger) Warning

func (l *DefaultLogger) Warning(v ...any)

func (*DefaultLogger) Warningf

func (l *DefaultLogger) Warningf(format string, v ...any)

type HashMapField

type HashMapField struct {
	Value StringValue
}

HashMapField is a HashMap field that holds a String value.

type HashMapFieldValueResult

type HashMapFieldValueResult struct {
	// Value is the value the field is holding.
	FieldValue HashMapField
	// Ok is true if the field exists. Otherwise false.
	Ok bool
}

type HashMapKey

type HashMapKey struct {
	Value HashMapValue
}

HashMapKey is a database key that holds a HashMap value.

type HashMapValue

type HashMapValue map[string]HashMapField

HashMapValue represents a HashMap data type value.

type LogLevel

type LogLevel uint8

func GetLogLevelFromStr

func GetLogLevelFromStr(levelStr string) (LogLevel, string, bool)

GetLogLevelFromStr returns the log level that matches its string equivalent. If invalid log level string is passed, this function returns the default log level. The returned string is the log level's string equivalent in lowercase. The returned bool is true if the passed string is valid log level.

type Logger

type Logger interface {
	Debug(v ...any)
	Debugf(format string, v ...any)

	Info(v ...any)
	Infof(format string, v ...any)

	Error(v ...any)
	Errorf(format string, v ...any)

	Warning(v ...any)
	Warningf(format string, v ...any)

	Fatal(v ...any)
	Fatalf(format string, v ...any)

	// SetLogLevel sets the log level.
	SetLogLevel(level LogLevel)
	// LogLevel returns the log level and its string equivalent.
	LogLevel() (LogLevel, string)
	// EnableLogFile enables log file.
	EnableLogFile(filePath string) error
	// CloseLogFile closes the log file if it is open.
	CloseLogFile() error
	// Disable disables all log outputs.
	Disable()
}

type StringKey

type StringKey struct {
	Value StringValue
}

StringKey is a database key that holds a String value.

type StringValue

type StringValue []byte

StringValue represents a String data type value.