hybridlog

README

HybridLog

HybridLog is an Golang implementation of an append-only file with concurrent reads/writes support and durability guarantee.

HybridLog writes as fast as os.File.Write with O_APPEND flag, but performs read ~20x faster than os.File.ReadAt on 512b-blocks, ~10x faster on 4KB-blocks and ~1.2x faster on 1M-blocks.

Getting Started

Installation

go get github.com/snowflk/hybridlog/...

Create a hybrid log

package main
import (
    "log"

    "github.com/snowflk/hybridlog"
)

func main()  {
    hlog, err := hybridlog.Open(hybridlog.Config{
        Path:"mydata.log",
    })
    if err != nil{
        log.Fatal(err)    
    }
    defer hlog.Close()
    // Do something
}

Write

HybridLog implements the interface io.Writer, so you can use it as follows:

if _, err := hlog.Write([]byte("My data")); err != nil {
    return err
}

Read

HybridLog implements the interface io.ReaderAt, so you can use it as follows:

// Read 128 bytes
data := make([]byte, 128)
if _, err := hlog.ReadAt(data, 0); err != nil {
    return err
}

Advanced configurations

Auto Compaction

Each write operation creates a fragment. Too fragmented data will result to negative impact on performance. AutoCompaction takes care of the defragmentation process.

Open HybridLog in AutoCompaction mode to enable this feature. There are 2 modes: TimeBased and FragmentationBased.

TimeBased mode performs compaction in a fixed interval. FragmentationBased mode performs compaction when the number of fragments exceeds a defined threshold.

hlog, err := hybridlog.Open(hybridlog.Config{
    Path:"mydata.log",
    AutoCompaction: true,
    CompactionMode: hybridlog.TimeBased,
    CompactAfter: 15 * time.Minute // Perform compaction every 15 minutes
    // ...
})

Sync Policy

In order to guarantee durability, fsync must be called. However, fsync has negative impact on write performance. Therefore, you can choose a sync policy to configure when to perform fsync, depending on your requirements.

There are 3 policies:

• NoSync let the system determine when to sync. This is the default policy.
• AlwaysSync for strong durability, this policy uses the flag O_SYNC for opening the file.
• SyncEverySec performs sync every second.

hlog, err := hybridlog.Open(hybridlog.Config{
    Path:"mydata.log",
    SyncPolicy: hybridlog.AlwaysSync,
    // ...
})

Benchmark

In this benchmark, HybridLog will be compared to the built-in os.File.

Technical specification

Processor: 2,8 GHz Quad-Core Intel Core i7
RAM: 16 GB 2133 MHz LPDDR3
Disk: Apple built-in SSD

HybridLog's Result

BenchmarkHybridLog_Write_512b
BenchmarkHybridLog_Write_512b-8    	  116798	      9645 ns/op
BenchmarkHybridLog_Write_1KB
BenchmarkHybridLog_Write_1KB-8     	  107182	     11045 ns/op
BenchmarkHybridLog_Write_4KB
BenchmarkHybridLog_Write_4KB-8     	   60058	     19029 ns/op
BenchmarkHybridLog_Write_128KB
BenchmarkHybridLog_Write_128KB-8   	   10000	    131224 ns/op
BenchmarkHybridLog_Write_1MB
BenchmarkHybridLog_Write_1MB-8     	    1600	    976735 ns/op
BenchmarkHybridLog_Read_512b
BenchmarkHybridLog_Read_512b-8     	21326368	        54.8 ns/op
BenchmarkHybridLog_Read_1KB
BenchmarkHybridLog_Read_1KB-8      	19704802	        61.0 ns/op
BenchmarkHybridLog_Read_4KB
BenchmarkHybridLog_Read_4KB-8      	13594483	        89.3 ns/op
BenchmarkHybridLog_Read_128KB
BenchmarkHybridLog_Read_128KB-8    	  292083	      3861 ns/op
BenchmarkHybridLog_Read_1MB
BenchmarkHybridLog_Read_1MB-8      	   24596	     49447 ns/op

os.File's Result

BenchmarkFile_Write_512b
BenchmarkFile_Write_512b-8         	  126115	      9682 ns/op
BenchmarkFile_Write_1KB
BenchmarkFile_Write_1KB-8          	  101589	     11216 ns/op
BenchmarkFile_Write_4KB
BenchmarkFile_Write_4KB-8          	   99919	     11087 ns/op
BenchmarkFile_Write_128KB
BenchmarkFile_Write_128KB-8        	   12734	     97205 ns/op
BenchmarkFile_Write_1MB
BenchmarkFile_Write_1MB-8          	    1420	    789027 ns/op
BenchmarkFile_Read_512b
BenchmarkFile_Read_512b-8          	 1325258	       916 ns/op
BenchmarkFile_Read_1KB
BenchmarkFile_Read_1KB-8           	 1333854	       922 ns/op
BenchmarkFile_Read_4KB
BenchmarkFile_Read_4KB-8           	 1205302	       981 ns/op
BenchmarkFile_Read_128KB
BenchmarkFile_Read_128KB-8         	  182815	      6725 ns/op
BenchmarkFile_Read_1MB
BenchmarkFile_Read_1MB-8           	   18992	     60004 ns/op

Documentation

Index

• Constants
• type CompactionMode
• type Config
• type HybridLog
  • func Open(cfg Config) (HybridLog, error)
• type SyncPolicy

Constants

const (
	TimeBased CompactionMode = iota
	FragmentationBased
	NoSync SyncPolicy = iota
	AlwaysSync
	SyncEverySecond
)

Types

type CompactionMode

type CompactionMode int

type Config

type Config struct {
	// simpleHybridLog configurations
	// Path simply locates where to store the data. The file extension needs to be provided.
	Path string
	// HighWaterMark defines the fullness of buffer (in percent) at which the remapping process will be started
	HighWaterMark int
	// The size of buffer (in bytes) for keeping the new written data in memory.
	// The larger the buffer size is, the more memory will be consumed.
	BufferSize  int
	OpenTimeout time.Duration
	// SyncPolicy denotes when to perform fdatasync
	SyncPolicy SyncPolicy

	// Compactor configurations
	AutoCompaction bool
	CompactionMode CompactionMode
	// CompactAfter defines when to launch a compaction process.
	// If CompactionMode is TimeBased, the value is the number of seconds.
	// If CompactionMode is FragmentationBased, the value is the number of fragments.
	CompactAfter int
	// The maximum size of the chunks to copy
	CompactionChunkSize int
}

type HybridLog

type HybridLog interface {
	Write(p []byte) (int, error)
	ReadAt(b []byte, off int64) (int, error)
	Size() int64
	Close() error
}

HybridLog represents an append-only file that supports extremely fast read / write operations. HybridLog is compliant to the interface io.Writer and io.ReaderAt. A HybridLog reads data from memory most of the time, and directly from disk if re-mapping is in progress. New data will be written into both disk and memory (into a buffer). If the data in the buffer exceed a defined high water mark, a re-mapping process will be performed.

func Open

func Open(cfg Config) (HybridLog, error)

Open opens a hybrid log at the given path in config. If the log does not exist, it creates a new one.

type SyncPolicy

type SyncPolicy int