kvfile

package module
v0.7.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 10, 2024 License: MIT Imports: 14 Imported by: 0

README

Key-Value File (kvfile)

Go Reference Go Report Card Widget

Introduction

go-kvfile stores key/value pairs to a file with O(log N) (binary search) lookup.

The values are concatenated together at the beginning of the file, followed by a set of length-suffixed entries containing each key and the offset of the associated value, followed by a list of positions of index entries.

The compress package supports seekable-zstd compressed kvfiles.

CLI

The kvfile CLI can be used to read/write a kvfile on the command line:

NAME:
   kvfile - A CLI tool for working with key-value files

COMMANDS:
   count    Print the number of keys in a k/v file.
   keys     Print all keys in a k/v file in sorted order.
   values   Print all key-value pairs in a k/v file.
   get      Get the value for a specific key.
   write    Write a new kvfile from JSON input.

GLOBAL OPTIONS:
   --binary-keys           read and log keys as binary (base58) (default: false)
   --binary-values         read and log values as binary (base58) (default: true)
   --file value, -f value  path to the kvfile to read
   --compress              use kvfile compression (default: false)

Usage

The Reader reads values from the kvfile and can search for specific keys using a binary search on the key index:

// Read keys from the file.
Get(): Looks up the value for the given key.
ReadTo(): Reads the value for the given key to the writer.
Exists(): Checks if the given key exists in the store.

// Iterate over keys in the file.
ScanPrefix(): iterates over key/value pairs with a prefix.
ScanPrefixKeys(): iterates over key/value pairs with a prefix, returning keys only.

// Utilities for reading the file structure.
ReadIndexEntry(): Reads the index entry at the given index.
SearchIndexEntry(): Looks up an index entry for the given key.
GetValuePosition(): Determines the position and length of the value for the key.

Write() writes the given key-value pairs to the file with the writer.

	var buf bytes.Buffer
	keys := [][]byte{
		[]byte("test-1"),
		[]byte("test-2"),
		[]byte("test-3"),
	}
	vals := [][]byte{
		[]byte("val-1"),
		[]byte("val-2"),
		[]byte("val-3"),
	}
	var index int
	err := Write(&buf, keys, func(wr io.Writer, key []byte) (uint64, error) {
		nw, err := wr.Write(vals[index])
		if err != nil {
			return 0, err
		}
		index++
		return uint64(nw), nil
	})

The Writer can be used to incrementally write keys and values to a file.

	var buf bytes.Buffer
	keys := [][]byte{
		[]byte("test-2"),
		[]byte("test-3"),
		[]byte("test-1"),
	}
	vals := [][]byte{
		[]byte("val-2"),
		[]byte("val-3"),
		[]byte("val-1"),
	}

	wr := NewWriter(&buf)
	for i, key := range keys {
		if err := wr.WriteValue(key, bytes.NewReader(vals[i])); err != nil {
			t.Fatal(err.Error())
		}
	}

	if err := wr.Close(); err != nil {
		t.Fatal(err.Error())
	}

Support

Please open a GitHub issue with any questions / issues.

... or feel free to reach out on Matrix Chat or Discord.

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Write 

func Write(writer io.Writer, keys [][]byte, writeValue WriteValueFunc) error

Write writes the given key/value pairs to the store in writer. Serializes and writes the key/value pairs. Note: keys must not contain duplicates or an error will be returned. The values will be stored in the order of the original keys slice. writeValue should write the given value to the writer returning the number of bytes written.

func WriteIndex added in v0.5.0 

func WriteIndex(writer io.Writer, index []*IndexEntry, pos uint64) (uint64, error)

WriteIndex sorts and checks the index entries and writes them to a file.

pos is the position the writer is located at in the file. returns the number of bytes written (end pos - pos).

func WriteIterator 

func WriteIterator(writer io.Writer, keyIterator KeyIteratorFunc, writeValueFunc WriteValueFunc) error

WriteIterator writes the key/value pairs using the given iterators.

WriteValueFunc writes a value and returns number of bytes written and any error. WriteIteratorFunc is a function that returns key/value pairs to write.

Note: keys must not contain duplicates or an error will be returned.

Types

type FileReaderAt 

type FileReaderAt interface {
	fs.File
	io.ReaderAt
}

FileReaderAt is a fs.File that implements ReaderAt.

type IndexEntry 

type IndexEntry struct {

	// Key is the key of the entry.
	Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
	// Offset is the position of the value in bytes.
	Offset uint64 `protobuf:"varint,2,opt,name=offset,proto3" json:"offset,omitempty"`
	// Size is the size of the value in bytes.
	Size uint64 `protobuf:"varint,3,opt,name=size,proto3" json:"size,omitempty"`
	// contains filtered or unexported fields
}

IndexEntry is an entry in the index. The index is sorted by key.

func (*IndexEntry) CloneMessageVT 

func (m *IndexEntry) CloneMessageVT() protobuf_go_lite.CloneMessage

func (*IndexEntry) CloneVT 

func (m *IndexEntry) CloneVT() *IndexEntry

func (*IndexEntry) EqualMessageVT 

func (this *IndexEntry) EqualMessageVT(thatMsg any) bool

func (*IndexEntry) EqualVT 

func (this *IndexEntry) EqualVT(that *IndexEntry) bool

func (*IndexEntry) GetKey 

func (x *IndexEntry) GetKey() []byte

func (*IndexEntry) GetOffset 

func (x *IndexEntry) GetOffset() uint64

func (*IndexEntry) GetSize added in v0.5.0 

func (x *IndexEntry) GetSize() uint64

func (*IndexEntry) MarshalJSON added in v0.6.0 

func (x *IndexEntry) MarshalJSON() ([]byte, error)

MarshalJSON marshals the IndexEntry to JSON.

func (*IndexEntry) MarshalProtoJSON added in v0.6.0 

func (x *IndexEntry) MarshalProtoJSON(s *json.MarshalState)

MarshalProtoJSON marshals the IndexEntry message to JSON.

func (*IndexEntry) MarshalProtoText added in v0.6.2 

func (x *IndexEntry) MarshalProtoText() string

func (*IndexEntry) MarshalToSizedBufferVT 

func (m *IndexEntry) MarshalToSizedBufferVT(dAtA []byte) (int, error)

func (*IndexEntry) MarshalToVT 

func (m *IndexEntry) MarshalToVT(dAtA []byte) (int, error)

func (*IndexEntry) MarshalVT 

func (m *IndexEntry) MarshalVT() (dAtA []byte, err error)

func (*IndexEntry) ProtoMessage 

func (*IndexEntry) ProtoMessage()

func (*IndexEntry) Reset 

func (x *IndexEntry) Reset()

func (*IndexEntry) SizeVT 

func (m *IndexEntry) SizeVT() (n int)

func (*IndexEntry) String 

func (x *IndexEntry) String() string

func (*IndexEntry) UnmarshalJSON added in v0.6.0 

func (x *IndexEntry) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals the IndexEntry from JSON.

func (*IndexEntry) UnmarshalProtoJSON added in v0.6.0 

func (x *IndexEntry) UnmarshalProtoJSON(s *json.UnmarshalState)

UnmarshalProtoJSON unmarshals the IndexEntry message from JSON.

func (*IndexEntry) UnmarshalVT 

func (m *IndexEntry) UnmarshalVT(dAtA []byte) error

type KeyIteratorFunc 

type KeyIteratorFunc func() (key []byte, err error)

WriteIteratorFunc is a function that returns key/value pairs to write. The callback should return one key at a time in the order they should be written to the file. Return nil, nil or nil, io.EOF if no keys remain.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader is a key/value file reader.

func BuildReader 

func BuildReader(rd io.ReaderAt, fileSize uint64) (*Reader, error)

BuildReader constructs a new Reader, reading the number of index entries.

func BuildReaderWithFile 

func BuildReaderWithFile(f FileReaderAt) (*Reader, error)

BuildReaderWithFile constructs a new Reader with an fs.File.

func BuildReaderWithSeeker 

func BuildReaderWithSeeker(rd ReaderAtSeeker) (*Reader, error)

BuildReaderWithSeeker constructs a new Reader with a io.ReaderSeeker reading the file size.

func (*Reader) Exists 

func (r *Reader) Exists(key []byte) (bool, error)

Exists checks if the given key exists in the store.

func (*Reader) Get 

func (r *Reader) Get(key []byte) ([]byte, bool, error)

Get looks up the value for the given key. Returns nil, false, nil if not found

func (*Reader) GetValuePosition 

func (r *Reader) GetValuePosition(key []byte) (idx, length int64, indexEntry *IndexEntry, indexEntryIdx int, err error)

GetValuePosition determines the position and length of the value for the key.

Returns -1, 1, nil, -1, nil if not found.

func (*Reader) GetValuePositionWithEntry 

func (r *Reader) GetValuePositionWithEntry(indexEntry *IndexEntry, indexEntryIdx int) (idx, length int64, err error)

GetValuePositionWithEntry determines the position and length of the value with an entry.

Returns -1, 1, nil, -1, nil if not found.

func (*Reader) GetValueSize added in v0.4.1 

func (r *Reader) GetValueSize(key []byte) (int64, error)

GetValueSize looks up the size of the value for the given key without reading the value. Returns -1, nil if not found.

func (*Reader) GetWithEntry 

func (r *Reader) GetWithEntry(indexEntry *IndexEntry, indexEntryIdx int) ([]byte, error)

GetWithEntry returns the value for the given index entry.

func (*Reader) ReadIndexEntry 

func (r *Reader) ReadIndexEntry(indexEntryIdx uint64) (*IndexEntry, error)

ReadIndexEntry reads the index entry at the given index.

func (*Reader) ReadTo 

func (r *Reader) ReadTo(key []byte, to io.Writer) (int, bool, error)

ReadTo reads the value for the given key to the writer.

Returns number of bytes read, found, and any error. Returns 0, false, nil if not found.

func (*Reader) ScanPrefix 

func (r *Reader) ScanPrefix(prefix []byte, cb func(key, value []byte) error) error

ScanPrefix iterates over key/value pairs with a prefix.

func (*Reader) ScanPrefixEntries 

func (r *Reader) ScanPrefixEntries(prefix []byte, cb func(indexEntry *IndexEntry, indexEntryIdx int) error) error

ScanPrefixEntries iterates over entries with the given key prefix.

func (*Reader) ScanPrefixKeys 

func (r *Reader) ScanPrefixKeys(prefix []byte, cb func(key []byte) error) error

ScanPrefixKeys iterates over keys with a prefix.

func (*Reader) SearchIndexEntryWithKey 

func (r *Reader) SearchIndexEntryWithKey(key []byte) (*IndexEntry, int, error)

SearchIndexEntryWithKey looks up an index entry for the given key.

If not found, returns nil, idx, err and idx is the index where the searched element would appear if inserted into the list.

func (*Reader) SearchIndexEntryWithPrefix 

func (r *Reader) SearchIndexEntryWithPrefix(prefix []byte, last bool) (*IndexEntry, int, error)

SearchIndexEntryWithPrefix returns the entry of the first key with the prefix.

If last is true returns the last element that matches the prefix.

If the key or prefix is not found, returns nil, idx, err, where idx is the element where an element with the given prefix would appear if inserted.

func (*Reader) Size 

func (r *Reader) Size() uint64

Size returns the number of key/value pairs in the store.

type ReaderAtSeeker 

type ReaderAtSeeker interface {
	io.ReaderAt
	io.ReadSeeker
}

ReaderAtSeeker is a ReaderAt and a ReadSeeker.

type WriteValueFunc 

type WriteValueFunc func(wr io.Writer, key []byte) (uint64, error)

WriteValueFunc is a function that writes the value for a key to a writer. Return the number of bytes written and any error.

type Writer added in v0.5.0 

type Writer struct {
	// contains filtered or unexported fields
}

Writer allows progressively writing values to a kvfile. The index will be written once the writer is closed (flushed). Note: keys must not contain duplicates or an error will be returned. Concurrency safe.

func NewWriter added in v0.5.0 

func NewWriter(out io.Writer) *Writer

NewWriter builds a new writer.

func (*Writer) Close added in v0.5.0 

func (w *Writer) Close() error

Close completes the Writer by writing the index to the file.

func (*Writer) GetPos added in v0.5.0 

func (w *Writer) GetPos() uint64

GetPos returns the current write position (written size).

func (*Writer) WriteValue added in v0.5.0 

func (w *Writer) WriteValue(key []byte, valueRdr io.Reader) error

WriteValue writes a key/value pair to the kvfile writer.

The writer is closed if an error is returned.

Source Files

View all Source files

Directories

Path Synopsis
cmd
kvfile

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.