genrawid

README

Warning This package has been deprecated. We have found a faster, simpler and more reliable way. See issue #12.

GenRawID

genrawid is a niche command line tool and/or Go package to generate a unique consistent number from the input.

$ genrawid --string "abcdefgh"
-2474118025671277174
$ # 'abcdefgh'           --> The data to store in SQLite3
$ # -2474118025671277174 --> The primary key/rawid of stored data

$ echo -n 'abcdefgh' > ./sample.txt
$ genrawid ./sample.txt
-2474118025671277174
$ # 'abcdefgh'           --> The data to store in SQLite3
$ # -2474118025671277174 --> The primary key/rawid of stored data

genrawid is similar to a hash function, but its value is a combination of a hash and a checksum.

The hash value is the digest of the input and the checksum is the CRC of the complete hash value.

By default, BLAKE3-512 is used for the hash algorithm and CRC-32C (CRC-32 with Castagnoli polynomial) for the checksum.

8 Bytes = The first 4 Bytes of the hash + 4 Bytes of the checksum of the hash

• See benchmark of BLAKE3 and CRC-32 comparing to other hash algorithms:
  • https://github.com/KEINOS/go-blake3-example/blob/main/bench_results/bench_results_stats.txt

Why?

The main objective is to use SQLite3 as a fast KVS (Key-Value-Store) for CAS (Content-Addressable-Storage) usage.

genrawid generates a 64-bit/8-byte signed decimal number that can be used as rawid in SQLite3.

Searching for a record with a specific rowid, or for all records with rowids within a specified range is around twice as fast as a similar search made by specifying any other PRIMARY KEY or indexed value.

_{^{(From "ROWIDs and the INTEGER PRIMARY KEY" @ sqlite.org)}}

So, in theory, if you know the rawid of the content, you can find it twice as fast.

On the other hand, this command/package is of little use when using SQLite3 as RDB (Relational Database) or when dealing with mutable content.

Install

Download the binary for your operating system and architecture and place it as an executable in your PATH.

• Latest Releases (Windows, macOS, Linux, RaspberryPi)

For Homebrew/Linuxbrew users:

brew install KEINOS/apps/genrawid

Statuses

• Unit Tests/Code Coverage

• Secrurity/Vulnerability Check

CONTRIBUTING

• GolangCI Lint rules: .golangci-lint.yml
• To run tests in a container:
  • docker-compose --file ./.github/docker-compose.yml run v1_17
  • This will run:
    • go test -cover -race ./...
    • golangci-lint run
    • golint ./...
• Branch to PR: main
  • It is recommended that DraftPR be done first to avoid duplication of work.

License

• MIT, Copyright (c) KEINOS and the GenRawID contributors.

Documentation

Overview

Package genrawid provides functions to generate rawid.

For the sample implementation see: ./cmd/genrawid/main.go

Index

• Variables
• func FromFile(path string) (rawid.ID, error)
• func FromStdin() (rawid.ID, error)
• func FromString(input string) (rawid.ID, error)

Examples

• FromFile
• FromFile (Fast_mode)
• FromStdin
• FromString

Variables

var IsModeFast = false

IsModeFast is the flag to use fast mode. It will use the last 16bit of the hash as the xor16 checksum.

var OsStdin = os.Stdin

OsStdin is a copy of os.Stdin to ease testing. Mock this variable during tests.

Functions

func FromFile

func FromFile(path string) (rawid.ID, error)

FromFile returns the rawid generated from the input file.

Example

const pathFile = "./testdata/msg.txt" // msg.txt ==> "abcdefgh"

rawid, err := genrawid.FromFile(pathFile)
if err != nil {
	log.Fatal(err)
}

fmt.Println(rawid.Hex())
fmt.Println(rawid.Dec())

Output:

ddaa2ac39b79058a
-2474118025671277174

Example (Fast_mode)

oldMode := genrawid.IsModeFast
defer func() {
	genrawid.IsModeFast = oldMode
}()

// Set to fast mode
genrawid.IsModeFast = true

pathFile := "./testdata/msg.txt" // msg.txt ==> "abcdefgh"

rawid, err := genrawid.FromFile(pathFile)
if err != nil {
	fmt.Println(err.Error())

	return
}

fmt.Println(rawid.Hex())
fmt.Println(rawid.Dec())

Output:

ddaa2ac30a98963b
-2474118028101904837

func FromStdin

func FromStdin() (rawid.ID, error)

FromStdin returns the rawid generated from stdin as its input.

Example

ExampleFromStdin

Since we can not receive input from stdin during the example run, we mock the OsStdin, the copy of os.Stdin in the package, by assigning a dummy file as its input.

// Backup and defer recover the stdin
oldStdin := genrawid.OsStdin
defer func() {
	genrawid.OsStdin = oldStdin
}()

// Mock the stdin by assigning a file to stdin.
pathFile := "./testdata/msg.txt" // msg.txt ==> "abcdefgh"

osFile, err := os.Open(pathFile)
if err != nil {
	fmt.Println(err.Error())

	return
}

genrawid.OsStdin = osFile // <-- mock!

// Here is the actual example usage of FromStdin method to get the input from
// stdin and generate a rawid.
rawid, err := genrawid.FromStdin()
if err != nil {
	fmt.Println(err.Error())

	return
}

fmt.Println(rawid.Hex())
fmt.Println(rawid.Dec())

Output:

ddaa2ac39b79058a
-2474118025671277174

func FromString

func FromString(input string) (rawid.ID, error)

FromString returns the rawid generated from the input string.

Example

input := "abcdefgh"

rawid, err := genrawid.FromString(input)
if err != nil {
	log.Fatal(err)
}

fmt.Println(rawid.Hex())
fmt.Println(rawid.Dec())

Output:

ddaa2ac39b79058a
-2474118025671277174