spaceflake

package
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2023 License: MIT, GPL-3.0 Imports: 6 Imported by: 0

README

❄ Spaceflake ❄

Go Reference Repository License Code Size Last Commit

A distributed generator to create unique IDs with ease; inspired by Twitter's Snowflake. Blog post about this project can be found here.

What is a Snowflake?

Apart from being a crystal of snow, a snowflake is a form of unique identifier which is being used in distributed computing. It has specific parts and is 64 bits long in binary. I simply named my type of snowflake, a Spaceflake, as it does not compose of the same parts of a Twitter Snowflake and is being used for Project Absence and other projects of myself.

Structure

A Spaceflake is structured like the following: Parts of a 64 bits Spaceflake

Spaceflake Network

A Spaceflake Network is a very basic concept where you have multiple independent nodes that themselves consist of multiple workers. These workers are the ones that can generate a Spaceflake.

Ideally a Spaceflake Network represents your entire application, or company. Each node represents a single server or application within the company, and each worker represents a single process which can generate a Spaceflake for a specific purpose. This way you can easily identify where a Spaceflake was generated by looking at its node ID and worker ID.

In the end you are free to use them as you wish, just make sure you use these nodes and workers to be able to identify the Spaceflake.

Example Network

An example network is structured like the following A simple Spaceflake Network We can consider Node 1 as being the API/backend of your application. The Worker (ID: 1) would be responsible for generating Spaceflakes for user IDs. The Worker (ID: 2) would be responsible for generating Spaceflakes for blog post IDs.

The Node 2 might be responsible for the logs of your components, and the log ID generated would be generated by the Worker (ID: 1) from that node.

Some Statistics

  • A Spaceflake network can hold up to 31 nodes and 31 workers per node. So you can have up to 961 workers in total in a single network that will generate Spaceflakes.
  • A single worker can generate up to 4095 Spaceflakes per millisecond.
  • A single node with 31 workers can generate up to 126'945 Spaceflakes per millisecond.
  • A single network with 31 nodes and 31 workers per node can generate up to 3'935'295 Spaceflakes per millisecond.

Example

A very basic example on using the library is by using the generator without nodes and worker objects, though this is not recommended and using nodes and workers is better.

package main

import (
	"fmt"

	"github.com/kkrypt0nn/spaceflake"
)

func main() {
	node := spaceflake.NewNode(1)
	worker := node.NewWorker()
	sf, err := worker.GenerateSpaceflake()
	if err != nil {
		panic(err)
	}
	fmt.Println(sf.Decompose()) // map[id:<Spaceflake> nodeID:1 sequence:1 time:<timestamp> workerID:1]
}

Some other examples:

Installation

If you want to use this library for one of your projects, you can install it like any other Go library

go get github.com/kkrypt0nn/spaceflake

⚠️ Disclaimers

Spaceflakes are Big Numbers

📜 TL;DR: If you use Spaceflakes in an API, return them as a string, not a number.

Since Spaceflakes are big numbers, it is most likely that if you use them for an API that returns a JSON you will need to return the Spaceflake as a string, otherwise you will lose some precision and it will alter the value of, most likely, the sequence of the Spaceflake. Example:

{
	"id": 144328692659220480 // ID actually generated in Go: 144328692659220481
}

The difference between the two numbers is not that big in the example above, though it plays a big role. The difference is not always the same, so you can't subtract. JavaScript, for example, sees no difference between both of these numbers:

console.log(144328692659220480 == 144328692659220481) // true

You can get the Spaceflake as a string and convert to a uint64 data type, when needed, in your Go code using the following:

spaceflakeID := "144328692659220481" // Will be the value returned by the API
id, _ := strconv.ParseUint(spaceflakeID, 10, 64)
sequence := spaceflake.ParseSequence(id)
"Random" Sequence Based on Time

📜 TL;DR: The sequence is not truly random, it is based on the time; and if you generate lots of Spaceflake in the same millisecond, there is a chance that two Spaceflakes will result to the same. Using nodes and workers is highly recommended.

When generating lots of Spaceflakes in a really short time and without using a worker, there is a chance that the same ID is generated twice. Consider making your program sleep for 1 millisecond or test around between the generations, example:

func GenerateLotsOfSpaceflakes() {
	spaceflakes := map[uint64]*Spaceflake{}
	settings := NewGeneratorSettings()

	for i := 0; i < 1000; i++ {
		sf, err := Generate(settings)
		if err != nil {
			t.Error(err)
		}
		if spaceflakes[sf.ID()] != nil {
			panic(err)
		}
		spaceflakes[sf.ID()] = sf

		// When using random there is a chance that the sequence will be twice the same due to Go's speed, hence using a worker is better. We wait a millisecond to make sure it's different.
		time.Sleep(1 * time.Millisecond)
	}
}

In that case it is recommended to use the workers, as they do not use a random value as a sequence number, but an incrementing value. Another option would be to use the bulk generator to create lots of unique Spaceflakes at once.

As a last resort you can replace the sequence with a better random number generator using the following:

settings.Sequence = ... // Replace with your number generator

License

This library was made with 💜 by Krypton and is under the GPL v3 license.

Documentation

Index

Constants

View Source 
const (
	// EPOCH is the base epoch for the Spaceflake generation. First second of 2015, year when I got my username "Krypton"
	EPOCH = 1420070400000
	// MAXWORKERBITS is the maximum value for a 5 bits number
	MAXWORKERBITS = 15
	MAXNODEBITS   = 3
	// MAXSEQUENCEBITS is the maximum value for a 12 bits number
	MAXSEQUENCEBITS = 1023 // 4095
	// MAX41BITS is the maximum value for a 41 bits number
	MAX41BITS = 2199023255551

	IdPad       = 53
	NodePad     = 2
	WorkerPad   = 4
	SequencePad = 10
	TimePad     = 37 // 41
)

Variables

This section is empty.

Functions

func Decompose 

func Decompose(spaceflakeID, baseEpoch uint64) map[string]uint64

Decompose returns all the parts of the Spaceflake ID

func DecomposeBinary 

func DecomposeBinary(spaceflakeID, baseEpoch uint64) map[string]string

DecomposeBinary returns all the parts of the Spaceflake ID in binary format

func ParseNodeID 

func ParseNodeID(spaceflakeID uint64) uint64

ParseNodeID returns the node ID from the Spaceflake ID

func ParseSequence 

func ParseSequence(spaceflakeID uint64) uint64

ParseSequence returns the sequence number from the Spaceflake ID

func ParseTime 

func ParseTime(spaceflakeID, baseEpoch uint64) uint64

ParseTime returns the time in milliseconds since the base epoch from the Spaceflake ID

func ParseWorkerID 

func ParseWorkerID(spaceflakeID uint64) uint64

ParseWorkerID returns the worker ID from the Spaceflake ID

Types

type BulkGeneratorSettings 

type BulkGeneratorSettings struct {
	// Amount is the amount of Spaceflakes to generate
	Amount int
	// BaseEpoch is the base epoch that the Spaceflake generator will use
	BaseEpoch uint64
}

BulkGeneratorSettings is a struct that contains the settings for the bulk Spaceflake generator

func NewBulkGeneratorSettings 

func NewBulkGeneratorSettings(amount int) BulkGeneratorSettings

NewBulkGeneratorSettings is used for the settings of the BulkGenerate function below

type GeneratorSettings 

type GeneratorSettings struct {
	// BaseEpoch is the epoch that the Spaceflake generator will use for the first 41 bits
	BaseEpoch uint64
	// NodeID is the node ID that the Spaceflake generator will use for the next 5 bits
	NodeID uint64
	// Sequence is the last 12 bits, usually an incremented number but can be anything. If set to 0, it will be random.
	Sequence uint64
	// WorkerID is the worker ID that the Spaceflake generator will use for the next 5 bits
	WorkerID uint64
}

GeneratorSettings is a struct that contains the settings for the Spaceflake generator

func NewGeneratorSettings 

func NewGeneratorSettings() GeneratorSettings

NewGeneratorSettings is used for the settings of the Generate function below

type Node 

type Node struct {

	// ID is the ID of the node
	ID uint64
	// contains filtered or unexported fields
}

Node is a node in the Spaceflake Network that can hold workers

func NewNode 

func NewNode(nodeID uint64) *Node

NewNode creates a new node in the Spaceflake Network

func (*Node) BulkGenerateSpaceflakes 

func (n *Node) BulkGenerateSpaceflakes(amount int) ([]*Spaceflake, error)

BulkGenerateSpaceflakes will generate the amount of Spaceflakes specified and auto scale the worker IDs

func (*Node) GetWorkers 

func (n *Node) GetWorkers() []*Worker

GetWorkers returns the list of workers in the node

func (*Node) NewWorker 

func (n *Node) NewWorker() *Worker

NewWorker creates a new worker in the node

func (*Node) RemoveWorker 

func (n *Node) RemoveWorker(w *Worker)

RemoveWorker removes a worker from the node

type Spaceflake 

type Spaceflake struct {
	// contains filtered or unexported fields
}

Spaceflake represents a Spaceflake

func BulkGenerate 

func BulkGenerate(s BulkGeneratorSettings) ([]*Spaceflake, error)

Bulkgenerate will generate the amount of Spaceflakes specified and auto scale the node and worker IDs

func Generate 

func Generate(s GeneratorSettings) (*Spaceflake, error)

Generate only a Spaceflake ID when you need to generate one without worker and node objects. Fastest way of creating a Spaceflake.

func GenerateAt 

func GenerateAt(s GeneratorSettings, at time.Time) (*Spaceflake, error)

GenerateAt generates a Spaceflake at a specific time

func (*Spaceflake) BinaryID 

func (s *Spaceflake) BinaryID() string

BinaryID returns the Spaceflake ID in binary format

func (*Spaceflake) Decompose 

func (s *Spaceflake) Decompose() map[string]uint64

Decompose returns all the parts of the Spaceflake

func (*Spaceflake) DecomposeBinary 

func (s *Spaceflake) DecomposeBinary() map[string]string

DecomposeBinary returns all the parts of the Spaceflake in binary format

func (*Spaceflake) ID 

func (s *Spaceflake) ID() uint64

ID returns the Spaceflake ID

func (*Spaceflake) NodeID 

func (s *Spaceflake) NodeID() uint64

NodeID returns the node ID from the Spaceflake

func (*Spaceflake) Sequence 

func (s *Spaceflake) Sequence() uint64

Sequence returns the sequence number from the Spaceflake

func (*Spaceflake) StringID 

func (s *Spaceflake) StringID() string

StringID returns the Spaceflake ID as a string

func (*Spaceflake) Time 

func (s *Spaceflake) Time() uint64

Time returns the time in milliseconds since the base epoch from the Spaceflake

func (*Spaceflake) WorkerID 

func (s *Spaceflake) WorkerID() uint64

WorkerID returns the worker ID from the Spaceflake

type Worker 

type Worker struct {
	// Node is the node that the worker belongs to
	Node *Node

	// BaseEpoch is the epoch that will be used for the first 41 bits when generating a Spaceflake
	BaseEpoch uint64
	// Sequence is the last 12 bits, usually an incremented number but can be anything. If set to 0, it will be the incremented number.
	Sequence uint64
	// ID is the worker ID that the Spaceflake generator will use for the next 5 bits
	ID uint64
	// contains filtered or unexported fields
}

Worker is a worker in the Spaceflake Network that can generate Spaceflakes

func (*Worker) BulkGenerateSpaceflakes 

func (w *Worker) BulkGenerateSpaceflakes(amount int) ([]*Spaceflake, error)

BulkGenerateSpaceflakes will generate the amount of Spaceflakes specified and scale nothing, it will wait one millisecond if the sequence is over 4095

func (*Worker) GenerateSpaceflake 

func (w *Worker) GenerateSpaceflake() (*Spaceflake, error)

GenerateSpaceflake generates a Spaceflake

func (*Worker) GenerateSpaceflakeAt 

func (w *Worker) GenerateSpaceflakeAt(at time.Time) (*Spaceflake, error)

GenerateSpaceflakeAt generates a Spaceflake at a specific time

Source Files

View all Source files

Directories

Path Synopsis
examples
bulk
generate
node_workers

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.