ksuid

README

ksuid

ksuid is a Go library that generated prefixed, k-sorted globally unique identifiers.

Each KSUID has a resource type and optionally an environment prefix (no environment prefix is for prod use only). They are roughly sortable down to per-second resolution.

Properties of a KSUID:

• resource type and environment prefixing
• lexicographically, time sortable
• no startup co-ordination
• guaranteed unique relative to process/machine

Usage

API

ksuid is primarily a Go package to be consumed by Cuvva services, below are examples of its API usage.

To generate a KSUID with a custom resource type and for the prod environment:

id := ksuid.Generate("user")
/* => ID{
	Environment: "prod",
	Resource: "user",
	Timestamp: time.Time{"2021-04-29T10:46:56Z"},
	MachineID: net.HardwareAddr{"1e:00:a2:3e:53:90"},
	ProcessID: 21124,
	SequenceID: 0,
} */

To parse a single given KSUID:

id, err := ksuid.Parse([]byte("user_000000C8BhY47NRDD94E5VZxVX4bo"))
/*
=> ID{
	Environment: "prod",
	Resource: "user",
	Timestamp: time.Time{"2021-04-29T10:46:56Z"},
	MachineID: net.HardwareAddr{"1e:00:a2:3e:53:90"},
	ProcessID: 21124,
	SequenceID: 0,
}, nil
*/

CLI

ksuid provides a helper utility to generate and parse KSUID on the command line, it contains two subcommands: parse and generate.

To generate two KSUID with a custom resource type and for the prod environment:

$ ksuid generate --resource=user --count=2
user_000000C8BhY47NRDD94E5VZxVX4bo
user_000000C8BhY47NRDD94E5VZxVX4bp

To parse a single given KSUID:

$ ksuid parse user_000000C8BhY47NRDD94E5VZxVX4bo
ID:          user_000000C8BhY47NRDD94E5VZxVX4bo
Resource:    user
Environment: prod
Timestamp:   2021-04-29T10:46:56Z
Machine ID:  1e:00:a2:3e:53:90
Process ID:  21124
Sequence ID: 0

Structure

Excluding the resource & environment prefix parts, KSUIDs are 29 bytes long when Base62 encoded, consisting of 21 bytes decoded:

• first 8 bytes: a 64-bit unix timestamp
• next 9 bytes: a 64-bit instance ID, prefixed by an 8-bit scheme
• next 4 bytes: a 32-bit incrementing counter, reset every second

Optionally a KSUID has two, underscore delimited prefixes. The first prefix is optional, and is the environment in which the KSUID was generated (test, dev, git commit etc), omitting the environment identifies prod only. The second prefix is the resource type (user, profile, vehicle etc) and is required.

Instance IDs

The instance ID is structured differently depending on the source environment - allowing the best choice for the given use-case.

The first byte indicates which kind of instance ID, which then defines the structure of the remaining 8 bytes:

• 0x44 (ASCII D): Docker
  • 8 bytes: truncated Docker container ID
• 0x48 (ASCII H): hardware
  • first 6 bytes: the 48-bit MAC address
  • next 2 bytes: the 16-bit process ID (truncated if necessary)
• 0x52 (ASCII R): random
  • 8 bytes: randomly generated bytes

The random option should only be used if no reliable instance ID is available.

Documentation

Index

• Constants
• func SetInstanceID(instanceID InstanceID)
• type ID
  • func Generate(ctx context.Context, resource string) ID
  • func MustParse(src string) ID
  • func Parse(str string) (id ID, err error)
  • func (id ID) Bytes() []byte
  • func (id ID) Equal(x ID) bool
  • func (id ID) IsZero() bool
  • func (id ID) MarshalBSONValue() (bsontype.Type, []byte, error)
  • func (id ID) MarshalJSON() ([]byte, error)
  • func (id *ID) Scan(src interface{}) error
  • func (id ID) String() string
  • func (id *ID) UnmarshalBSONValue(t bsontype.Type, raw []byte) (err error)
  • func (id *ID) UnmarshalJSON(b []byte) (err error)
  • func (id ID) Value() (driver.Value, error)
• type InstanceID
  • func NewDockerID() (InstanceID, error)
  • func NewHardwareID() (InstanceID, error)
  • func NewRandomID() InstanceID
  • func (i InstanceID) Bytes() [8]byte
  • func (i InstanceID) Scheme() byte
• type Iterator
  • func (i *Iterator) Next() (more bool)
  • func (i *Iterator) Value() ID
• type Node
  • func NewNode(environment string, instanceID InstanceID) *Node
  • func (n *Node) Generate(ctx context.Context, resource string) (id ID)
• type ParseError
  • func (pe ParseError) Error() string
• type Set
  • func NewSet(x ...ID) *Set
  • func (s *Set) Append(id ID) (appended bool)
  • func (s *Set) Delete(id ID)
  • func (s *Set) Exists(id ID) (found bool)
  • func (s *Set) Iter() *Iterator
  • func (s *Set) Len() int

Constants

const Production = "prod"

Production is the internal name for production ksuid, but is omitted during marshaling.

Functions

func SetInstanceID

func SetInstanceID(instanceID InstanceID)

SetInstanceID overrides the default instance id in the exported node. This will effect all invocations of the Generate function.

Types

type ID

type ID struct {
	Environment string
	Resource    string

	Timestamp  uint64
	InstanceID InstanceID
	SequenceID uint32
}

ID is an optionally prefixed, k-sortable globally unique ID.

func Generate

func Generate(ctx context.Context, resource string) ID

Generate returns a new ID for the current machine and resource configured.

func MustParse

func MustParse(src string) ID

MustParse unmarshals an ID from a string and panics on error.

func Parse

func Parse(str string) (id ID, err error)

Parse unmarshals an ID from a series of bytes.

func (ID) Bytes

func (id ID) Bytes() []byte

Bytes stringifies and returns ID as a byte slice.

func (ID) Equal

func (id ID) Equal(x ID) bool

Equal returns true if the given ID matches id of the caller.

func (ID) IsZero

func (id ID) IsZero() bool

IsZero returns true if id has not yet been initialized.

func (ID) MarshalBSONValue

func (id ID) MarshalBSONValue() (bsontype.Type, []byte, error)

MarshalBSONValue implements bson.ValueMarshaler

func (ID) MarshalJSON

func (id ID) MarshalJSON() ([]byte, error)

MarshalJSON implements a custom JSON string marshaler.

func (*ID) Scan

func (id *ID) Scan(src interface{}) error

Scan implements a custom database/sql.Scanner to support unmarshaling from standard database drivers.

func (ID) String

func (id ID) String() string

String stringifies and returns ID as a string.

func (*ID) UnmarshalBSONValue

func (id *ID) UnmarshalBSONValue(t bsontype.Type, raw []byte) (err error)

UnmarshalBSONValue implements bson.ValueUnmarshaler

func (*ID) UnmarshalJSON

func (id *ID) UnmarshalJSON(b []byte) (err error)

UnmarshalJSON implements a custom JSON string unmarshaler.

func (ID) Value

func (id ID) Value() (driver.Value, error)

Value implements a custom database/sql/driver.Valuer to support marshaling to standard database drivers.

type InstanceID

type InstanceID struct {
	SchemeData byte
	BytesData  [8]byte
}

func NewDockerID

func NewDockerID() (InstanceID, error)

NewDockerID returns a DockerID for the current Docker container.

func NewHardwareID

func NewHardwareID() (InstanceID, error)

NewHardwareID returns a HardwareID for the current node.

func NewRandomID

func NewRandomID() InstanceID

NewRandomID returns a RandomID initialized by a PRNG.

func (InstanceID) Bytes

func (i InstanceID) Bytes() [8]byte

func (InstanceID) Scheme

func (i InstanceID) Scheme() byte

type Iterator

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator goes over every item currently in the set in a thread-safe way.

func (*Iterator) Next

func (i *Iterator) Next() (more bool)

Next returns true if there is at least one more KSUID in the set available for iteration.

func (*Iterator) Value

func (i *Iterator) Value() ID

Value returns the next iterated ID.

type Node

type Node struct {
	InstanceID InstanceID
	// contains filtered or unexported fields
}

Node contains metadata used for ksuid generation for a specific machine.

func NewNode

func NewNode(environment string, instanceID InstanceID) *Node

NewNode returns a ID generator for the current machine.

func (*Node) Generate

func (n *Node) Generate(ctx context.Context, resource string) (id ID)

Generate returns a new ID for the machine and resource configured.

type ParseError

type ParseError struct {
	// contains filtered or unexported fields
}

ParseError is returned when unexpected input is encountered when parsing user input to an ID.

func (ParseError) Error

func (pe ParseError) Error() string

type Set

type Set struct {
	// contains filtered or unexported fields
}

Set is a thread-safe, ordered array of KSUID where each item exists exactly once.

All Read/Write operations are O(1). Deletion is O(n).

func NewSet

func NewSet(x ...ID) *Set

NewSet initializes a KSUID set for unique sets of KSUID, optionally based on an initial array of KSUID.

func (*Set) Append

func (s *Set) Append(id ID) (appended bool)

Append inserts the given ID to the end of the set, if it does not already exist. Returns true if item is new to the set.

Complexity: O(1)

func (*Set) Delete

func (s *Set) Delete(id ID)

Delete removes ID if it exists within the set.

Complexity: O(n)

func (*Set) Exists

func (s *Set) Exists(id ID) (found bool)

Exists returns true if ID exists within the set.

Complexity: O(1)

func (*Set) Iter

func (s *Set) Iter() *Iterator

Iter returns a new Iterator for going over every item in a thread-safe manor.

func (*Set) Len

func (s *Set) Len() int

Len returns the total length of the Set.

Complexity: O(1)