snowflake

README

❄️ Go-Snowflake

A Snowflake Generator for Go

A simple to use Go (golang) package to generate or parse Twitter snowflake IDs

Snowflake简介

在单机系统中我们会使用自增id作为数据的唯一id，自增id在数据库中有利于排序和索引，但是在分布式系统中如果还是利用数据库的自增id会引起冲突，自增id非常容易被爬虫爬取数据。在分布式系统中有使用uuid作为数据唯一id的，但是uuid是一串随机字符串，所以它无法被排序。

Twitter设计了Snowflake算法为分布式系统生成ID,Snowflake的id是int64类型，它通过datacenterId和workerId来标识分布式系统，下面看下它的组成：

1bit	41bit	5bit	5bit	12bit
符号位（保留字段）	时间戳(当前时间-纪元时间)	数据中心id	机器id	自增序列

算法简介

在使用Snowflake生成id时，首先会计算时间戳timestamp（当前时间 - 纪元时间），如果timestamp数据超过41bit则异常。同样需要判断datacenterId和workerId不能超过5bit(0-31)，在处理自增序列时，如果发现自增序列超过12bit时需要等待，因为当前毫秒下12bit的自增序列被用尽，需要进入下一毫秒后自增序列继续从0开始递增。

---

🚀 快速开始

🕹 克隆 & 运行

git clone https://github.com/houseme/snowflake.git

go run ./.example/main.go

💾 安装 & 导入

go get github.com/houseme/snowflake

// 在项目中导入模块
import "github.com/houseme/snowflake"

⚠️注意事项

• 在多实例（多个snowflake对象）的并发环境下，请确保每个实例（datacenterId，workerId）的唯一性，否则生成的ID可能冲突。

📊 测试

本机测试：

参数	配置
OS	MacBook Pro (16-inch, 2019)
CPU	Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
RAM	64 GB 2667 MHz DDR4

测试代码

func TestLoad() {
    var wg sync.WaitGroup
    s, err := snowflake.NewSnowflake(int64(0), int64(0))
    if err != nil {
        glog.Error(err)
        return
    }
    var check sync.Map
    t1 := time.Now()
    for i := 0; i < 200000; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            val := s.NextVal()
            if _, ok := check.Load(val); ok {
                // id冲突检查
                glog.Error(fmt.Errorf("error#unique: val:%v", val))
                return
            }
            check.Store(val, 0)
            if val == 0 {
                glog.Error(fmt.Errorf("error"))
                return
            }
        }()
    }
    wg.Wait()
    elapsed := time.Since(t1)
    glog.Infof("generate 20k ids elapsed: %v", elapsed)
}

运行结果

🗂 使用说明

创建Snowflake对象

// NewSnowflake(datacenterId, workerId int64) (*Snowflake, error)
// 参数1 (int64): 数据中心ID (可用范围:0-31)
// 参数2 (int64): 机器ID    (可用范围:0-31)
// 返回1 (*Snowflake): Snowflake对象 | nil
// 返回2 (error): 错误码
s, err := snowflake.NewSnowflake(int64(0), int64(0))
if err != nil {
    glog.Error(err)
    return
}

生成唯一ID

s, err := snowflake.NewSnowflake(int64(0), int64(0))
// ......
// (s *Snowflake) NextVal() int64
// 返回1 (int64): 唯一ID
id := s.NextVal()
// ......

通过ID获取数据中心ID与机器ID

// ......
// GetDeviceID(sid int64) (datacenterId, workerId int64)
// 参数1 (int64): 唯一ID
// 返回1 (int64): 数据中心ID
// 返回2 (int64): 机器ID
datacenterid, workerid := snowflake.GetDeviceID(id))

通过ID获取时间戳（创建ID时的时间戳 - epoch）

// ......
// GetTimestamp(sid int64) (timestamp int64)
// 参数1 (int64): 唯一ID
// 返回1 (int64): 从epoch开始计算的时间戳
t := snowflake.GetTimestamp(id)

通过ID获取生成ID时的时间戳

// ......
// GetGenTimestamp(sid int64) (timestamp int64)
// 参数1 (int64): 唯一ID
// 返回1 (int64): 唯一ID生成时的时间戳
t := snowflake.GetGenTimestamp(id)

通过ID获取生成ID时的时间（精确到：秒）

// ......
// GetGenTime(sid int64)
// 参数1 (int64): 唯一ID
// 返回1 (string): 唯一ID生成时的时间
tStr := snowflake.GetGenTime(id)

查看时间戳字段使用占比（41bit能存储的范围：从epoch开始往后69年）

// ......
// GetTimestampStatus() (state float64)
// 返回1 (float64): 时间戳字段使用占比（范围 0.0 - 1.0）
status := snowflake.GetTimestampStatus()

Performance

With default settings, this snowflake generator should be sufficiently fast enough on most systems to generate 4096 unique ID's per millisecond. This is the maximum that the snowflake ID format supports. That is, around 243-244 nanoseconds per operation.

Since the snowflake generator is single threaded the primary limitation will be the maximum speed of a single processor on your system.

To benchmark the generator on your system run the following command inside the snowflake package directory.

go test -run=^$ -bench=.

License

Go-snowflake is primarily distributed under the terms of both the Apache License (Version 2.0), thanks for GUAIK-ORG and Bwmarrin.

Documentation

Index

• Variables
• func GetDatacenterIDMax() int64
• func GetDeviceID(sid int64) (datacenterID, workerID int64)
• func GetGenTime(sid ID) (t string)
• func GetGenTimestamp(sid ID) (timestamp int64)
• func GetSequenceMask() int64
• func GetTimestamp(sid ID) (timestamp int64)
• func GetTimestampMax() int64
• func GetTimestampStatus() (state float64)
• func GetWorkerIDMax() int64
• type ID
  • func ParseBase2(id string) (ID, error)
  • func ParseBase32(b []byte) (ID, error)
  • func ParseBase36(id string) (ID, error)
  • func ParseBase58(b []byte) (ID, error)
  • func ParseBase64(id string) (ID, error)
  • func ParseBytes(id []byte) (ID, error)
  • func ParseInt64(id int64) ID
  • func ParseIntBytes(id [8]byte) ID
  • func ParseString(sid string) (ID, error)
  • func (sid ID) Base2() string
  • func (sid ID) Base32() string
  • func (sid ID) Base36() string
  • func (sid ID) Base58() string
  • func (sid ID) Base64() string
  • func (sid ID) Bytes() []byte
  • func (sid ID) Int64() int64
  • func (sid ID) IntBytes() [8]byte
  • func (sid ID) MarshalJSON() ([]byte, error)
  • func (sid ID) String() string
  • func (sid ID) Time() int64
  • func (sid *ID) UnmarshalJSON(b []byte) error
• type JSONSyntaxError
  • func (j JSONSyntaxError) Error() string
• type Snowflake
  • func NewSnowflake(datacenterID, workerID int64) (*Snowflake, error)
  • func (s *Snowflake) NextVal() ID

Variables

var ErrInvalidBase32 = errors.New("invalid base32")

ErrInvalidBase32 is returned by ParseBase32 when given an invalid []byte

var ErrInvalidBase58 = errors.New("invalid base58")

ErrInvalidBase58 is returned by ParseBase58 when given an invalid []byte

Functions

func GetDatacenterIDMax

func GetDatacenterIDMax() int64

GetDatacenterIDMax Maximum number of data center id supported.

func GetDeviceID

func GetDeviceID(sid int64) (datacenterID, workerID int64)

GetDeviceID returns an int64 of the snowflake center ID and machine ID number

func GetGenTime

func GetGenTime(sid ID) (t string)

GetGenTime returns Gets the time string when the ID was created (precision: seconds)

func GetGenTimestamp

func GetGenTimestamp(sid ID) (timestamp int64)

GetGenTimestamp returns Get the timestamp when the ID was created

func GetSequenceMask

func GetSequenceMask() int64

GetSequenceMask Maximum number of sequence id supported

func GetTimestamp

func GetTimestamp(sid ID) (timestamp int64)

GetTimestamp returns an int64 unix timestamp in milliseconds of the snowflake ID time

func GetTimestampMax

func GetTimestampMax() int64

GetTimestampMax Timestamp maximum

func GetTimestampStatus

func GetTimestampStatus() (state float64)

GetTimestampStatus returns an float64 unix timestamp in milliseconds of the snowflake ID time Get the percentage of timestamps used: range (0.0-1.0)

func GetWorkerIDMax

func GetWorkerIDMax() int64

GetWorkerIDMax Maximum number of machine id supported

Types

type ID

type ID int64

An ID is a custom type used for a snowflake ID. This is used so we can attach methods onto the ID.

func ParseBase2

func ParseBase2(id string) (ID, error)

ParseBase2 converts a Base2 string into a snowflake ID

func ParseBase32

func ParseBase32(b []byte) (ID, error)

ParseBase32 parses a base32 []byte into a snowflake ID NOTE: There are many different base32 implementations so becareful when doing any interoperation.

func ParseBase36

func ParseBase36(id string) (ID, error)

ParseBase36 converts a Base36 string into a snowflake ID

func ParseBase58

func ParseBase58(b []byte) (ID, error)

ParseBase58 parses a base58 []byte into a snowflake ID

func ParseBase64

func ParseBase64(id string) (ID, error)

ParseBase64 converts a base64 string into a snowflake ID

func ParseBytes

func ParseBytes(id []byte) (ID, error)

ParseBytes converts a byte slice into a snowflake ID

func ParseInt64

func ParseInt64(id int64) ID

ParseInt64 converts an int64 into a snowflake ID

func ParseIntBytes

func ParseIntBytes(id [8]byte) ID

ParseIntBytes converts an array of bytes encoded as big endian integer as a snowflake ID

func ParseString

func ParseString(sid string) (ID, error)

ParseString converts a string into a snowflake ID

func (ID) Base2

func (sid ID) Base2() string

Base2 returns a string base2 of the snowflake ID

func (ID) Base32

func (sid ID) Base32() string

Base32 uses the z-base-32 character set but encodes and decodes similar to base58, allowing it to create an even smaller result string. NOTE: There are many different base32 implementations so becareful when doing any interoperation.

func (ID) Base36

func (sid ID) Base36() string

Base36 returns a base36 string of the snowflake ID

func (ID) Base58

func (sid ID) Base58() string

Base58 returns a base58 string of the snowflake ID

func (ID) Base64

func (sid ID) Base64() string

Base64 returns a base64 string of the snowflake ID

func (ID) Bytes

func (sid ID) Bytes() []byte

Bytes return a byte slice of the snowflake ID

func (ID) Int64

func (sid ID) Int64() int64

Int64 returns an int64 of the snowflake ID

func (ID) IntBytes

func (sid ID) IntBytes() [8]byte

IntBytes returns an array of bytes of the snowflake ID, encoded as a big endian integer.

func (ID) MarshalJSON

func (sid ID) MarshalJSON() ([]byte, error)

MarshalJSON returns a json byte array string of the snowflake ID.

func (ID) String

func (sid ID) String() string

String returns a string of the snowflake ID

func (ID) Time

func (sid ID) Time() int64

Time returns an int64 unix timestamp in milliseconds of the snowflake ID time

func (*ID) UnmarshalJSON

func (sid *ID) UnmarshalJSON(b []byte) error

UnmarshalJSON converts a json byte array of a snowflake ID into an ID type.

type JSONSyntaxError

type JSONSyntaxError struct {
	// contains filtered or unexported fields
}

A JSONSyntaxError is returned from UnmarshalJSON if an invalid ID is provided.

func (JSONSyntaxError) Error

func (j JSONSyntaxError) Error() string

type Snowflake

type Snowflake struct {
	sync.Mutex
	// contains filtered or unexported fields
}

Snowflake is a custom type

func NewSnowflake

func NewSnowflake(datacenterID, workerID int64) (*Snowflake, error)

NewSnowflake returns a new snowflake node that can be used to generate snowflake

func (*Snowflake) NextVal

func (s *Snowflake) NextVal() ID

NextVal creates and returns a unique snowflake ID To help guarantee uniqueness - Make sure your system is keeping accurate system time - Make sure you never have multiple nodes running with the same node ID