katsubushi

package module
v2.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 14, 2024 License: MIT Imports: 30 Imported by: 1

README

katsubushi

katsubushi(鰹節) is stand alone application to generate unique ID.

Example

$ telnet localhost 11212
Trying ::1...
Connected to localhost.
Escape character is '^]'.
GET new
VALUE new 0 20
8070450532247928832
END

Installation

Download from releases or build from source code.

$ go get github.com/kayac/go-katsubushi/v2
$ cd $GOPATH/github.com/kayac/go-katsubushi
make

Docker image

$ docker pull katsubushi/katsubushi
$ docker run -p 11212:11212 katsubushi/katsubushi -worker-id 1
$ docker run -p 11212:11212 katsubushi/katsubushi -redis redis://your.redis.host:6379/0

Usage

$ cd $GOPATH/github.com/kayac/go-katsubushi/cmd/katsubushi
./katsubushi -worker-id=1 -port=7238
./katsubushi -worker-id=1 -sock=/path/to/unix-domain.sock

Protocol

katsubushi use protocol compatible with memcached.

Some commands are available with text and binary protocol.

But the others are available only with text protocol.

API
GET, GETS

Binary protocol is also available only for single key GET.

GET id1 id2
VALUE id1 0 18
283890203179880448
VALUE id2 0 18
283890203179880449
END

VALUE(s) are unique IDs.

STATS

Returns a stats of katsubushi.

Binary protocol is also available.

STAT pid 8018
STAT uptime 17
STAT time 1487754986
STAT version 1.1.2
STAT curr_connections 1
STAT total_connections 2
STAT cmd_get 2
STAT get_hits 3
STAT get_misses 0
VERSION

Returns a version of katsubushi.

Binary protocol is available, too.

VERSION 1.1.2
QUIT

Disconnect an established connection.

Protocol (HTTP)

katsubushi also runs an HTTP server specified with -http-port.

GET /id

Get a single ID.

When Accept HTTP header is 'application/json', katsubushi will return an ID as JSON format as below.

{"id":"1025441401866821632"}

Otherwise, katsubushi will return ID as text format.

1025441401866821632
GET /ids?n=(number_of_ids)

Get multiple IDs.

When Accept HTTP header is 'application/json', katsubushi will return an IDs as JSON format as below.

{"ids":["1025442579472195584","1025442579472195585","1025442579472195586"]}

Otherwise, katsubushi will return ID as text format delimiterd with "\n".

1025442579472195584
1025442579472195585
1025442579472195586
GET /stats

Returns a stats of katsubushi.

This API returns a JSON always.

{
  "pid": 1859630,
  "uptime": 50,
  "time": 1664761614,
  "version": "1.8.0",
  "curr_connections": 1,
  "total_connections": 5,
  "cmd_get": 15,
  "get_hits": 25,
  "get_misses": 0
}

Protocol (gRPC)

katsubushi also runs an gRPC server specified with -grpc-port.

See grpc/README.md.

Algorithm

katsubushi use algorithm like snowflake to generate ID.

Commandline Options

-worker-id or -redis is required.

-worker-id

ID of the worker, must be unique in your service.

-redis

URL of Redis server. e.g. redis://example.com:6379/0

redis://{host}:{port}/{db}?ns={namespace}

If you are using Redis Cluster, you will need to specify the URL as rediscluster://{host}:{port}?ns={namespace}.

This option is specified, katsubushi will assign an unique worker ID via Redis.

All katsubushi process for your service must use a same Redis URL.

-min-worker-id -max-worker-id

These options work with -redis.

If we use multi katsubushi clusters, worker-id range for each clusters must not be overlapped. katsubushi can specify the worker-id range by these options.

-port

Optional. Port number used for connection. Default value is 11212.

-sock

Optional. Path of unix doamin socket.

-idle-timeout

Optional. Connection idle timeout in seconds. 0 means infinite. Default value is 600.

-log-level

Optional. Default value is info.

-enable-pprof

Optional. Boolean flag. Enable profiling API by net/http/pprof. Endpoint is /debug/pprof.

-enable-stats

Optional. Boolean flag. Enable stats API by github.com/fukata/golang-stats-api-handler. Endpoint is /debug/stats.

-debug-port

Optional. Port number for listen http used for pprof and stats API. Defalut value is 8080.

-http-port

Optional. Port number of HTTP server. Default value is 0 (disabled).

-grpc-port

Optional. Port number of gRPC server. Default value is 0 (disabled).

Licence

MIT

Author

handlename

Documentation

Index

Constants

View Source 
const (
	WorkerIDBits = 10
	SequenceBits = 12
)

for bitshift

View Source 
const (
	MaxGRPCBulkSize = 1000
)
View Source 
const (
	MaxHTTPBulkSize = 1000
)

Variables

View Source 
var (

	// DefaultIdleTimeout is the default idle timeout.
	DefaultIdleTimeout = 600 * time.Second

	// InfiniteIdleTimeout means that idle timeout is disabled.
	InfiniteIdleTimeout = time.Duration(0)
)
View Source 
var (
	ErrInvalidWorkerID    = errors.New("invalid worker id")
	ErrDuplicatedWorkerID = errors.New("duplicated worker")
)

errors

View Source 
var DefaultClientTimeout = 5 * time.Second

DefaultClientTimeout is default timeout for katsubushi client

View Source 
var Epoch = time.Date(2015, 1, 1, 0, 0, 0, 0, time.UTC)

Epoch is katsubushi epoch time (2015-01-01 00:00:00 UTC) Generated ID includes elapsed time from Epoch.

View Source 
var (
	// Version number
	Version = "development"
)

Functions

func Dump 

func Dump(id uint64) (t time.Time, workerID uint64, sequence uint64)

Dump returns the structure of id.

func SetLogLevel 

func SetLogLevel(str string) error

SetLogLevel sets log level. Log level must be one of debug, info, warning, error, fatal and panic.

func StdLogger 

func StdLogger() *stdlog.Logger

StdLogger returns the standard logger.

func ToID 

func ToID(t time.Time) uint64

ToID returns the minimum id which will be generated at time t.

func ToTime 

func ToTime(id uint64) time.Time

ToTime returns the time when id was generated.

Types

type App 

type App struct {
	Listener net.Listener
	// contains filtered or unexported fields
}

App is main struct of the Application.

func New 

func New(workerID uint) (*App, error)

New create and returns new App instance.

func NewAppWithGenerator 

func NewAppWithGenerator(gen Generator, workerID uint) (*App, error)

NewAppWithGenerator create and returns new App instance with specified Generator.

func (*App) BytesToBinaryCmd 

func (app *App) BytesToBinaryCmd(req bRequest) (cmd MemdCmd, err error)

BytesToCmd converts byte array to a MemdBCmd and returns it.

func (*App) BytesToCmd 

func (app *App) BytesToCmd(data []byte) (cmd MemdCmd, err error)

BytesToCmd converts byte array to a MemdCmd and returns it.

func (*App) GetStats 

func (app *App) GetStats() MemdStats

GetStats returns MemdStats of app

func (*App) HTTPGetMultiID 

func (app *App) HTTPGetMultiID(w http.ResponseWriter, req *http.Request)

func (*App) HTTPGetSingleID 

func (app *App) HTTPGetSingleID(w http.ResponseWriter, req *http.Request)

func (*App) HTTPGetStats 

func (app *App) HTTPGetStats(w http.ResponseWriter, req *http.Request)

func (*App) IsBinaryProtocol 

func (app *App) IsBinaryProtocol(r *bufio.Reader) (bool, error)

IsBinaryProtocol judges whether a protocol is binary or text

func (*App) ListenerSock 

func (app *App) ListenerSock(sockpath string) (net.Listener, error)

ListenerSock starts listen Unix Domain Socket on sockpath.

func (*App) ListenerTCP 

func (app *App) ListenerTCP(addr string) (net.Listener, error)

ListenerTCP starts listen on host:port.

func (*App) NextID 

func (app *App) NextID() (uint64, error)

NextID generates new ID.

func (*App) Ready 

func (app *App) Ready() chan interface{}

Ready returns a channel which become readable when the app can accept connections.

func (*App) RespondToBinary 

func (app *App) RespondToBinary(r io.Reader, conn net.Conn)

RespondToBinary responds to a binary request with a binary response. A request should be read from r, not conn. Because the request reader might be buffered.

func (*App) RunGRPCServer 

func (app *App) RunGRPCServer(ctx context.Context, cfg *Config) error

func (*App) RunHTTPServer 

func (app *App) RunHTTPServer(ctx context.Context, cfg *Config) error

func (*App) RunServer 

func (app *App) RunServer(ctx context.Context, kc *Config) error

func (*App) Serve 

func (app *App) Serve(ctx context.Context, l net.Listener) error

Serve starts a server.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is katsubushi client

func NewClient 

func NewClient(addrs ...string) *Client

NewClient creates Client

func (*Client) Fetch 

func (c *Client) Fetch(ctx context.Context) (uint64, error)

Fetch fetches id from katsubushi

func (*Client) FetchMulti 

func (c *Client) FetchMulti(ctx context.Context, n int) ([]uint64, error)

FetchMulti fetches multiple ids from katsubushi

func (*Client) SetTimeout 

func (c *Client) SetTimeout(t time.Duration)

SetTimeout sets timeout to katsubushi servers

type Config 

type Config struct {
	IdleTimeout time.Duration
	LogLevel    string

	Port     int
	Sockpath string

	HTTPPort       int
	HTTPPathPrefix string
	HTTPListener   net.Listener

	GRPCPort     int
	GRPCListener net.Listener
}

type Generator 

type Generator interface {
	NextID() (uint64, error)
	WorkerID() uint
}

Generator is an interface to generate unique ID.

func NewGenerator 

func NewGenerator(workerID uint) (Generator, error)

NewGenerator returns new generator.

type HTTPClient 

type HTTPClient struct {
	// contains filtered or unexported fields
}

func NewHTTPClient 

func NewHTTPClient(urls []string, pathPrefix string) (*HTTPClient, error)

NewHTTPClient creates HTTPClient

func (*HTTPClient) Fetch 

func (c *HTTPClient) Fetch(ctx context.Context) (uint64, error)

Fetch fetches id from katsubushi via HTTP

func (*HTTPClient) FetchMulti 

func (c *HTTPClient) FetchMulti(ctx context.Context, n int) ([]uint64, error)

FetchMulti fetches multiple ids from katsubushi via HTTP

func (*HTTPClient) SetTimeout 

func (c *HTTPClient) SetTimeout(t time.Duration)

SetTimeout sets timeout to katsubushi servers

type MemdBCmdGet 

type MemdBCmdGet struct {
	Name   string
	Key    string
	Opaque [4]byte
}

MemdCmdGet defines binary Get command.

func (*MemdBCmdGet) Execute 

func (cmd *MemdBCmdGet) Execute(app *App, w io.Writer) error

Execute generates new ID.

type MemdBCmdStat 

type MemdBCmdStat struct {
	Key    string
	Opaque [4]byte
}

MemdCmdStat defines binary Stat command.

func (*MemdBCmdStat) Execute 

func (cmd *MemdBCmdStat) Execute(app *App, w io.Writer) error

Execute writes binary stat ref. https://github.com/memcached/memcached/wiki/BinaryProtocolRevamped#stat

type MemdBCmdVersion 

type MemdBCmdVersion struct {
	Opaque [4]byte
}

MemdBCmdVersion defines binary VERSION command.

func (MemdBCmdVersion) Execute 

func (cmd MemdBCmdVersion) Execute(app *App, w io.Writer) error

Execute writes binary Version number.

type MemdCmd 

type MemdCmd interface {
	Execute(*App, io.Writer) error
}

MemdCmd defines a command.

type MemdCmdGet 

type MemdCmdGet struct {
	Name string
	Keys []string
}

MemdCmdGet defines Get command.

func (*MemdCmdGet) Execute 

func (cmd *MemdCmdGet) Execute(app *App, conn io.Writer) error

Execute generates new ID.

type MemdCmdQuit 

type MemdCmdQuit int

MemdCmdQuit defines QUIT command.

func (MemdCmdQuit) Execute 

func (cmd MemdCmdQuit) Execute(app *App, conn io.Writer) error

Execute disconnect by server.

type MemdCmdStats 

type MemdCmdStats int

MemdCmdStats defines STATS command.

func (MemdCmdStats) Execute 

func (cmd MemdCmdStats) Execute(app *App, conn io.Writer) error

Execute writes STATS response.

type MemdCmdVersion 

type MemdCmdVersion int

MemdCmdVersion defines VERSION command.

func (MemdCmdVersion) Execute 

func (cmd MemdCmdVersion) Execute(app *App, w io.Writer) error

Execute writes Version number.

type MemdStats 

type MemdStats struct {
	Pid              int    `memd:"pid" json:"pid"`
	Uptime           int64  `memd:"uptime" json:"uptime"`
	Time             int64  `memd:"time" json:"time"`
	Version          string `memd:"version" json:"version"`
	CurrConnections  int64  `memd:"curr_connections" json:"curr_connections"`
	TotalConnections int64  `memd:"total_connections" json:"total_connections"`
	CmdGet           int64  `memd:"cmd_get" json:"cmd_get"`
	GetHits          int64  `memd:"get_hits" json:"get_hits"`
	GetMisses        int64  `memd:"get_misses" json:"get_misses"`
}

MemdStats defines result of STATS command.

func (MemdStats) WriteTo 

func (s MemdStats) WriteTo(w io.Writer) (int64, error)

WriteTo writes result of STATS command to io.Writer.

type MemdValue 

type MemdValue struct {
	Keys   []string
	Flags  int
	Values []string
}

MemdValue defines return value for client.

func (MemdValue) WriteTo 

func (v MemdValue) WriteTo(w io.Writer) (int64, error)

WriteTo writes content of MemdValue to io.Writer. Its format is compatible to memcached protocol.

Source Files

View all Source files

Directories

Path Synopsis
cmd
katsubushi
katsubushi-dump

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.