shard

README

shard

Package shard connects to multiple physical Redis instances, and emulates a single logical Redis instance. Clients are expected (but not required) to use their Redis keys as hash keys to select a Redis instance. The package maintains a pool of connections to each instance.

Usage

Simple usage with a single key.

s := shard.New(...)
defer s.Close()

key, value := "foo", "bar"
if err := s.With(key, func(c redis.Conn) error {
	_, err := c.Do("SET", key, value)
	return err
}); err != nil {
	log.Printf("Failure: %s", err)
}

Keys may be pre-hashed with the Index method, and connections used for pipelining.

m := map[int][]string{} // index: keys to INCR
for _, key := range keys {
	index := s.Index(key)
	m[index] = append(m[index], key)
}

wg := sync.WaitGroup{}
wg.Add(len(m))
for index, keys := range m {
	// Shards are safe for concurrent access.
	go s.WithIndex(index, func(c redis.Conn) error) {
		defer wg.Done()
		for _, key := range keys {
			if err := c.Send("INCR", key); err != nil {
				return err
			}
		}
		if err := c.Flush(); err != nil {
			return err
		}
		for _ = range keys {
			if _, err := c.Receive(); err != nil {
				return err
			}
		}
		return nil
	})
}
wg.Wait()

Documentation

Overview

Package shard performs key-based sharding over multiple Redis instances.

Index

• func FNV(s string) uint32
• func FNVa(s string) uint32
• func Murmur3(s string) uint32
• type Shards
  • func New(addresses []string, connectTimeout, readTimeout, writeTimeout time.Duration, ...) *Shards
  • func (s *Shards) Close() error
  • func (s *Shards) Index(key string) int
  • func (s *Shards) Size() int
  • func (s *Shards) With(key string, do func(redis.Conn) error) error
  • func (s *Shards) WithIndex(index int, do func(redis.Conn) error) error

Functions

func FNV

func FNV(s string) uint32

FNV implements the FNV-1 string hashing function. It can be passed to NewCluster.

func FNVa

func FNVa(s string) uint32

FNVa implements the FNV-1a string hashing function. It can be passed to NewCluster.

func Murmur3

func Murmur3(s string) uint32

Murmur3 implements the Murmur3 string hashing function. It can be passed to NewCluster.

https://github.com/reusee/mmh3

Types

type Shards

type Shards struct {
	// contains filtered or unexported fields
}

Shards maintains a connection pool for multiple Redis instances.

func New

func New(
	addresses []string,
	connectTimeout, readTimeout, writeTimeout time.Duration,
	maxConnectionsPerInstance int,
	hash func(string) uint32,
) *Shards

New creates and returns a new Shards object.

Addresses are host:port strings for each underlying Redis instance. The number and order of the addresses determines the hash slots, so be careful to make that deterministic.

Connect timeout is the timeout for establishing a connection to any underlying Redis instance. Read timeout is the timeout for reading a reply to a command via an established connection. Write timeout is the timeout for writing a command to an established connection.

Max connections per instance is the size of the connection pool for each Redis instance. Hash defines the hash function used by the With methods. Any function that takes a string and returns a uint32 may be used. Package shard ships with several options, including Murmur3, FNV, and FNVa.

func (*Shards) Close

func (s *Shards) Close() error

Close closes all available (idle) connections in the cluster. Close does not affect oustanding (in-use) connections.

func (*Shards) Index

func (s *Shards) Index(key string) int

Index returns a reference to the connection pool that will be used to satisfy any request for the given key. Pass that value to WithIndex.

func (*Shards) Size

func (s *Shards) Size() int

Size returns how many instances the shards sits over. Useful for ranging over with WithIndex.

func (*Shards) With

func (s *Shards) With(key string, do func(redis.Conn) error) error

With is a convenience function that combines Index and WithIndex, for simple/single Redis requests on a single key.

func (*Shards) WithIndex

func (s *Shards) WithIndex(index int, do func(redis.Conn) error) error

WithIndex selects a single Redis instance from the referenced connection pool, and then calls the given function with that connection. If the function returns a nil error, WithIndex returns the connection to the pool after it's used. Otherwise, WithIndex discards the connection.

WithIndex will return an error if it wasn't able to successfully retrieve a connection from the referenced connection pool, and will forward any error returned by the `do` function.