reckon

README

Reckon

A Go package for sampling and reporting on random keys on a set of redis instances

Inspired/influenced by redis-sampler from antirez, the author of redis.

Background

We love redis here at zulily. We store millions of keys across many redis instances, and we've built our own internal distributed cache on top of it.

One problem with running a large, distributed cache using redis is the opaque nature of the keyspaces; it's hard to tell what the composition of your redis dataset is, especially when you've got multiple codebases or teams using the same redis instance(s), or you're sharding your dataset over a large number of redis instances.

While there are some existing solutions for sampling a redis keyspace, the reckon package has a few advantages:

Programmatic access to sampling results

Results are returned in data structures, not just printed to stdout or a file. This is what allows a user of reckon to sample data across a cluster of redis instances and merge the results to get an overall picture of the keyspaces. We've included some sample code to do just that, in the examples.

Aggregation

reckon also allows you to define arbitrary buckets based on the name of the sampled key and/or the redis data type (hash, set, list, etc.). During sampling, reckon compiles statistics about the various redis data types, and aggregates those statistics according to the buckets you defined.

Any type that implements the Aggregator interface can instruct reckon as to how to aggregate the redis keys that it samples. This is best illustrated with some simple, contrived examples:

To aggregate only redis sets whose keys start with the letter a:

func setsThatStartWithA(key string, valueType reckon.ValueType) []string {
  if strings.HasPrefix(key, "a") && valueType == reckon.TypeSet {
    return []string{"setsThatStartWithA"}
  }
  return []string{}
}

To aggregate sampled keys of any redis data type that are longer than 80 characters:

func longKeys(key string, valueType reckon.ValueType) []string {
if len(key) > 80 {
  return []string{"long-keys"}
  }
  return []string{}
}

Reports

When you are done sampling, aggregating, and/or combining the results produced by reckon you can easily produce a report of the findings in either plain-text or static HTML. An example HTML report is shown below:

Quick Start

Get the code:

$ go get github.com/zulily/reckon

Build the example binaries:

$ cd $GOPATH/src/github.com/zulily/reckon
$ go install -v ./...

Use one of the provided example binaries to sample from a redis instance and output results to static HTML files in the current directory:

$ reckoning-single-instance -host=localhost -port=6379 \
    -sample-rate=0.1 -min-samples=100

Or to sample from multiple instances:

$ reckoning-multiple-instances -sample-rate=0.1 \
    -redis=localhost:6379 \
    -redis=localhost:6380 \
    -redis=localhost:6381

Or, use the package in your own binary:

package main

import (
  "log"
  "os"

  "github.com/zulily/reckon"
)

func main() {

  opts := reckon.Options{
    Host:       "localhost",
    Port:       6379,
    MinSamples: 10000,
  }

  stats, keyCount, err := reckon.Run(opts, reckon.AggregatorFunc(reckon.AnyKey))
  if err != nil {
    panic(err)
  }

  log.Printf("total key count: %d\n", keyCount)
  for k, v := range stats {
    log.Printf("stats for: %s\n", k)

    v.Name = k
    if f, err := os.Create(fmt.Sprintf("output-%s.html", k)); err != nil {
      panic(err)
    } else {
      defer f.Close()
      log.Printf("Rendering totals for: '%s' to %s:\n", k, f.Name())
      if err := reckon.RenderHTML(v, f); err != nil {
        panic(err)
      }
    }
  }
}

Limitations

Since reckon makes use of redis' RANDOMKEY and INFO commands, it is not able to sample data via a twemproxy proxy, since twemproxy implements a subset of the redis protocol that does not include these commands.

However, instead of sampling through a proxy, you can easily run reckon against multiple redis instances, and merge the results. We include code that does just that in the examples.

Documentation

Overview

Package reckon provides support for sampling and reporting on the keys and values in one or more redis instances

Index

• Constants
• func AnyKey(key string, valueType ValueType) []string
• func Asset(name string) ([]byte, error)
• func AssetDir(name string) ([]string, error)
• func AssetInfo(name string) (os.FileInfo, error)
• func AssetNames() []string
• func ComputePowerOfTwoFreq(m map[int]int64) map[int]int64
• func MustAsset(name string) []byte
• func RenderHTML(s *Results, out io.Writer) error
• func RenderText(s *Results, out io.Writer) error
• func RestoreAsset(dir, name string) error
• func RestoreAssets(dir, name string) error
• func Run(opts Options, aggregator Aggregator) (map[string]*Results, int64, error)
• type Aggregator
• type AggregatorFunc
  • func (f AggregatorFunc) Groups(key string, valueType ValueType) []string
• type Options
• type Results
  • func NewResults() *Results
  • func (r *Results) Merge(other *Results)
• type Statistics
  • func ComputeStatistics(m map[int]int64) Statistics
  • func NewStatistics() *Statistics
• type ValueType

Constants

const (
	// MaxExampleKeys sets an upper bound on the number of example keys that will
	// be captured during sampling
	MaxExampleKeys = 10
	// MaxExampleElements sets an upper bound on the number of example elements that
	// will be captured during sampling
	MaxExampleElements = 10
	// MaxExampleValues sets an upper bound on the number of example values that
	// will be captured during sampling
	MaxExampleValues = 10
)

Functions

func AnyKey

func AnyKey(key string, valueType ValueType) []string

AnyKey is an AggregatorFunc that puts any sampled key (regardless of key name or redis data type) into a generic "any-key" bucket.

func Asset

func Asset(name string) ([]byte, error)

Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetDir

func AssetDir(name string) ([]string, error)

AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy:

data/
  foo.txt
  img/
    a.png
    b.png

then AssetDir("data") would return []string{"foo.txt", "img"} AssetDir("data/img") would return []string{"a.png", "b.png"} AssetDir("foo.txt") and AssetDir("notexist") would return an error AssetDir("") will return []string{"data"}.

func AssetInfo

func AssetInfo(name string) (os.FileInfo, error)

AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetNames

func AssetNames() []string

AssetNames returns the names of the assets.

func ComputePowerOfTwoFreq

func ComputePowerOfTwoFreq(m map[int]int64) map[int]int64

ComputePowerOfTwoFreq converts a frequency map into a new frequency map, where each map key is the smallest power of two that is greater than or equal to the original map key.

func MustAsset

func MustAsset(name string) []byte

MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

func RenderHTML

func RenderHTML(s *Results, out io.Writer) error

RenderHTML renders an HTML report for a Results instance to the supplied io.Writer

func RenderText

func RenderText(s *Results, out io.Writer) error

RenderText renders a plaintext report for a Results instance to the supplied io.Writer

func RestoreAsset

func RestoreAsset(dir, name string) error

Restore an asset under the given directory

func RestoreAssets

func RestoreAssets(dir, name string) error

Restore assets under the given directory recursively

func Run

func Run(opts Options, aggregator Aggregator) (map[string]*Results, int64, error)

Run performs the configured sampling operation against the redis instance, returning aggregated statistics using the provided Aggregator, as well as the actual key count for the redis instance. If any errors occur, the sampling is short-circuited, and the error is returned. In such a case, the results should be considered invalid.

Types

type Aggregator

type Aggregator interface {
	Groups(key string, valueType ValueType) []string
}

An Aggregator returns 0 or more arbitrary strings, to be used during a sampling operation as aggregation groups or "buckets". For example, an Aggregator that takes the first letter of the key would cause reckon to aggregate stats by each letter of the alphabet

type AggregatorFunc

type AggregatorFunc func(key string, valueType ValueType) []string

The AggregatorFunc type is an adapter to allow the use of ordinary functions as Aggregators. If f is a function with the appropriate signature, AggregatorFunc(f) is an Aggregator object that calls f.

func (AggregatorFunc) Groups

func (f AggregatorFunc) Groups(key string, valueType ValueType) []string

Groups provides 0 or more groups to aggregate `key` to, when sampling redis keys.

type Options

type Options struct {
	Host string
	Port int

	// MinSamples indicates the minimum number of random keys to sample from the redis
	// instance.  Note that this does not mean **unique** keys, just an absolute
	// number of random keys.  Therefore, this number should be small relative to
	// the number of keys in the redis instance.
	MinSamples int

	// SampleRate indicates the percentage of the keyspace to sample.
	// Accordingly, values should be between 0.0 and 1.0.  If a non-zero value is
	// given for both `SampleRate` and `MinSamples`, the actual number of keys
	// sampled will be the greater of the two values, once the key count has been
	// calculated using the `SampleRate`.
	SampleRate float32
}

Options is a configuration struct that instructs the reckon pkg to sample the redis instance listening on a particular host/port with a specified number/percentage of random keys.

type Results

type Results struct {
	Name     string
	KeyCount int64

	// Strings
	StringSizes  map[int]int64
	StringKeys   map[string]bool
	StringValues map[string]bool

	// Sets
	SetSizes        map[int]int64
	SetElementSizes map[int]int64
	SetKeys         map[string]bool
	SetElements     map[string]bool

	// Sorted Sets
	SortedSetSizes        map[int]int64
	SortedSetElementSizes map[int]int64
	SortedSetKeys         map[string]bool
	SortedSetElements     map[string]bool

	// Hashes
	HashSizes        map[int]int64
	HashElementSizes map[int]int64
	HashValueSizes   map[int]int64
	HashKeys         map[string]bool
	HashElements     map[string]bool
	HashValues       map[string]bool

	// Lists
	ListSizes        map[int]int64
	ListElementSizes map[int]int64
	ListKeys         map[string]bool
	ListElements     map[string]bool
}

Results stores data about sampled redis data structures. Map keys represent lengths/sizes, while map values represent the frequency with which those lengths/sizes occurred in the sampled data. Example keys are stored in golang "sets", which are maps with bool values.

func NewResults

func NewResults() *Results

NewResults constructs a new, zero-valued Results struct

func (*Results) Merge

func (r *Results) Merge(other *Results)

Merge adds the results from `other` into the method receiver. This method can be used to combine sampling results from multiple redis instances into a single result set.

type Statistics

type Statistics struct {
	Mean   float64
	Min    int
	Max    int
	StdDev float64
}

Statistics are basic descriptive statistics that summarize data in a frequency table

func ComputeStatistics

func ComputeStatistics(m map[int]int64) Statistics

ComputeStatistics computes basic descriptive statistics about a frequency map

func NewStatistics

func NewStatistics() *Statistics

NewStatistics creates a new zero-valued Statistics instance

type ValueType

type ValueType string

A ValueType represents the various data types that redis can store. The string representation of a ValueType matches what is returned from redis' `TYPE` command.

var (
	// TypeString represents a redis string value
	TypeString ValueType = "string"

	// TypeSortedSet represents a redis sorted set value
	TypeSortedSet ValueType = "zset"

	// TypeSet represents a redis set value
	TypeSet ValueType = "set"

	// TypeHash represents a redis hash value
	TypeHash ValueType = "hash"

	// TypeList represents a redis list value
	TypeList ValueType = "list"

	// TypeUnknown means that the redis value type is undefined, and indicates an error
	TypeUnknown ValueType = "unknown"

	// ErrNoKeys is the error returned when a specified redis instance contains
	// no keys, or the key count could not be determined
	ErrNoKeys = errors.New("No keys are present in the configured redis instance")
)