choices

README

choices

A way to choose things.

Building

In order to build you need go installed.

You will need go1.7.1 if you are deploying to kubernetes.

To build choices library and all the included binaries run the following in a terminal.

go get -u github.com/Nordstrom/choices/...

If you are only interested in the library you can run the following in a terminal.

go get -u github.com/Nordstrom/choices

Running locally

There are two main components of choices. The storage server (bolt-store or mongo-store) and the frontend experiment server (elwin). You should first start the storage server. You will need a local version of running, bolt-store will be the easiest to setup. It only requires a local file similar to sqlite.

cd bolt-store && go build
./bolt-store

You can skip to starting the experiment server if you are using the bolt-store.

If you want to use mongo refer to this guide Mongo Installation. If you are using docker you can run the following docker cmd.

docker run -d --name mongo-storage -p 27017:27017 mongo

With mongo running you can start the storage server.

# from the choice/cmd/mongo-store directory run
./mongo-store

This creates the storage service on port 8080

Now that the storage server is up you can start the experiment server (elwin).

# from the choices/cmd/elwin directory run
JSON_ADDRESS=":8181" GRPC_ADDRESS=":8282" MONGO_ADDRESS="localhost:8080" MONGO_DATABASE="elwin" MONGO_COLLECTION="staging" ./elwin

The MONGO_ADDRESS MONGO_DATABASE and MONGO_COLLECTION are an artifact of previously only supporting mongo. These may change in an upcoming version.

The experiment store loads some example data into the staging environment. You should be able to open a web browser and navigate to localhost:8181?label=test&userid=1234 and see some results. You can also run the following command from a terminal if you have curl installed.

The bolt-store does not load sample data.

curl "localhost:8181?label=test&userid=1234"

Documentation

Overview

Package choices provides a library for simple a/b and multivariate testing.

choices uses hashing to uniquely assign users to experiments and decide the values the user is assigned. This allows us to quickly assign a user to an experiment with out having to look up what they were assigned previously. Most of the ideas in this package are based off of Facebook's Planout.

In choices there are three main concepts. Namespaces, Experiments, and Params. Namespaces split traffic between the experiments they contain. Experiments are the element you are testing. Experiments are made up of one or more Params. Params are a key value pair. The value can be either a Uniform Choice value or a Weighted Choice value.

In most cases you will want to create one namespace per experiment. If you have experiments that might have interactions you can use namespaces to split the traffic between them. For example, if you are running a test on a banner and another test that takes over the whole page, you will want to split the traffic between these two tests. Another example, is if you want to run a test on a small percent of traffic. Namespaces contain a list of TeamID's. These TeamID's are used to deterime which Experiments to return to the caller.

Experiments contain the Params for a running experiment. When a caller queries Namespaces, Namespaces will hash the user into a segment, It will then check if the segment is contained in an Experiment. If the segment is contained in the experiment then the experiment will be evaluated. An experiment will in turn evaluate each of it's Params.

Params are key-value pairs. They Options for value are Uniform choice or Weighted choice. Uniform choices will be selected in a uniform fashion. Weighted choices will be selected based on the proportions supplied in the weights.

Index

• Constants
• Variables
• func HashExperience(namespace, experiment, param, userID string) (uint64, error)
• func InSegment(namespace, userID string, s segments) bool
• func SetGlobalSalt(s string) error
• type Config
  • func NewChoices(ctx context.Context, opts ...ConfigOpt) (*Config, error)
  • func (c *Config) IsHealthy() error
  • func (ec *Config) Namespaces(userID string, selector labels.Selector) ([]ExperimentResponse, error)
• type ConfigOpt
  • func WithGlobalSalt(s string) ConfigOpt
  • func WithMaxUpdateFailTime(dur time.Duration) ConfigOpt
  • func WithStorageConfig(addr string, env int, updateInterval time.Duration) ConfigOpt
  • func WithUpdateInterval(dur time.Duration) ConfigOpt
• type ErrUpdateStorage
  • func (e ErrUpdateStorage) Error() string
• type Experiment
  • func FromExperiment(s *storage.Experiment) Experiment
  • func NewExperiment(name string) *Experiment
  • func (e *Experiment) MarshalJSON() ([]byte, error)
  • func (e *Experiment) SampleSegments(ns *Namespace, num int) *Experiment
  • func (e *Experiment) SetSegments(seg segments) *Experiment
  • func (e *Experiment) ToExperiment() *storage.Experiment
• type ExperimentResponse
• type Namespace
  • func FromNamespace(s *storage.Namespace) (Namespace, error)
  • func NewNamespace(name string) *Namespace
  • func (n *Namespace) MarshalJSON() ([]byte, error)
  • func (n *Namespace) ToNamespace() *storage.Namespace
• type Param
  • func FromParam(s *storage.Param) Param
  • func (p *Param) MarshalJSON() ([]byte, error)
  • func (p *Param) ToParam() *storage.Param
• type ParamValue
• type Uniform
  • func (u *Uniform) MarshalJSON() ([]byte, error)
  • func (u *Uniform) Value(i uint64) (string, error)
• type Value
• type ValueType
• type Weighted
  • func (w *Weighted) MarshalJSON() ([]byte, error)
  • func (w *Weighted) Value(i uint64) (string, error)

Constants

const (
	StorageEnvironmentBad = iota
	StorageEnvironmentDev
	StorageEnvironmentProd
)

constants for storage environments. So far only support a staging and production.

Variables

var ErrBadStorageEnvironment = errors.New("bad storage environment")

ErrBadStorageEnvironment is an error for when the storage environment is not set correctly.

var (

	// ErrGlobalSaltAlreadySet is the error returned when a SetGlobalSalt
	// has been called more than once.
	ErrGlobalSaltAlreadySet = errors.New("global salt already set")
)

var (
	// ErrSegmentNotInExperiment occurs when a user is hashed into a
	// segment that has not been claimed by an experiment.
	ErrSegmentNotInExperiment = errors.New("Segment is not assigned to an experiment")
)

var (
	// ErrSegmentUnavailable is thrown when you request an a segment set to
	// 0, an unavailable segment.
	ErrSegmentUnavailable = errors.New("segment unavailable")
)

Functions

func HashExperience

func HashExperience(namespace, experiment, param, userID string) (uint64, error)

HashExperience takes the supplied arguments and returns the hashed uint64 that can be used for determining a segment.

func InSegment

func InSegment(namespace, userID string, s segments) bool

InSegment takes a namespace name, userID and segments and returns whether the userID is in a segment that is claimed. Typical use case for this is determining whether or not a user is in a given experiment.

func SetGlobalSalt

func SetGlobalSalt(s string) error

SetGlobalSalt this sets the global salt used for hashing users. It should only ever be called once. It returns an error if it is called more than once.

Types

type Config

type Config struct {
	ErrChan chan error
	// contains filtered or unexported fields
}

Config is the configuration struct used in an elwin server.

func NewChoices

func NewChoices(ctx context.Context, opts ...ConfigOpt) (*Config, error)

NewChoices sets the storage engine. It starts a ticker that will call s.Update() until the context is cancelled. To change the tick interval call SetUpdateInterval(d time.Duration). Must cancel the context before calling NewChoices again otherwise you will leak go routines.

func (*Config) IsHealthy

func (c *Config) IsHealthy() error

IsHealthy returns an error if the time since the last successful update is longer than the max allowed time to fail. The interval for updates can be set by WithUpdateInterval. The time until failure can be set with WithMaxUpdateFailTime.

func (*Config) Namespaces

func (ec *Config) Namespaces(userID string, selector labels.Selector) ([]ExperimentResponse, error)

Namespaces determines the assignments for the a given user's id based on the current set of namespaces and experiments. It returns a []ExperimentResponse if it is successful or an error if something went wrong.

type ConfigOpt

type ConfigOpt func(*Config) error

ConfigOpt is a type that modifies Config. It is used when calling NewChoices to configure choices.

func WithGlobalSalt

func WithGlobalSalt(s string) ConfigOpt

WithGlobalSalt is a configuration option for Config that sets the globalSalt to something other than the default.

func WithMaxUpdateFailTime

func WithMaxUpdateFailTime(dur time.Duration) ConfigOpt

WithMaxUpdateFailTime changes the max duration allowed for failing updates.

func WithStorageConfig

func WithStorageConfig(addr string, env int, updateInterval time.Duration) ConfigOpt

WithStorageConfig is where you set the address and environment you'd like to point. This is used as a ConfigOpt in NewChoices.

func WithUpdateInterval

func WithUpdateInterval(dur time.Duration) ConfigOpt

WithUpdateInterval changes the update interval for Storage. Must call SetStorage after this or cancel context of the current Storage and call SetStorage again.

type ErrUpdateStorage

type ErrUpdateStorage struct {
	// contains filtered or unexported fields
}

ErrUpdateStorage is an error type that is returned when storage fails to update.

func (ErrUpdateStorage) Error

func (e ErrUpdateStorage) Error() string

Error is to implement the error interface

type Experiment

type Experiment struct {
	Name     string
	Labels   labels.Set
	Params   []Param
	Segments segments
}

Experiment is a structure that represents a single experiment in elwin. It can contain multiple parameters. Experiments are evaluated through the call to Namespaces.

func FromExperiment

func FromExperiment(s *storage.Experiment) Experiment

FromExperiment converts a *storage.Experiment into an Experiment

func NewExperiment

func NewExperiment(name string) *Experiment

NewExperiment creates an experiment with the supplied name and no segments cliamed. In order for any traffic to be assigned to this experiment you will need to call Experiment.SetSegments or Experiment.SampleSegments.

func (*Experiment) MarshalJSON

func (e *Experiment) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for Experiments.

func (*Experiment) SampleSegments

func (e *Experiment) SampleSegments(ns *Namespace, num int) *Experiment

SampleSegments takes a namespace and an amount of segments you want in your experiment and returns a random sample of the unclaimed segments from the namespace.

func (*Experiment) SetSegments

func (e *Experiment) SetSegments(seg segments) *Experiment

SetSegments copies the segments supplied to the experiment.

func (*Experiment) ToExperiment

func (e *Experiment) ToExperiment() *storage.Experiment

ToExperiment is a helper function that converts an Experiment into a *storage.Experiment.

type ExperimentResponse

type ExperimentResponse struct {
	Name      string
	Namespace string
	Params    []ParamValue
}

ExperimentResponse holds the data for an evaluated expeiment.

type Namespace

type Namespace struct {
	Name        string
	Segments    segments
	Experiments []Experiment
}

Namespace is a container for experiments. Segments in the namespace divide traffic. Units are the keys that will hash experiments.

func FromNamespace

func FromNamespace(s *storage.Namespace) (Namespace, error)

FromNamespace converts a *storage.Namespace into a Namespace.

func NewNamespace

func NewNamespace(name string) *Namespace

NewNamespace creates a new namespace with all segments available. It returns an error if no units are given.

func (*Namespace) MarshalJSON

func (n *Namespace) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for Namespaces.

func (*Namespace) ToNamespace

func (n *Namespace) ToNamespace() *storage.Namespace

ToNamespace is a helper function that converts a Namespace into a *storage.Namespace.

type Param

type Param struct {
	Name  string
	Value Value
}

Param is a struct that represents a single parameter in an experiment. Param is evaluated through the call to Namespaces.

func FromParam

func FromParam(s *storage.Param) Param

FromParam converts a *storage.Param into a Param

func (*Param) MarshalJSON

func (p *Param) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for Params.

func (*Param) ToParam

func (p *Param) ToParam() *storage.Param

ToParam is a helper function that converts a Param into a *storage.Param.

type ParamValue

type ParamValue struct {
	Name  string
	Value string
}

ParamValue is a key value pair returned from an evalated experiment parameter.

type Uniform

type Uniform struct {
	Choices []string
}

Uniform is a way to select from a list of Choices with uniform probability.

func (*Uniform) MarshalJSON

func (u *Uniform) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for Uniform choices.

func (*Uniform) Value

func (u *Uniform) Value(i uint64) (string, error)

Value implements the Value interface for Uniform choices.

type Value

type Value interface {
	Value(i uint64) (string, error)
}

Value is the interface Param Values must implement. They take a hash value and return the string that represents the value or an error.

type ValueType

type ValueType int

ValueType are the different types of Values a Param can have. This is used for parsing params in storage.

const (
	ValueTypeBad ValueType = iota
	ValueTypeUniform
	ValueTypeWeighted
)

Constants for ValueTypes

type Weighted

type Weighted struct {
	Choices []string
	Weights []float64
}

Weighted is a way to select from a list of Choices with probability ratio supplied in Weights.

func (*Weighted) MarshalJSON

func (w *Weighted) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface for Weighted choices.

func (*Weighted) Value

func (w *Weighted) Value(i uint64) (string, error)

Value implements the Value interface for Weighted choices.