feature

package module
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 10, 2021 License: MIT Imports: 20 Imported by: 0

README

feature build status

Feature gate database designed for simplicity and efficiency.

Motivation

Feature gates are an important part of controlling the risk associated with software releases, they bring safe guards and granular knobs over the exposure of data to new code paths.

However, these promises can only be kept if programs can reliably access the feature gate data, and query the data set with high efficiency. Most feature gate systems rely on performing network calls to a foreign system, creating opportunities for cascading failures in distributed systems where feature gate checks are often performed on critical data paths.

The feature package was designed to offer high availbility of the feature gates, and high query performance, allowing its use in large scale systems with many nines of uptime like those run by Segment.

Reliability

The feature database is represented by an immutable set of directories and files on a file system. The level of reliability offered by a set of files on disk exceeds by a wide margin what can be achieved with a daemon process serving the data over a network interface. Would the program updating the feature database be restarted or crashed, the files would remain available for consumers to read and query. The system is known to fail static: in the worst case scenario, nothing changes.

Efficiency

The feature database being immutable, it enables very efficient access to the data. Programs can implement simple and high performance caching layers because they do not need to manage cache expirations or transactional updates. The database files are mapped to read-only memory areas and therefore can be shared by all collocated processes, ensuring that a single copy of the data ever exists in memory.

Data Models

Collections

Collections are lists of unique identifiers that programs can query the state of gates for; gates are either open or closed. The collections are arranged in groups and tiers. Each group may have multiple tiers, within each tier the collection files contain the list of identifiers, one by line.

Here is an example of the on-disk representation of collections:

$ tree
.
└── standard
    ├── 1
    │   ├── collections
    │   │   ├── source
    │   │   ├── workspace
    │   │   └── write_key
...

For the standard group, tier 1, there are three collections of source, workspace and write keys.

$ cat ./standard/1/collections/source
ACAtsprztv
B458ru47n7
CQRxBaQSt8
EJw9i04Lsv
IbQor7hHBU
LZK0HYwDTH
MKOxgJsedB
OmNMfU6RbP
Q5lmdTzq1Y
SqNT0bDYl7
...

On-disk file structures with large number of directories and small files cause space usage amplification, leading to large amounts of wasted space. By analyzing the volume of data used to represent feature flags, we observed that most of the space was used by the collections of identifiers. Text files provide a compact representation of the identifiers, minimizing storage space waste caused by block size alignment, and offering a scalable model to grow the number of collections and identifiers in the system.

Gates

The second core data type are feature gates, which are grouped by family, name, and collections that they apply to. The gate family and name are represented as directories, and the gate data per collection are stored in text files of key/value pairs.

Continuing on our previous example, here is a view of the way gates are laid out in the file system:

$ tree
.
└── standard
    ├── 1
...
    │   └── gates
    │       ├── access-management
    │       │   └── invite-flow-enabled
    │       │       └── workspace
...

For the standard group, tier 1, gate invite-flow-enabled of the access-management family is enabled for workspaces.

$ cat ./standard/1/gates/access-management/invite-flow-enabled/workspace
open	true
salt	3653824901
volume	1

The gate files contain key value pairs for the few properties of a gate, which determine which of the identifiers will see the gate open or closed.

Key Value
open true/false, indicates the default behavior for identifiers that are not in the collection file
salt random value injected in the hash function used to determine the gate open state
volume floating point number between 0 and 1 defining the volume of identifiers that the gate is open for

Using the CLI

The cmd/feature program can be used to explore the state of a feature database. The CLI has multiple subcommands, we present a few useful ones in this section.

All subcommand understand the following options:

Option Environment Variable Description
-p, --path FEATURE_PATH Path to the feature database to run commands on

The FEATURE_PATH environment variable provides a simple mechanism to configure configure the default database used by the command:

$ export FEATURE_PATH=/path/to/features

By default, the $HOME/feature directory is used.

feature get gates [collection] [id]

This command prints the list of gates enabled for an identifier, it is useful to determine whether a gate is open for a given id, for example:

# empty output if the gate is not open
$ feature get gates source B458ru47n7 | grep gate-family | grep gate-name
feature get tiers

This command prints a summary of the tiers that exist in the feature database, here is an example:

$ feature get tiers
GROUP      TIER  COLLECTIONS  FAMILIES  GATES
standard   7     0            17        39
standard   6     0            18        40
standard   1     3            20        109
standard   8     0            17        39
standard   4     3            18        41
standard   3     0            18        41
standard   2     3            19        107
standard   5     3            18        40
feature describe collection [-g group] [-t tier] [collection]

This command prints the list of identifiers in a collection, with the option to filter on a group and tier; by default all groups and tiers are shown.

$ feature describe collection workspace
96x782dXhZmn6RpPJVDXgG
4o74gqFGmTgq7GS6EN3ZQJ
mcYdYvfZQcUaid1CVdC9F3
nRRroPD8pV3giaetjpDmu7
96x782dXhZmn6RpPJVDXgG
1232rt203
9a2aceada5
cus_HbXktPfAbH3weZ
opzvxHK692ZJJicNxz1AfL
pkpdcdSLNX14Za6qpD7wtv
...

Note: the identifiers are not displayed in any particular order, this command iterates over the directories and scans the collection files.

feature describe tier [group] [tier]

This command shows a verbose description of a tier, including the list of collections, and the state of each gate in the tier:

$ feature describe tier standard 1
Group:	standard
Tier:	1

Collections:
 - write_key
 - workspace
 - source

Gates:
  integrations-consumer/observability-discards-gate
  - workspace	(100%, default: open)

  destinations-59ceac7c2828a60001d22936/centrifuge_qa
  - workspace	(100%, default: open)

  destinations-54521fdc25e721e32a72ef04/webhook-flagon-centrifuge
  - write_key	(100%, default: close)

...

Using the Go API

The feature package provides APIs to consume the feature gate data set, this section presents on the most common use cases that programs have and how they are solved by the package.

import (
    "github.com/segmentio/feature"
)
feature.MountPoint

The feature.MountPoint type represents a path on the file system where a feature database is mounted. This type is the entry point to all other APIs, a common pattern is for programs to construct a mount point from a configuration option or environment variable:

mountPoint := feature.MountPoint("/path/to/features")

Note: prefer using an absolute path for the mount point, so operations are not dependent on the working directory.

feature.Store

From a mount point, a program can open a feature database, which is materialized by a feature.Store object.

features, err := mountPoint.Open()
if err != nil {
    fmt.Fprintf(os.Stderr, "ERROR: %s\n", err)
} else {
    ...
}

The feature.Store type will watch for changes to the mount point, and automatically reload the content of the feature database when a change is detected. This mechanism assumes that the feature database is immutable, programs that intend to apply updates to the database must recreate it and replace the entire directory structure (which should be done in an atomic fashion via the use of the rename(2) syscall for example).

feature.(*Store).GateOpen

This is the most common use case for programs, the GateOpen method tests whether a gate is open for a given identifier.

The gate is defined by the pair of gate family and name, while the identifier is expressed as a pair of the collection and its value.

if features.GateOpen("gate-family", "gate-name", "collection", "1234") {
    ...
}
feature.(*Store).LookupGates

Another common use case is for programs to lookup the list of gates that are enabled on an identifier. The LookupGates method solves for this use case.

for _, gate := range features.LookupGates("gate-family", "collection", "1234") {
    ...
}

Note: the feature.Store type uses an internal cache to optimize gate lookups, programs must treat the returned slice as an immutable value to avoid race conditions. If the slice needs to be modified, a copy must be made first.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Scan 

func Scan(it Iter, do func(string) error) error

Scan is a helper function used to iterate over each name exposed by an iterator.

Types

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is an in-memory view of feature mount point on a file system.

The cache is optimized for fast lookups of gates, and fast test of gate open states for an id. The cache is also immutable, and therefore safe to use concurrently from multiple goroutines.

The cache is designed to minimize the memory footprint. The underlying files containing the id collections are memory mapped so multiple programs are able to share the memory pages.

func (*Cache) Close 

func (c *Cache) Close() error

Close releases resources held by the cache.

func (*Cache) GateOpen 

func (c *Cache) GateOpen(family, gate, collection, id string) bool

GateOpen returns true if a gate is opened for a given id.

The method does not retain any of the strings passed as arguments.

func (*Cache) LookupGates 

func (c *Cache) LookupGates(family, collection, id string) []string

LookupGates returns the list of open gates in a family for a given id.

The method does not retain any of the strings passed as arguments.

type Collection 

type Collection struct {
	// contains filtered or unexported fields
}

func (*Collection) Add 

func (col *Collection) Add(id string) error

func (*Collection) Close 

func (col *Collection) Close() error

func (*Collection) IDs 

func (col *Collection) IDs() *IDIter

func (*Collection) Path 

func (col *Collection) Path() string

func (*Collection) Sync 

func (col *Collection) Sync() error

type CollectionIter 

type CollectionIter struct {
	// contains filtered or unexported fields
}

func (*CollectionIter) Close 

func (it *CollectionIter) Close() error

func (*CollectionIter) IDs 

func (it *CollectionIter) IDs() *IDIter

func (*CollectionIter) Name 

func (it *CollectionIter) Name() string

func (*CollectionIter) Next 

func (it *CollectionIter) Next() bool

type FamilyIter 

type FamilyIter struct {
	// contains filtered or unexported fields
}

func (*FamilyIter) Close 

func (it *FamilyIter) Close() error

func (*FamilyIter) Gates 

func (it *FamilyIter) Gates() *GateIter

func (*FamilyIter) Name 

func (it *FamilyIter) Name() string

func (*FamilyIter) Next 

func (it *FamilyIter) Next() bool

type GateCreatedIter 

type GateCreatedIter struct {
	// contains filtered or unexported fields
}

func (*GateCreatedIter) Close 

func (it *GateCreatedIter) Close() error

func (*GateCreatedIter) Name 

func (it *GateCreatedIter) Name() string

func (*GateCreatedIter) Next 

func (it *GateCreatedIter) Next() bool

type GateDisabledIter 

type GateDisabledIter struct {
	// contains filtered or unexported fields
}

func (*GateDisabledIter) Close 

func (it *GateDisabledIter) Close() error

func (*GateDisabledIter) Family 

func (it *GateDisabledIter) Family() string

func (*GateDisabledIter) Gate 

func (it *GateDisabledIter) Gate() string

func (*GateDisabledIter) Name 

func (it *GateDisabledIter) Name() string

func (*GateDisabledIter) Next 

func (it *GateDisabledIter) Next() bool

type GateEnabledIter 

type GateEnabledIter struct {
	// contains filtered or unexported fields
}

func (*GateEnabledIter) Close 

func (it *GateEnabledIter) Close() error

func (*GateEnabledIter) Family 

func (it *GateEnabledIter) Family() string

func (*GateEnabledIter) Gate 

func (it *GateEnabledIter) Gate() string

func (*GateEnabledIter) Name 

func (it *GateEnabledIter) Name() string

func (*GateEnabledIter) Next 

func (it *GateEnabledIter) Next() bool

type GateIter 

type GateIter struct {
	// contains filtered or unexported fields
}

func (*GateIter) Close 

func (it *GateIter) Close() error

func (*GateIter) Created 

func (it *GateIter) Created() *GateCreatedIter

func (*GateIter) Name 

func (it *GateIter) Name() string

func (*GateIter) Next 

func (it *GateIter) Next() bool

type GroupIter 

type GroupIter struct {
	// contains filtered or unexported fields
}

func (*GroupIter) Close 

func (it *GroupIter) Close() error

func (*GroupIter) Name 

func (it *GroupIter) Name() string

func (*GroupIter) Next 

func (it *GroupIter) Next() bool

func (*GroupIter) Tiers 

func (it *GroupIter) Tiers() *TierIter

type IDIter 

type IDIter struct {
	// contains filtered or unexported fields
}

func (*IDIter) Close 

func (it *IDIter) Close() error

func (*IDIter) Name 

func (it *IDIter) Name() string

func (*IDIter) Next 

func (it *IDIter) Next() bool

type Iter 

type Iter interface {
	Close() error
	Next() bool
	Name() string
}

Iter is an interface implemented by the iterator types exposed by this package.

type MountPoint 

type MountPoint string

MountPoint represents the mount point of a feature file system.

The type is a simple string alias, its value is the path to the root directory of the file system.

func Mount 

func Mount(path string) (MountPoint, error)

func (MountPoint) CreateTier 

func (path MountPoint) CreateTier(group, name string) (*Tier, error)

func (MountPoint) DeleteGroup 

func (path MountPoint) DeleteGroup(group string) error

func (MountPoint) DeleteTier 

func (path MountPoint) DeleteTier(group, name string) error

func (MountPoint) Groups 

func (path MountPoint) Groups() *GroupIter

func (MountPoint) Load 

func (path MountPoint) Load() (*Cache, error)

The Load method loads the features at the mount point it is called on, returning a Cache object exposing the state.

The returned cache holds operating system resources and therefore must be closed when the program does not need it anymore.

func (MountPoint) Open 

func (path MountPoint) Open() (*Store, error)

The Open method opens the features at the mount point it was called on, returning a Store object exposing the state.

The returned store holds operating system resources and therefore must be closed when the program does not need it anymore.

func (MountPoint) OpenTier 

func (path MountPoint) OpenTier(group, name string) (*Tier, error)

func (MountPoint) Tiers 

func (path MountPoint) Tiers(group string) *TierIter

func (MountPoint) Wait 

func (path MountPoint) Wait(ctx context.Context) error

Wait blocks until the path exists or ctx is cancelled.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store is similar to Cache, but automatically reloads when updates are made to the underlying file system.

func (*Store) Close 

func (s *Store) Close() error

Close closes the store, releasing all associated resources.

func (*Store) GateOpen 

func (s *Store) GateOpen(family, gate, collection, id string) bool

GateOpen returns true if a gate is opened for a given id.

func (*Store) LookupGates 

func (s *Store) LookupGates(family, collection, id string) []string

LookupGates returns the list of open gates in a family for a given id.

type Tier 

type Tier struct {
	// contains filtered or unexported fields
}

func (*Tier) Close 

func (tier *Tier) Close() error

func (*Tier) Collections 

func (tier *Tier) Collections() *CollectionIter

func (*Tier) CreateCollection 

func (tier *Tier) CreateCollection(collection string) (*Collection, error)

func (*Tier) CreateGate 

func (tier *Tier) CreateGate(family, name, collection string, salt uint32) error

func (*Tier) DeleteCollection 

func (tier *Tier) DeleteCollection(collection string) error

func (*Tier) DeleteGate 

func (tier *Tier) DeleteGate(family, name, collection string) error

func (*Tier) EnableGate 

func (tier *Tier) EnableGate(family, name, collection string, volume float64, open bool) error

func (*Tier) Families 

func (tier *Tier) Families() *FamilyIter

func (*Tier) Gates 

func (tier *Tier) Gates(family string) *GateIter

func (*Tier) GatesCreated 

func (tier *Tier) GatesCreated(family, gate string) *GateCreatedIter

func (*Tier) GatesDisabled 

func (tier *Tier) GatesDisabled(collection, id string) *GateDisabledIter

func (*Tier) GatesEnabled 

func (tier *Tier) GatesEnabled(collection, id string) *GateEnabledIter

func (*Tier) Group 

func (tier *Tier) Group() string

func (*Tier) IDs 

func (tier *Tier) IDs(collection string) *IDIter

func (*Tier) Name 

func (tier *Tier) Name() string

func (*Tier) OpenCollection 

func (tier *Tier) OpenCollection(collection string) (*Collection, error)

func (*Tier) ReadGate 

func (tier *Tier) ReadGate(family, name, collection string) (open bool, salt string, volume float64, err error)

func (*Tier) String 

func (tier *Tier) String() string

type TierIter 

type TierIter struct {
	// contains filtered or unexported fields
}

func (*TierIter) Close 

func (it *TierIter) Close() error

func (*TierIter) Name 

func (it *TierIter) Name() string

func (*TierIter) Next 

func (it *TierIter) Next() bool

Source Files

View all Source files

Directories

Path Synopsis
cmd
feature

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.