stonecutters

README

Stonecutters

This package is still in active development; APIs may change so lock to a semantic version

This package looks deep within a goroutine's soul and assigns it a name based on the order in which it joined.

Designed to address container namespace pollution of metric collection systems.(Carbon in particular, where namespace pollution kills Graphite's performance.) Yet another reason to use Prometheus instead.

For distributed systems; assigning UIDs to running processes is a common identity metchanism to match with their metrics. However older metric designes like Carbon/Graphite don't handle unique naming which polute and cause large wildcard search paths. Thus recycling names as prefixes can be used to reduce namespace pollution by uids, but still keep processes effectively unique.

Distributed processes which need to share names but for identification and maintain uniqueness. This works but having a shared static set of names which are claimed using etcd(v3) as the distributed lock. Each process iterates over the ordered list, and claim the first name which isn't regestered/claimed in etcd.

Provided static names are the top 100 highest mountains in North America, ordered by decending peak elevation. However any list of identifiers can be passed into stonecutters.Join(...)

eg owners/hosts named:

foo-ns31 -> Denali
foo-3le9 -> MtLogan
...
foo-wedc -> MtRainier

etcd stonecutters Transaction to assign an Identifier

1. if key does not exist; claim key with PUT the key:value{Identifier: Owner} pair using a lease
2. if key exists; iterate to next identifier and retry claim transaction
3. If all identifiers are claimed; return error.

API

IDs := []string{"coffee", "tea", "bikes"}
ctx, cancl := context.WithCancel()

// Responsibility of keeping lease alive is up to caller
lease, err :=  etcdclient.Grant(ctx, int64(5))
...

// Join the stonecutters IDs list
member, err := stonecutters.Join(etcdclient, ctx, leaseID.ID, "homer", IDs)
...

// List all members
members, err := stonecutters.Members(etcdclient, IDs)
...

Testing

Since etcd is critical to the stonecutters, tests are all effectively integration tests.

Recomended strategy is to run etcd in standalone along with the tests.Downloading etcd and then run with eg:./etcd-v3.2.10-linux-amd64/etcd

Embeded etcd can be configured to start and run with the tests by setting ETCDEMBED=1 in the test environment. This make starting tests take a while though.

All tests attempt to clean up and revoke all keys after finishing as to not polute etcd between runs.

Documentation

Overview

This package looks deep within a goroutine's soul and assigns it a name based on the order in which it joined.

Designed to address the container namespace pollution of metric collection systems. Carbon https://github.com/graphite-project/carbon in particular, where namespace pollution kills Graphite's performance.)

Distributed processes which need to share names but for identification and maintain uniqueness. This works but having a shared static set of names which are claimed using etcd(v3) as the distributed lock. Each process iterates over the ordered list, and claim the first name which isn't regestered/claimed in etcd.

Provided static names are the top 100 highest mountains in North America, ordered by decending peak elevation. However any list of identifiers can be passed into stonecutters.Join(...)

Request an atomic ID from etcdv3:

IDs := []string{"coffee", "tea", "bikes"}
ctx, cancl := context.WithCancel()

// Responsibility of keeping lease alive is up to caller
lease, err :=  etcdclient.Grant(ctx, int64(5))
...

// Join the stonecutters IDs list
member, err := stonecutters.Join(etcdclient, ctx, leaseID.ID, "homer", IDs)
...

// List all members
members, err := stonecutters.Members(etcdclient, IDs)
...

Index

• Variables
• func NormalizedNaMountains() []string
• func PrefixedNumerics(prefix string, max int) []string
• type Member
  • func Join(c *clientv3.Client, ctx context.Context, leaseID clientv3.LeaseID, name string, ...) (*Member, error)
  • func Members(c *clientv3.Client, ids []string) ([]*Member, error)

Variables

var (
	GetIdFailure        = errors.New("lock: failed to get identifier from list")
	PutSucceededFailure = errors.New("lock: key already registered")
	VerificationError   = errors.New("lock: k-v values do not match txn request") // very unlikely but strange error
)

var NAMountains = []string{
	"Denali",
	"MtLogan",
	"PicodeOrizaba",
	"MtStElias",
	"VolcanPopocatepetl",
	"MtForaker",
	"MtLucania",
	"VolcanIztaccihuatl",
	"KingPeak",
	"MtBona",
	"MtSteele",
	"MtBlackburn",
	"MtSanford",
	"MtWood",
	"MtVancouver",
	"NevadodeToluca",
	"MtFairweather",
	"MtHubbard",
	"MtBear",
	"MtWalsh",
	"MtHunter",
	"VolcanLaMalinche",
	"MtWhitney",
	"UniversityPeak",
	"MtElbert",
	"MtHarvard",
	"MtRainier",
	"BlancaPeak",
	"UncompahgrePeak",
	"McArthurPeak",
	"CrestonePeak",
	"MtLincoln",
	"GraysPeak",
	"MtAntero",
	"CastlePeak",
	"MtEvans",
	"LongsPeak",
	"WhiteMountainPeak",
	"MtWilson",
	"NorthPalisade",
	"NevadodeColima",
	"MtPrinceton",
	"MtWrangell",
	"MtShasta",
	"MaroonPeak",
	"MtSneffels",
	"PikesPeak",
	"MtEolus",
	"MtAugusta",
	"CulebraPeak",
	"SanLuisPeak",
	"MtoftheHolyCross",
	"MtHumphreys",
	"MtOuray",
	"MtStrickland",
	"VermilionPeak",
	"AtnaPeaks",
	"RegalMountain",
	"VolcánTajumulco",
	"MtHayes",
	"MtSilverheels",
	"GannettPeak",
	"MtKaweah",
	"VolcanCofredePerote",
	"GrandTeton",
	"MtCook",
	"MtMorgan",
	"MtGabb",
	"BaldMountain",
	"WestSpanishPeak",
	"MtPowell",
	"HaguesPeak",
	"MtDubois",
	"KingsPeak",
	"TreasureMountain",
	"MtPinchot",
	"MtNatazhat",
	"MtJarvis",
	"VolcánTacaná",
	"MtHerard",
	"SummitPeak",
	"AntoraPeak",
	"HesperusMountain",
	"MtSilverthrone",
	"JacquePeak",
	"WindRiverPeak",
	"MtWaddington",
	"MtMarcusBaker",
	"CloudPeak",
	"WheelerPeak",
	"TwilightPeak",
	"FrancsPeak",
	"SouthRiverPeak",
	"MtRitter",
	"BushnellPeak",
	"TruchasPeak",
	"WheelerPeak",
	"MtDana",
	"SpringGlacierPeak",
	"VolcánAcatenango",
}

Functions

func NormalizedNaMountains

func NormalizedNaMountains() []string

NormalizedNaMountains returns a North American Mountain name slice of ASCII strings

func PrefixedNumerics

func PrefixedNumerics(prefix string, max int) []string

PrefixedNumerics returns a list of ids with a simple name prefix followed by an integer separated by a dash. Naming starts at '1' not '0'.

Types

type Member

type Member struct {
	Key   string // Identifier granted
	Value string // Owner/Hostname
}

Member is a struct to encapuslate the etcd data pairing to data Key[Identifier]: Value:[Owner]

func Join

func Join(c *clientv3.Client, ctx context.Context, leaseID clientv3.LeaseID,
	name string, ids []string) (*Member, error)

Join iterates over the passed 'ids' and attempts to claim one in etcd with a Lease which is persisted until the context is closed. If the list of ids are all claimed, returns GetIdFailure error with the expectation the caller will handle managing the id list retrys.

func Members

func Members(c *clientv3.Client, ids []string) ([]*Member, error)

Members returns a list of all Identifiers assigned to an owner.