endpoint

Documentation

Overview

Package endpoint contains stats for client endpoints. An endpoint corresponds to an mlab-ns request signature that we expect may represent an individual requester. (IP alone is insufficient, because of CG-NAT and use of proxies). We use the userAgent, resource string, and IP address. NOTE: We currently limit results to 20K endpoints, and as of Sept 2018, we are seeing about 8K endpoints with more than 12 requests per day. The limit is imposed because we don't have enough experience to predict how mlab-ns might behave if bad endpoints grew to 100K or 200K.

Index

• func CalculateClientSignature(ip string, userAgent string, resource string) string
• func DSKey(name string) *datastore.Key
• func DeleteAllKeys(ctx context.Context, client *datastore.Client, namespace string, kind string) (int, error)
• func FetchEndpointStats(ctx context.Context, dsExt *bqx.Dataset, threshold int) ([]map[string]bigquery.Value, error)
• func GetAllKeys(ctx context.Context, client *datastore.Client, namespace string, kind string) ([]*datastore.Key, error)
• func PutMulti(ctx context.Context, client *datastore.Client, keys []*datastore.Key, ...) error
• func SetMulti(c context.Context, keys []*datastore.Key, endpoints []Stats) error
• type Stats
  • func MakeKeysAndStats(rows []map[string]bigquery.Value, threshold int) ([]*datastore.Key, []Stats, error)
  • func StatsFromMap(row map[string]bigquery.Value, threshold int) (string, Stats)
  • func (ep *Stats) Save(ctx context.Context, client datastore.Client, key string) error

Functions

func CalculateClientSignature

func CalculateClientSignature(ip string, userAgent string, resource string) string

CalculateClientSignature generates a signature based on the client IP, UserAgent and requested resource. The resulting signature must not be longer than 250 bytes so it can be used as a memcache key (having MAX_LENGTH=250).

func DSKey

func DSKey(name string) *datastore.Key

DSKey creates a Datastore Key by adding namespace and kind to name string.

func DeleteAllKeys

func DeleteAllKeys(ctx context.Context, client *datastore.Client, namespace string, kind string) (int, error)

DeleteAllKeys deletes all keys for a namespace and kind from Datastore, dividing into blocks of 500 to satisfy datastore API constraint. If there are errors, it returns the last error message, but the length returned will not reflect the failures.

func FetchEndpointStats

func FetchEndpointStats(ctx context.Context, dsExt *bqx.Dataset, threshold int) ([]map[string]bigquery.Value, error)

FetchEndpointStats executes simpleQuery, and returns a slice of rows containing endpoint signatures and request counts. TODO - move the body (excluding simpleQuery) into go/bqext

func GetAllKeys

func GetAllKeys(ctx context.Context, client *datastore.Client, namespace string, kind string) ([]*datastore.Key, error)

GetAllKeys fetches all keys from Datastore for a namespace and kind.

func PutMulti

func PutMulti(ctx context.Context, client *datastore.Client, keys []*datastore.Key, endpoints []Stats) error

PutMulti writes a set of keys and endpoints to datastore, dividing into blocks of 500 to satisfy datastore API constraint.

func SetMulti

func SetMulti(c context.Context, keys []*datastore.Key, endpoints []Stats) error

SetMulti writes the keys and endpoints Probability. To preserve cross-language compatibility, SetMulti formats the endpoints Probability (float) as a string encoded integer. Readers should parse the value as an int, then divide by 10000 to recover the original probability.

Types

type Stats

type Stats struct {
	RequestsPerDay int64   `datastore:"requests_per_day"` // Number of requests made per day.
	Probability    float32 `datastore:"probability"`      // Fraction of requests that should be sent to standard backend.
}

Stats contains information about request rate for an endpoint, and probability for mlab-ns to use in routing new requests.

func MakeKeysAndStats

func MakeKeysAndStats(rows []map[string]bigquery.Value, threshold int) ([]*datastore.Key, []Stats, error)

MakeKeysAndStats converts slice of bigquery rows into DSKeys and Stats objects.

func StatsFromMap

func StatsFromMap(row map[string]bigquery.Value, threshold int) (string, Stats)

StatsFromMap creates a Key and Stats object from a bigquery result map.

func (*Stats) Save

func (ep *Stats) Save(ctx context.Context, client datastore.Client, key string) error

Save saves an endpoint to datastore