api

Documentation

Overview

Package api contains interfaces and low level structs required across multiple packages or repositories.

Index

• Constants
• Variables
• func ExtractDateFromFilename(filename string) (time.Time, error)
• type ASData
  • func (as *ASData) BestASN() (int64, error)
• type Annotations
• type Annotator
• type CachingLoader
• type GeoData
• type GeolocationIP
• type RequestData
• type RequestWrapper
• type System

Constants

const (
	// MaxmindPrefix is the folder containing the maxmind files in GCS.
	MaxmindPrefix = "Maxmind/"
)

Variables

var ErrNilOrEmptyASData = errors.New("Empty or Nil ASData")

ErrNilOrEmptyASData is returned by BestASN if the ASData is nil or empty.

var (
	// MaxmindBucketName is the bucket containing maxmind files.
	MaxmindBucketName = "downloader-" + os.Getenv("GCLOUD_PROJECT")
)

Functions

func ExtractDateFromFilename

func ExtractDateFromFilename(filename string) (time.Time, error)

ExtractDateFromFilename return the date for a filename like gs://downloader-mlab-oti/Maxmind/2017/05/08/20170508T080000Z-GeoLiteCity.dat.gz TODO: both geoloader and geolite2 package use this func, so leave it here for now.

Types

type ASData

type ASData struct {
	IPPrefix string `json:",omitempty"` // the IP prefix found in the table.
	CIDR     string `json:",omitempty"` // The IP prefix found in the RouteViews data.
	ASNumber uint32 `json:",omitempty"` // First AS number.
	ASName   string `json:",omitempty"` // AS name for that number, data from IPinfo.io
	Missing  bool   `json:",omitempty"` // True when the ASN data is missing from RouteViews.

	// One or more "Systems".  There must always be at least one System.  If there are more than one,
	// then this is a Multi-Origin AS, and the component Systems are in order of frequency in routing tables,
	// most common first.
	Systems []System `json:",omitempty"`
}

ASData contains the Autonomous System information associated with the IP prefix. Roughly 99% of mappings consist of a single System with a single ASN.

Looking at Routeviews data from 2019/01/01, the MOAS and AS set stats look like: IPv4: Single: 99% MOAS: 1% AS set: .005% (of entries) 1/2 of AS sets start with MOAS IPv6: Single: 99.3% MOAS: 0.6% AS set: .01% (of entries) 1/3 of AS sets start with MOAS NOTE: This is NOT intended to be used directly as the BigQuery schema.

func (*ASData) BestASN

func (as *ASData) BestASN() (int64, error)

BestASN returns a plausible ASN from a possibly complex ASData.

type Annotations

type Annotations struct {
	Geo     *GeolocationIP // Holds the geolocation data
	Network *ASData        // Holds the associated network Autonomous System data.
}

Annotations is the main struct for annotation metadata, which holds pointers to the Geolocation data and the IP/ASN data. This is what we parse the JSON response from the annotator into.

type Annotator

type Annotator interface {
	// Annotate populates one or more annotation fields in the GeoData object.
	// If it fails, it will return a non-nil error and will leave the target unmodified.
	Annotate(ip string, ann *Annotations) error

	// The date associated with the dataset.
	AnnotatorDate() time.Time
}

Annotator defines the methods required annotating

type CachingLoader

type CachingLoader interface {
	// UpdateCache causes the loader to load any new annotators and add them to the cached list.
	UpdateCache() error

	// Fetch returns a copy of the current list of annotators.
	// May return an empty slice, but must not return nil.
	Fetch() []Annotator
}

CachingLoader keeps a cache of loaded annotators, updates the cache on request, and returns a copy of the cache on request.

type GeoData

type GeoData = Annotations

GeoData is the main struct for the geo metadata, which holds pointers to the Geolocation data and the IP/ASN data. This is what we parse the JSON response from the annotator into. Deprecated: please use api.Annotations

type GeolocationIP

type GeolocationIP struct {
	ContinentCode       string  `json:"continent_code,,omitempty" bigquery:"continent_code"` // Gives a shorthand for the continent
	CountryCode         string  `json:"country_code,,omitempty"   bigquery:"country_code"`   // Gives a shorthand for the country
	CountryCode3        string  `json:"country_code3,,omitempty"  bigquery:"country_code3"`  // Gives a shorthand for the country
	CountryName         string  `json:"country_name,,omitempty"   bigquery:"country_name"`   // Name of the country
	Region              string  `json:"region,,omitempty"         bigquery:"region"`         // Region or State within the country
	Subdivision1ISOCode string  `json:",omitempty"`                                          // ISO3166-2 first-level country subdivision ISO code
	Subdivision1Name    string  `json:",omitempty"`                                          // ISO3166-2 first-level country subdivision name
	Subdivision2ISOCode string  `json:",omitempty"`                                          // ISO3166-2 second-level country subdivision ISO code
	Subdivision2Name    string  `json:",omitempty"`                                          // ISO3166-2 second-level country subdivision name
	MetroCode           int64   `json:"metro_code,,omitempty"     bigquery:"metro_code"`     // Metro code within the country
	City                string  `json:"city,,omitempty"           bigquery:"city"`           // City within the region
	AreaCode            int64   `json:"area_code,,omitempty"      bigquery:"area_code"`      // Area code, similar to metro code
	PostalCode          string  `json:"postal_code,,omitempty"    bigquery:"postal_code"`    // Postal code, again similar to metro
	Latitude            float64 `json:"latitude,,omitempty"       bigquery:"latitude"`       // Latitude
	Longitude           float64 `json:"longitude,,omitempty"      bigquery:"longitude"`      // Longitude
	AccuracyRadiusKm    int64   `json:"radius,,omitempty"         bigquery:"radius"`         // Accuracy Radius (geolite2 from 2018)

	Missing bool `json:",omitempty"` // True when the Geolocation data is missing from MaxMind.
}

The GeolocationIP struct contains all the information needed for the geolocation data that will be inserted into big query. The fields are capitalized for exporting, although the originals in the DB schema are not. This is in common because it is used by the etl repository.

type RequestData

type RequestData struct {
	IP        string    // Holds the IP from an incoming request
	IPFormat  int       // Holds the ip format, 4 or 6
	Timestamp time.Time // Holds the timestamp from an incoming request
}

The RequestData schema is the schema for the json that we will send down the pipe to the annotation service. DEPRECATED Should instead use a single Date (time.Time) and array of net.IP. See the v2 API.

type RequestWrapper

type RequestWrapper struct {
	RequestType string
	Body        json.RawMessage
}

RequestWrapper will be used for all future request types.

type System

type System struct {
	// ASNs contains a single ASN, or AS set.  There must always be at least one ASN.
	// If there are more than one ASN, they are (arbitrarily) listed in increasing numerical order.
	ASNs []uint32
}

A System is the base element. It may contain a single ASN, or multiple ASNs comprising an AS set.