kero

README

Kero 📊

Kero is a privacy-friendly, embeddable, analytics dashboard for your Go websites. With its drop-in integrations, it's the easiest way to get an overview of the key web metrics.

• Privacy-friendly: Kero tracks server-side requests with limited access to identifiable user data, compared to client-side solutions.
• Embedded: Import Kero middleware and you're ready to go, there are no additional databases or servers to provision and maintain.
• Easy to understand: Kero comes with a glanceable dashboard that contains all the data you care about on a single page.

Getting started

To get started with minimal Kero configuration you can simply mount its tracker middleware and dashboard route before initializing your routes in your web server.

Start by fetching the library and adding it to your go.mod:

go get github.com/josip/kero

[!NOTE] This will add indirect dependencies to web frameworks you're not necessarily using as well as many other dependencies pulled in by Prometheus. These will not be included in the built binary.

Within your application you'll have to import both the main Kero library as well as middleware specific to your web framework. Currently Gin and Fiber are supported.

package main

import (
    "os"
    "github.com/gin-gonic/gin"
    "github.com/josip/kero"
    keromw "github.com/josip/kero/keroginmw"
)

Start by creating a new Kero instance:

k, _ := kero.New(
    kero.WithDBPath("./kero"),
    kero.WithDashboardPath("/_kero"),
)
defer k.Close()

DBPath is where the data will be persisted, with the folder automatically created if it doesn't exist yet. DashboardPath is the URL from which the web dashboard will be available.

As the last step, attach the Kero middleware to your web server:

keromw.Mount(r, k, gin.Accounts{
    os.Getenv("KERO_ADMIN_USER"): os.Getenv("KERO_ADMIN_PASS")
})

Mount adds the required tracking middleware and exposes the DashboardPath route. Access to the dashboard itself is protected with HTTP Basic Auth using Gin's built-in middleware.

After starting your web server you can now access the dashboard at /_kero.

Full Gin example

package main

import (
    "os"

    "github.com/gin-gonic/gin"

    "github.com/josip/kero"
    keromw "github.com/josip/kero/keroginmw"
)

func Main() {
    r := gin.New()
    k, _ := kero.New(
        kero.WithDBPath("./kero-stats"),
        kero.WithDashboardPath("/_kero"),
        kero.WithRequestMeasurements(true),
        kero.WithWebAssetsIgnored(true),
        kero.WithBotsIgnored(true),
        kero.WithPixelPath("/track.gif")
    )
    defer k.Close()

    keromw.Mount(r, k, gin.Accounts{
        os.Getenv("KERO_ADMIN_USER"): os.Getenv("KERO_ADMIN_PW"),
    })


    r.GET("/hello", func(ctx *gin.Context) {
        ctx.String(200, "Hello")
    })

    r.Run()
}

Full Fiber example

package main

import (
    "os"

    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/basicauth"

    "github.com/josip/kero"
    keromw "github.com/josip/kero/kerofibermw"
)

func Main() {
    app := fiber.New()
    k, _ := kero.New(
        kero.WithDBPath("./kero-stats"),
        kero.WithDashboardPath("/_kero"),
        kero.WithRequestMeasurements(true),
        kero.WithWebAssetsIgnored(true),
        kero.WithBotsIgnored(true),
        kero.WithPixelPath("/track.gif")
    )
    defer k.Close()

    keromw.Mount(app, k, basicauth.Config{
        Users: {
            os.Getenv("KERO_ADMIN_USER"): os.Getenv("KERO_ADMIN_PW"),
        },
    })

    app.Get("/hello", func(ctx *fiber.Ctx) error {
        return ctx.SendString("Hello")
    })

    app.Listen(":8080")
}

Want to see support for other HTTP frameworks? Create a ticket or submit a PR :octocat:.

Configuration

When creating a new Kero instance you can configure and toggle a number of features:

• WithDBPath(string): path to the database, required
• WithRetention(time.Duration): for how long should be the data stored. Defaults to 15 days
• WithDashboardPath(string): path to the dashboard URL. Defaults to /_kero.
• WithPixelPath(string): path to the pixel tracker. Response is always a 1x1px GIF. If empty, the tracker is disabled. Empty by default.
• WithGeoIPDB(string): path to the GeoLite2 database (.mmdb file) used for reverse geocoding of IP addresses. If empty, geocoding is disabled. Empty by default.
• WithRequestMeasurements(bool): controls if request duration should be tracked to provide "Slowest routes". false by default.
• WithWebAssetsIgnored(bool): controls if requests to .css/.js/etc. files should be ignored see godoc for full list. false by default.
• WithBotsIgnored(bool): controls if requests from know bots and http libraries should be ignored. false by defaults.
• WithDntIgnored(bool): controls if the value of DNT header should be respected or not. false by default.

Recommended configuration:

kero.New(
    kero.WithDBPath("./kero"),
    kero.WithDashboardPath("/_kero"),
    kero.WithPixelPath("/visit.gif"),
    kero.WithGeoIPDB("./GeoLite2-City.mmdb"),
    kero.WithRequestMeasurements(true),
    kero.WithWebAssetsIgnored(true),
    kero.WithDntIgnored(true),
)

Tracked visitor data

Availability and accuracy of the data collected varies and should be considered as best-effort since browsers themselves and user-installed extensions can introduce noisy data.

• Browser name and its version
• OS name and its version
• Device name
• Device form factor (phone, tablet, desktop, bot)
• Referrer (based on the HTTP header) and UTM query parameters
• Country, region and city based on user's IP address (disabled by default)
• Visitor ID (see below)

IP addresses and full User-Agent strings are nor stored nor logged in any manner by Kero.

How are visitors counted?

Each visitor is assigned a hashed ID that encodes their IP address, Accept-Encoding, Accept-Language and User-Agent HTTP headers.

These values are not guaranteed to be unique even between consecutive visits of the same user so they provide an approximative but indicative data, while staying privacy-friendly as much as possible.

Value of the DNT header might be ignored.

Tracked request metadata

• Request duration, if enabled
• Route (ie. /user/:id vs /user/kero; not tracked with Fiber)
• Distinction whether request was made by a web browser or programatically via different HTTP client libraries

Data storage

Kero is using TSDB from Prometheus to store the data on the disk.

Who's using Kero

• Linkship

Documentation

Index

• Constants
• Variables
• type AggregatedMetric
• type AggregationMethod
• type BarChartData
  • func (bcd *BarChartData) FormattedTimestamp() string
• type Dashboard
  • func (d *Dashboard) LoadData(k *Kero, timeframe string)
  • func (d *Dashboard) Write(wr io.Writer) error
• type DashboardStat
• type GroupMetricBy
• type Kero
  • func New(options ...KeroOption) (*Kero, error)
  • func (k *Kero) AggregateDistinct(metricName string, groupBy GroupMetricBy, labelFilters MetricLabels, ...) ([]AggregatedMetric, error)
  • func (k *Kero) Close() error
  • func (k *Kero) Count(metric string, start int64, end int64) int
  • func (k *Kero) CountDistinctByVisitor(metricName string, groupBy GroupMetricBy, labelFilters MetricLabels, ...) ([]AggregatedMetric, error)
  • func (k *Kero) CountDistinctByVisitorAndLabel(metric string, label string, labelFilters MetricLabels, start int64, end int64) ([]AggregatedMetric, error)
  • func (k *Kero) CountHistogram(metric string, start int64, end int64) [][2]int64
  • func (k *Kero) CountVisitors(metric string, labelFilters MetricLabels, start int64, end int64) (int, error)
  • func (k *Kero) MeasureHttpRequest(req TrackedHttpReq, handler func())
  • func (k *Kero) Query(metric string, labelFilters MetricLabels, start int64, end int64) ([]Metric, error)
  • func (k *Kero) ShouldTrackHttpRequest(path string) bool
  • func (k *Kero) Track(metric string, labels MetricLabels, value float64) error
  • func (k *Kero) TrackHttpRequest(req TrackedHttpReq) error
  • func (k *Kero) TrackOne(metric string, labels MetricLabels) error
  • func (k *Kero) TrackOneWithRequest(metric string, labels MetricLabels, req TrackedHttpReq) error
  • func (k *Kero) TrackWithRequest(metric string, labels MetricLabels, value float64, req TrackedHttpReq) error
  • func (k *Kero) VisitorsHistogram(metric string, filters MetricLabels, start int64, end int64) [][2]int64
• type KeroOption
  • func WithBotsIgnored(value bool) KeroOption
  • func WithDB(dbPath string) KeroOption
  • func WithDashboardPath(path string) KeroOption
  • func WithDntIgnored(value bool) KeroOption
  • func WithGeoIPDB(geoIPDBPath string) KeroOption
  • func WithPixelPath(path string) KeroOption
  • func WithRequestMeasurements(value bool) KeroOption
  • func WithRetention(duration time.Duration) KeroOption
  • func WithWebAssetsIgnored(value bool) KeroOption
• type LabelFormatter
• type Metric
• type MetricLabels
• type TrackedHttpReq
  • func TrackedRequestFromHttp(httpReq *http.Request) TrackedHttpReq
• type Trend
  • func (t *Trend) PercentChange() float64

Constants

const (
	AggregateByHour    time.Duration = time.Hour
	AggregateByDay                   = time.Hour * 24
	AggregateByWeek                  = time.Hour * 24 * 7
	AggregateByMonth                 = time.Hour * 24 * 31
	AggregateByQuarter               = time.Hour * 24 * 31 * 3
	AggregateByYear                  = time.Hour * 24 * 31 * 12
)

const BrowserDeviceLabel = "$browser_device"

const BrowserFormFactorLabel = "$browser_form_factor"

const BrowserNameLabel = "$browser_name"

const BrowserOSLabel = "$browser_os"

const BrowserOSVersionLabel = "$browser_os_version"

const BrowserVersionLabel = "$browser_version"

const CityLabel = "$city"

const ClickIdFbLabel = "$clid_fb"

const ClickIdGoogleLabel = "$clid_go"

const ClickIdMsLabel = "$clid_ms"

const ClickIdTwLabel = "$clid_tw"

const CountryLabel = "$country"

const FormFactorBot = "bot"

const FormFactorDesktop = "desktop"

const FormFactorMobile = "mobile"

const FormFactorTablet = "tablet"

const HttpMethodLabel = "$http_method"

const HttpPathLabel = "$http_path"

const HttpReqDurationMetricName = "http_req_dur"

const HttpReqMetricName = "http_req"

const HttpRouteLabel = "$http_route"

const IsBotLabel = "$is_bot"

const MetricName = labels.MetricName

const ReferrerDomainLabel = "$referrer_domain"

const ReferrerLabel = "$referrer"

const RegionLabel = "$region"

const UTMCampaignLabel = "$utm_campaign"

const UTMContentLabel = "$utm_content"

const UTMMediumLabel = "$utm_medium"

const UTMSourceLabel = "$utm_source"

const UTMTermLabel = "$utm_term"

const VisitorIdLabel = "$visitor_id"

Variables

var DashboardWebAssets embed.FS

var DefaultDashboard = Dashboard{
	Title:      "App stats",
	ShowFooter: true,
	Rows: [][]DashboardStat{
		{
			{
				Title:            "Top pages",
				UnitDisplayLabel: "Page",
				CountLabel:       "Visitors",

				QueryMetric:      HttpReqMetricName,
				QueryLabel:       HttpPathLabel,
				QueryByVisitor:   true,
				QueryExcludeBots: true,
			},
			{
				Title:            "Top referrals",
				UnitDisplayLabel: "Site",
				CountLabel:       "Visitors",

				QueryMetric:      HttpReqMetricName,
				QueryLabel:       ReferrerDomainLabel,
				QueryByVisitor:   true,
				QueryExcludeBots: true,
			},
			{
				Title:            "Top locations",
				UnitDisplayLabel: "Country",
				CountLabel:       "Visitors",

				QueryMetric:      HttpReqMetricName,
				QueryLabel:       CountryLabel,
				QueryByVisitor:   true,
				QueryExcludeBots: true,

				FormatLabel: func(am AggregatedMetric) string {
					cc := am.Label
					return string(0x1F1E6+rune(cc[0])-'A') + string(0x1F1E6+rune(cc[1])-'A') + " " + cc
				},
			},
		},

		{
			{
				Title:            "Top form factors",
				UnitDisplayLabel: "Form factor",
				CountLabel:       "Visitors",

				QueryMetric:    HttpReqMetricName,
				QueryLabel:     BrowserFormFactorLabel,
				QueryByVisitor: true,

				FormatLabel: func(am AggregatedMetric) string {
					if emoji, ok := formFactorEmojis[am.Label]; ok {
						return emoji
					}

					return am.Label
				},
			},
			{
				Title:            "Top browsers",
				UnitDisplayLabel: "Browser",
				CountLabel:       "Visitors",

				QueryMetric:      HttpReqMetricName,
				QueryLabel:       BrowserNameLabel,
				QueryByVisitor:   true,
				QueryExcludeBots: true,
			},

			{
				Title:            "Top operating systems",
				UnitDisplayLabel: "Operating system",
				CountLabel:       "Visitors",

				QueryMetric: HttpReqMetricName,
				QueryFilters: MetricLabels{
					(BrowserFormFactorLabel + "!="): FormFactorBot,
				},
				QueryLabel:       BrowserOSLabel,
				QueryByVisitor:   true,
				QueryExcludeBots: true,
			},
		},

		{
			{
				Title:            "Top routes",
				UnitDisplayLabel: "Route",
				CountLabel:       "Visitors",

				QueryMetric:    HttpReqMetricName,
				QueryLabel:     HttpRouteLabel,
				QueryByVisitor: true,
			},

			{
				Title:            "Slowest routes",
				UnitDisplayLabel: "Route",
				CountLabel:       "avg ms",

				QueryMetric:      HttpReqDurationMetricName,
				QueryGroupBy:     groupByRoute,
				QueryAggregateBy: AggregateAvg,
			},

			{
				Title:            "Top bots and libraries",
				UnitDisplayLabel: "Bot",
				CountLabel:       "Rqs",

				QueryMetric: HttpReqMetricName,
				QueryFilters: MetricLabels{
					BrowserFormFactorLabel: FormFactorBot,
				},
				QueryLabel: BrowserNameLabel,
			},
		},
	},
}

var Pixel, _ = base64.StdEncoding.DecodeString("R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==")

1x1px transparent GIF. Source must be StackOverflow

var PixelSize = int64(len(Pixel))

Types

type AggregatedMetric

type AggregatedMetric struct {
	Label string  `json:"label"` // Metric label as it was recorded or formatted with GroupMetricBy
	Value float64 `json:"value"`
}

type AggregationMethod

type AggregationMethod int

const (
	AggregateCount AggregationMethod = iota // Aggregates by counting number of matched events
	AggregateSum                            // Aggregates by summing values of matched events
	AggregateAvg                            // Aggregates by calculating an average value of matched events
)

type BarChartData

type BarChartData struct {
	Timestamp int64
	Value     int64
	Percent   float64
}

func (*BarChartData) FormattedTimestamp

func (bcd *BarChartData) FormattedTimestamp() string

type Dashboard

type Dashboard struct {
	Title      string
	ShowFooter bool
	BasePath   string

	VisitorsChartData []BarChartData
	VisitorsTrend     Trend

	ViewsChartData []BarChartData
	ViewsTrend     Trend
	Rows           [][]DashboardStat
}

func (*Dashboard) LoadData

func (d *Dashboard) LoadData(k *Kero, timeframe string)

func (*Dashboard) Write

func (d *Dashboard) Write(wr io.Writer) error

type DashboardStat

type DashboardStat struct {
	Title            string
	UnitDisplayLabel string
	CountLabel       string

	QueryMetric      string
	QueryLabel       string
	QueryGroupBy     GroupMetricBy
	QueryFilters     MetricLabels
	QueryByVisitor   bool
	QueryAggregateBy AggregationMethod
	QueryExcludeBots bool

	FormatLabel LabelFormatter

	Data []AggregatedMetric
}

type GroupMetricBy

type GroupMetricBy func(m Metric) string

type Kero

type Kero struct {
	DashboardPath          string
	PixelPath              string
	MeasureRequestDuration bool
	IgnoreCommonPaths      bool
	IgnoreBots             bool
	IgnoreDNT              bool

	// path prefixes to which requests will be ignored. see file for default list.
	IgnoredPrefixes []string
	// path suffixes to which requestw will be ignored. see file for default list.
	IgnoredSuffixes []string
	// user-agent values to be ignored. see file for default list.
	IgnoredAgents []string
	// contains filtered or unexported fields
}

func New

func New(options ...KeroOption) (*Kero, error)

New automatically creates a new Kero database on-disk if one doesn't exist already. See WithXXX functions for option configuration.

func (*Kero) AggregateDistinct

func (k *Kero) AggregateDistinct(
	metricName string,
	groupBy GroupMetricBy,
	labelFilters MetricLabels,
	aggregateBy AggregationMethod,
	start int64,
	end int64,
) ([]AggregatedMetric, error)

AggregateDistinct provides advanced options to query the database. Data can be filtered using the metric name or any combination of labels (including negation). Additionally data can be grouped by a calculated key and aggregated using count, sum or average. Example:

 func QueryExample() {
   k, _ := kero.New(kero.WithDatabasePath("./kero"))
   data, _ := k.AggregateDistinct(
     "http_req", // get all "http_req" metrics
     func(m Metric) string { return m.Labels["$city"] }, // group them by city
     MetricLabels{ "country": "CH", "region!=": "ZH" } // filtering only requests coming from Switzerland, from any region except Zurich,
     kero.AggregateCount, // and return only the count of matched rows
     0, // from beginning of time
     time.Now().Unix(), // ...until now
	  )

   fmt.Println("Found", len(data), "records:")
   for _, row := range data {
     fmt.Println(row.Value, "visitors from", row.Label)
   }
 }

Results are sorted by highest value first.

func (*Kero) Close

func (k *Kero) Close() error

func (*Kero) Count

func (k *Kero) Count(metric string, start int64, end int64) int

Count is an optimized version of AggregateDistinct counting occurrences of a metric in the specified timeframe.

func (*Kero) CountDistinctByVisitor

func (k *Kero) CountDistinctByVisitor(
	metricName string,
	groupBy GroupMetricBy,
	labelFilters MetricLabels,
	start int64,
	end int64,
) ([]AggregatedMetric, error)

CountDistinctByVisitor returns a number of unique visitors for which the matching events have been tracked.

func (*Kero) CountDistinctByVisitorAndLabel

func (k *Kero) CountDistinctByVisitorAndLabel(
	metric string,
	label string,
	labelFilters MetricLabels,
	start int64,
	end int64,
) ([]AggregatedMetric, error)

CountDistinctByVisitorAndLabel is a convenience method that's groups metrics simply by using the specified label. If filtering by [Kero.HttpRouteLabel], requests are grouped by both the HTTP method and the route, this way a distinction can be made between `GET /user/:id` and `POST /user/:id`. Events without the label itselfz are excluded from the count.

func (*Kero) CountHistogram

func (k *Kero) CountHistogram(metric string, start int64, end int64) [][2]int64

CountHistogram returns metric count within the specified timeframe for each time subdivision based on the duration between start and end time. Subdivisions are determined as follows:

• duration up to 3 days: 72 subdivisions each of 1 hour
• duration up to 31 days (ie. 1 month): 1 day
• duration up to 93 days (ie. 3 months): 1 week
• for durations longer than 3 months: 1 month

func (*Kero) CountVisitors

func (k *Kero) CountVisitors(metric string, labelFilters MetricLabels, start int64, end int64) (int, error)

CountVisitors counts number of unique visitors for which an event with matching filters has been tracked within the specified timeframe.

func (*Kero) MeasureHttpRequest

func (k *Kero) MeasureHttpRequest(req TrackedHttpReq, handler func())

func (*Kero) Query

func (k *Kero) Query(metric string, labelFilters MetricLabels, start int64, end int64) ([]Metric, error)

Query looks for matching metrics within the specified timeframe.

func (*Kero) ShouldTrackHttpRequest

func (k *Kero) ShouldTrackHttpRequest(path string) bool

func (*Kero) Track

func (k *Kero) Track(metric string, labels MetricLabels, value float64) error

func (*Kero) TrackHttpRequest

func (k *Kero) TrackHttpRequest(req TrackedHttpReq) error

func (*Kero) TrackOne

func (k *Kero) TrackOne(metric string, labels MetricLabels) error

func (*Kero) TrackOneWithRequest

func (k *Kero) TrackOneWithRequest(metric string, labels MetricLabels, req TrackedHttpReq) error

func (*Kero) TrackWithRequest

func (k *Kero) TrackWithRequest(metric string, labels MetricLabels, value float64, req TrackedHttpReq) error

func (*Kero) VisitorsHistogram

func (k *Kero) VisitorsHistogram(metric string, filters MetricLabels, start int64, end int64) [][2]int64

VisitorsHistogram returns a number of unique visitors per time subdivision in the specified timeframe. See Kero.CountHistogram for reference on time subdivisions.

type KeroOption

type KeroOption func(*Kero) error

func WithBotsIgnored

func WithBotsIgnored(value bool) KeroOption

WithBotsIgnored sets whether Kero should ignore requests made by bots, web scrapers and HTTP libraries. Bots and libraries are detected using their `User-Agent` header.

func WithDB

func WithDB(dbPath string) KeroOption

WithDB sets the location of the database folder. Automatically created if it doesn't exist.

func WithDashboardPath

func WithDashboardPath(path string) KeroOption

WithDashboardPath sets the URL at which the dashboard will be mounted. Defaults to `"/_kero"`.

func WithDntIgnored

func WithDntIgnored(value bool) KeroOption

WithDntIgnored sets whether Kero should ignore value of DNT header. If configured to ignore, requests with DNT: 1 (opt-out of tracking) will be still tracked.

func WithGeoIPDB

func WithGeoIPDB(geoIPDBPath string) KeroOption

WithGeoIPDB loads the MaxMind GeoLine2 and GeoIP2 database for IP-reverse lookup of visitors. If not provided, IP reverse-lookup is disabled.

func WithPixelPath

func WithPixelPath(path string) KeroOption

WithPixelPath defines the route at which the pixel tracker will be available to external applications. The pixel can be referenced from static websites or services not directly served by the Go server. Requests referer header will be used as the path, with other headers and query parameters used unchanged. Response of the pixel path will be a 1x1px transparent GIF.

func WithRequestMeasurements

func WithRequestMeasurements(value bool) KeroOption

WithRequestMeasurements sets whether Kero should automatically measure response of handlers time.

func WithRetention

func WithRetention(duration time.Duration) KeroOption

WithRetention defines the how long data points should be kept. Defaults to 15 days.

func WithWebAssetsIgnored

func WithWebAssetsIgnored(value bool) KeroOption

WithWebAssetsIgnored sets whether Kero should ignore requests made to common web assets.

Ignored paths include:

• .css and .js files
• images (.png, .svg, .jpg, .webp, etc.)
• fonts (.woff2, .ttf, .otf, etc.)
• common URLs accessed by web scrapers (.php, .asp, etc.)
• any path starting with /., /_, /wp- and /public.

Kero's resources and paths are always excluded.

type LabelFormatter

type LabelFormatter func(AggregatedMetric) string

type Metric

type Metric struct {
	Ts     int64        `json:"timestamp"`
	Name   string       `json:"name"`
	Labels MetricLabels `json:"labels"`
	Value  float64      `json:"value"`
}

type MetricLabels

type MetricLabels map[string]string

type TrackedHttpReq

type TrackedHttpReq struct {
	Method     string
	Path       string
	Headers    http.Header
	Query      url.Values
	Route      string
	ClientIp   string
	RemoteAddr string
}

func TrackedRequestFromHttp

func TrackedRequestFromHttp(httpReq *http.Request) TrackedHttpReq

type Trend

type Trend struct {
	CurrentValue  int64
	PreviousValue int64
}

func (*Trend) PercentChange

func (t *Trend) PercentChange() float64