go

README

Go libraries/commands

This directory contains several libraries (pkg) and auxiliary utilities (cmd) for running Opstrace.

Run unit tests

The unit tests manage Docker containers, i.e. the Docker daemon needs to be running.

cd go
make unit-tests

Tracing

Test Tracing locally

$ make build-tracing
$ ./tracing-api -listen=localhost:8081 -TODO-ARGS
TODO LOGS

curl -v localhost:8081/metrics
TODO EXAMPLE CURL QUERY
curl -v localhost:8081/metrics

Auto-generation of mocks and interfaces

Some of the mocks and interfaces in this repo are autogenerated using go generate tool via make regenerate Makefile target.

The idea is to not to have to manually write mocks and interface specifications and also to keep the definitions in sync - the CI calls make regenerate target during the run and verifies that all changes are committed to the repo.

For interface generation we are using ifacemaker and for generating mock the mockery.

Check the comment lines in the source that start with go:generate stanzas to see how these tools are called.

Rate-limiting in ErrorTracking

Rate limiting in Error Tracking is based on this proposal. The proposal describes the architecture of the implementation and gives some context for the decisions that were made.

The key configuration option for RateLimiting is the RATE_LIMITING_CONFIG_FILE environment variable. If not set - the rate limiting is disabled, and no rate-limiting headers are set. If set - the rate-limiting code will load the configuration from the file pointed in the env variable and use it to determine limits for given projects/groups/endpoints.

While running, the code will also detect the changes to the configuration file and reload itself automatically. The reload is based on the fact that changes to the config map are reflected by kubelet in the pod. The updates are reflected withing 1-2 minutes, which means any modifications to existing limits configuration will take 1-2 minutes to appear. The reload itself does not interrupt processing of the requests, the pods do not need to be restarted.

The rate-limiting configuration for Kind clusters is generated by makefile scripts, whereas for prod/stg environments - it is stored in the infra repos.

The sample rate-limiting configuration look like following:

api_version: 1
sizings:
  default:
    errortracking_api_reads: 75
    errortracking_api_writes: 75
    tracing_bytes_write: 1024
  xl:
    errortracking_api_reads: 1000
    errortracking_api_writes: 1000
    tracing_bytes_write: 102400
  xxl:
    errortracking_api_reads: 10000
    errortracking_api_writes: 10000
    tracing_bytes_write: 202400
bindings:
  groups:
    22: xl
  projects:
    1: xl

Sizing section defines limits for given labels (here xl, xxl and default). Each sizing is a free-form string, it can be anything that is a valid dict key in yaml. The default label is special, they are used when no binding is available for a given group/project. This enforces the default rate limits on operations. The bindings section defines the sizings for individual groups/projects. In our example the group 22 and project 1 is given rate-limits defined in sizing xl.

The limits are currently defined only for two groups: read endpoints and write endpoints in a 60 seconds window (or number of requests per minute). The reason why we use per-minute window instead of per-hour is that redis-rate does not update the data stored in Redis after updating the limits config, so even though we can bump limits for given sizing, we will get inconsistent limits for the remaining part of the hour. The solution is to use PerMinute instead, so if the limits are changed, they are updated relatively quickly.