tcg

README

The Transit Connection Generator (TCG). TCG contains two sub-systems/packages:

1. Transit agent - connects and sends metrics to Groundwork Monitor
2. Controller service - an http server for external control of agent

Table of Contents

1. Dependencies
2. Building
3. Running
4. Docker
5. Testing
6. Environment variables

Dependencies

The TCG project is built with Go Modules. See go.mod for a list of dependencies. Here are some main frameworks used by this project:

1. Gin Web Framework

   Gin is a web framework written in Go (Golang). It features a martini-like API with much better performance, up to 40 times faster.

    github.com/gin-gonic/gin
   

2. Sessions

   Gin middleware for session management with multi-backend support.

    github.com/gin-gonic/contrib/sessions
   

3. NATS Streaming System

   About NATS

    github.com/nats-io/go-nats-streaming
    github.com/nats-io/nats-streaming-server/server
    github.com/nats-io/nats-streaming-server/stores
   

4. Envconfig

   Package envconfig implements decoding of environment variables based on a user defined specification. A typical use is using environment variables for configuration settings.

    github.com/kelseyhightower/envconfig
   

5. Go-Cache

   Go-Cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major advantage is that, being essentially a thread-safe map[string]interface{} with expiration times, it doesn't need to serialize or transmit its contents over the network. Any object can be stored, for a given duration or forever, and the cache can be safely used by multiple goroutines.

    github.com/patrickmn/go-cache
   

6. Testify

   Go code (golang) set of packages that provide many tools for testifying that your code will behave as you intend.

    github.com/stretchr/testify
   

7. Zerolog

   The zerolog package provides a fast and simple logger dedicated to JSON output.

    github.com/rs/zerolog
   

8. Gopsutil

   The challenge is porting all psutil functions on some architectures.

    github.com/shirou/gopsutil
   

9. Gin-Swagger

   Gin Gonic middleware to automatically generate RESTful API documentation with Swagger 2.0.

    github.com/swaggo/gin-swagger
   

   Generate 'docs.go' for Swagger UI

    $ swag init
   

   Swagger url:

    {host}:{port}/swagger/index.html
   

Building

$ cd tcg
$ go build .

Building tcg shared module:

$ go build -ldflags "-X 'github.com/gwos/tcg/config.buildTime=`date --rfc-3339=s`' -X 'github.com/gwos/tcg/config.buildTag=<TAG>'" -buildmode=c-shared -o libtransit/libtransit.so libtransit/libtransit.go

or use Makefiles

Building Connectors:

LINUX:

$ cd connectors/<connector>
$ go build -ldflags "-X 'github.com/gwos/tcg/config.buildTime=`date --rfc-3339=s`' -X 'github.com/gwos/tcg/config.buildTag=<TAG>'"

OS X:

$ cd connectors/<connector>
$ go build -ldflags "-X 'github.com/gwos/tcg/config.buildTime=`date -u +"%Y-%m-%dT%H:%M:%SZ"`' -X 'github.com/gwos/tcg/config.buildTag=<TAG>'"

Building Connectors for OS Targets (Cross Compiling)

env GOOS=linux GOARCH=386 go build -ldflags "-X 'github.com/gwos/tcg/config.buildTime=`date -u +"%Y-%m-%dT%H:%M:%SZ"`' -X 'github.com/gwos/tcg/config.buildTag=8.1.0.1'"

To view supported platforms use commands

go tool dist list
go tool dist list -json
go tool dist list -json | jq '.[] | select(.CgoSupported == true)'

Installing as a service

To enable:

sudo systemctl enable tcg-elastic

To start:

sudo systemctl start tcg-elastic

Show status:

sudo systemctl status tcg-elastic

To stop:

sudo systemctl stop tcg-elastic

To disable:

sudo systemctl disable tcg-elastic

To reconfigure:

sudo systemctl daemon-reload

To tail:

journalctl -f -u tcg-elastic

Running

$ cd tcg
$ go run .

Docker

Build image:

$ docker build -t groundworkdevelopment/tcg --build-arg GITHUB_TOKEN .

Testing

The gotests tool can generate Go tests.

Run package tests:

With logs:

$ go test -v ./<package_one>/ ./<package_two>/

Run integration tests:

For integration tests you must provide environment variables for Groundwork Connection. Also have to deal with TLS certs: get it in local trust storage or suppress check.

$ TCG_TLS_CLIENT_INSECURE=TRUE \
    TCG_GWCONNECTIONS_0_USERNAME=remote TCG_GWCONNECTIONS_0_PASSWORD=remote \
    go test -v ./integration/

Examples:

$ go test -v ./config/ ./services/

$ go test -v ./libtransit/

$ go test -v $(go list ./... | grep -v tcg/integration)

$ GOFLAGS="-count=1" \
    TCG_TLS_CLIENT_INSECURE=TRUE \
    TCG_GWCONNECTIONS_0_USERNAME=remote TCG_GWCONNECTIONS_0_PASSWORD=remote \
    TCG_CONNECTOR_LOGCOLORS=TRUE TCG_CONNECTOR_LOGLEVEL=3 \
    OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
    TCG_CONNECTOR_AGENTID=TEST11 \
    go test -failfast -v ./integration/

$ GOFLAGS="-count=1" \
    TCG_TLS_CLIENT_INSECURE=TRUE \
    TCG_GWCONNECTIONS_0_USERNAME=remote TCG_GWCONNECTIONS_0_PASSWORD=remote \
    TCG_CONNECTOR_LOGCOLORS=TRUE TCG_CONNECTOR_LOGLEVEL=3 \
    __OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
    __TCG_HTTP_CLIENT_TIMEOUT_GW=120s \
    __TEST_FLAG_CLIENT=true \
    TEST_RESOURCES_COUNT=40 TEST_SERVICES_COUNT=100 \
    TCG_CONNECTOR_BATCHMETRICS=1s TCG_CONNECTOR_BATCHMAXBYTES=204800  TCG_CONNECTOR_NATSMAXPAYLOAD=40920 \
    go test -benchtime=10x -benchmem -run=^$ -bench ^BenchmarkE2E$  ./integration/ \
    | grep _STATS

Environment variables

LIBTRANSIT

Defines the path to libtransit.so library in docker container and tests.

$ export LIBTRANSIT=/path/to/libtransit.so

TCG config variables

By default, the config file is looked for in the work directory as tcg_config.yaml.

The path to config file and any config option can be overridden with env vars:

$ export TCG_CONFIG=/path/to/tcg_config.yaml
$ export TCG_CONNECTOR_NATSSTORETYPE=MEMORY

For more info see package config and tests.

Other variables

There are additional variables supported:

* OTEL_EXPORTER_OTLP_ENDPOINT=http://jaegertracing:4317
* TCG_HTTP_CLIENT_TIMEOUT=10s
* TCG_HTTP_CLIENT_TIMEOUT_GW=120s
* TCG_SUPPRESS_DOWNTIMES=true
* TCG_SUPPRESS_EVENTS=true
* TCG_SUPPRESS_INVENTORY=true
* TCG_SUPPRESS_METRICS=true

Run golangci-lint locally:

$ go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.54.2
$ ~/go/bin/golangci-lint --config ./.golangci.yaml run ./... --deadline=2m