query

README

Query TXN2 data by Account, Model and index pattern. Save a Query and execute a saved Query.

Configuration

Configuration is inherited from txn2/micro. The following configuration is specific to query:

Flag	Environment Variable	Description
-esServer	ELASTIC_SERVER	Elasticsearch Server (default "http://elasticsearch:9200")
-provisionServer	PROVISION_SERVER	Provision Server (txn2/provision)
-authCache	AUTH_CACHE	Seconds to cache key (BasicAuth) authentication.

Routes

Method	Route Pattern	Description
POST	run/:account	Run a query to test it. This action does not save the query.
GET	exec/:account/:id	Executes a saved query.
POST	upsert/:account	Upsert (save) a query.
GET	get/:account/:id	Get a saved query.
POST	search/:account	Search for saved queries.

Local Development

The project includes a docker-compose.yml file with Elasticsearch, Logstash, Kibana, Cerebro, txn2/rxtx, txn2/rtbeat, txn2/provision and txn2/tm:

docker-compose up

Add test account with txn2/provision:

curl -X POST \
  http://localhost:8070/account \
  -H 'Content-Type: application/json' \
  -d '{
    "id": "test",
    "description": "This is a test account",
    "display_name": "Test Organization",
    "active": true,
    "access_keys": [
        {
            "name": "test",
            "key": "PDWgYr3bQGNoLptBRDkLTGQcRmCMqLGRFpXoXJ8xMPsMLMg3LHvWpJgDu2v3LYBA",
            "description": "Generic access key 2",
            "active": true
        }
    ],
    "modules": [
        "wx",
        "data_science"
    ]
}'

Add an admin User with with txn2/provision given access to the test account:

curl -X POST \
  http://localhost:8070/user \
  -H 'Content-Type: application/json' \
  -d '{
	"id": "test",
	"description": "Test User admin",
	"display_name": "Test User",
	"active": true,
	"sysop": false,
	"password": "eWidL7UtiWJABHgn8WA",
	"sections_all": false,
	"sections": ["api", "config", "data"],
	"accounts": ["test"],
	"admin_accounts": ["test"]
}'

Get a user Token from txn2/provision with the User's id and password:

TOKEN=$(curl -s -X POST \
          http://localhost:8070/authUser?raw=true \
          -d '{
        	"id": "test",
        	"password": "eWidL7UtiWJABHgn8WA"
        }') && echo $TOKEN

Insert a sample Model into the test account using txn2/tm running on port 8085:

curl -X POST http://localhost:8085/model/test \
 -H "Authorization: Bearer $TOKEN" \
 -H 'Content-Type: application/json' \
 -d '{
  "machine_name": "some_metrics",
  "display_name": "Some Metrics",
  "description": "A sample model describing some metrics sent through rxtx",
  "fields": [
    {
      "machine_name": "device_id",
      "display_name": "Device ID",
      "data_type": "keyword"
    },
    {
      "machine_name": "random_number",
      "display_name": "Random Number",
      "data_type": "integer"
    },
    {
      "machine_name": "another_number",
      "display_name": "Another Number",
      "data_type": "integer"
    }
  ]
}'

Within Elasticsearch there is now a template _template/test-data-some_metrics for the account test describing txn2/rxtx/txn2/rtbeat inbound data matching index pattern test-data-some_metrics-*. Send some sample data to Elasticsearch through txn2/rxtx and wait for the batch interval (specified in the docker-compose) to complete:

curl -X POST \
  http://localhost:8090/rx/test/some_metrics/sample-data \
  -H 'Content-Type: application/json' \
  -d "{
  \"device_id\": \"12345\",
  \"random_number\": \"${RANDOM}\",
  \"another_number\": 12345
}"

The fields in the data sent to txn2/rxtx should match the fields described in the txn2/tm Model. Although the value for "random_number" is represented here as a string, the template mapping added with the Model instructs Elasticsearch to index it as an integer.

Example Queries

Run query from source. Configure it to use the services running from docker-compose above.

go run ./cmd/query.go --esServer=http://localhost:9200 --provisionServer=http://localhost:8070 --tokenKey="somesharedkey"

Run a Query:

curl -X POST \
  http://localhost:8080/run/test \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "machine_name": "count_some_metrics",
    "display_name": "Get all records from the some_metrics index.",
    "description": "Return all matches",
    "model": "some_metrics",
    "idx_pattern": "-ts-*",
    "query": {
      "size": 0,
	  "query": {
	    "match_all": {}
	  }
	}
}'

Upsert a Query:

(must have admin access to account)

curl -X POST \
  http://localhost:8080/upsert/test \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "machine_name": "count_some_metrics",
    "display_name": "Get all records from the some_metrics index.",
    "description": "Return all matches",
    "model": "some_metrics",
    "idx_pattern": "-ts-*",
    "query": {
      "size": 0,
	  "query": {
	    "match_all": {}
	  }
	}
}'

Search for queries:

curl -X POST \
  http://localhost:8080/search/test \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
  "size": 10,
  "query": {
    "match_all": {}
  }
}'

Get a Query:

curl -X GET \
  http://localhost:8080/get/test/count_some_metrics \
  -H "Authorization: Bearer $TOKEN"

Execute a Query:

curl -X GET \
  http://localhost:8080/exec/test/count_some_metrics \
  -H "Authorization: Bearer $TOKEN"

Execute a Query with BasicAuth using an Account AccessKey:

curl -X GET \
  http://localhost:8080/exec/test/count_some_metrics \
  -H 'Authorization: Basic dGVzdDpQRFdnWXIzYlFHTm9McHRCUkRrTFRHUWNSbUNNcUxHUkZwWG9YSjh4TVBzTUxNZzNMSHZXcEpnRHUydjNMWUJB'

Query Templates

Query templates use Go's build in template system along with Sprig. The following example tests a query with /run and demonstrates a query string override of default values provided in the "parameters" section:

curl -X POST \
  'http://localhost:8080/run/test?size=0&idx=2019.%2A' \
  -H 'Authorization: Basic dGVzdDpQRFdnWXIzYlFHTm9McHRCUkRrTFRHUWNSbUNNcUxHUkZwWG9YSjh4TVBzTUxNZzNMSHZXcEpnRHUydjNMWUJB' \
  -d '{
    "machine_name": "get_some_metrics",
    "display_name": "Get records from the some_metrics index.",
    "description": "Get some_metrics",
    "model": "some_metrics",
    "idx_pattern": "-ts-{{ index . \"idx\" }}",
    "query_template": "{\"size\": {{ index . \"size\" }},\"query\": {\"match_all\": {}}}",
	"parameters": [
		{
			"machine_name": "size",
			"default_value": "1"
		},
		{
			"machine_name": "idx",
			"default_value": "2019.*"
		}
	]
}'

Release Packaging

Build test release:

goreleaser --skip-publish --rm-dist --skip-validate

Build and release:

GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist

Documentation

Overview

Package query implements an api for adding and executing Lucene queries associate with an account.

Index

• Constants
• func GetQueryTemplateMapping() es.IndexTemplate
• type Api
  • func NewApi(cfg *Config) (*Api, error)
  • func (a *Api) ExecuteQuery(account string, query Query, c *gin.Context) (int, *es.Obj, *es.ErrorResponse, error)
  • func (a *Api) ExecuteQueryHandlerF(system bool) gin.HandlerFunc
  • func (a *Api) GetQuery(account string, id string) (int, *Result, error)
  • func (a *Api) GetQueryHandler(c *gin.Context)
  • func (a *Api) RunQueryHandler(c *gin.Context)
  • func (a *Api) SearchQueries(account string, searchObj *es.Obj) (int, SearchResults, *es.ErrorResponse, error)
  • func (a *Api) SearchQueryHandler(c *gin.Context)
  • func (a *Api) UpsertQuery(account string, query *Query) (int, es.Result, *es.ErrorResponse, error)
  • func (a *Api) UpsertQueryHandler(c *gin.Context)
• type Config
• type Query
• type Result
• type SearchResults
• type SearchResultsAck

Constants

const IdxQuery = "queries"

Functions

func GetQueryTemplateMapping

func GetQueryTemplateMapping() es.IndexTemplate

GetModelsTemplateMapping

Types

type Api

type Api struct {
	*Config
}

Api

func NewApi

func NewApi(cfg *Config) (*Api, error)

NewApi

func (*Api) ExecuteQuery

func (a *Api) ExecuteQuery(account string, query Query, c *gin.Context) (int, *es.Obj, *es.ErrorResponse, error)

ExecuteQuery

func (*Api) ExecuteQueryHandlerF

func (a *Api) ExecuteQueryHandlerF(system bool) gin.HandlerFunc

ExecuteQueryHandler

func (*Api) GetQuery

func (a *Api) GetQuery(account string, id string) (int, *Result, error)

GetModel

func (*Api) GetQueryHandler

func (a *Api) GetQueryHandler(c *gin.Context)

GetQueryHandler

func (*Api) RunQueryHandler

func (a *Api) RunQueryHandler(c *gin.Context)

ExecuteQueryHandler

func (*Api) SearchQueries

func (a *Api) SearchQueries(account string, searchObj *es.Obj) (int, SearchResults, *es.ErrorResponse, error)

SearchQueries

func (*Api) SearchQueryHandler

func (a *Api) SearchQueryHandler(c *gin.Context)

SearchQueryHandler

func (*Api) UpsertQuery

func (a *Api) UpsertQuery(account string, query *Query) (int, es.Result, *es.ErrorResponse, error)

UpsertQuery

func (*Api) UpsertQueryHandler

func (a *Api) UpsertQueryHandler(c *gin.Context)

UpsertQueryHandler

type Config

type Config struct {
	Logger     *zap.Logger
	HttpClient *micro.Client

	// used for communication with Elasticsearch
	// if nil, HttpClient will be used.
	Elastic       *es.Client
	ElasticServer string

	// for storage and retrieval of system_ queries
	SystemIdxPrefix string
}

Config

type Query

type Query struct {
	// a lowercase under score delimited uniq id
	MachineName string `json:"machine_name" mapstructure:"machine_name"`

	// short human readable display name
	DisplayName string `json:"display_name" mapstructure:"display_name"`

	// a single sentence description
	BriefDescription string `json:"description_brief" mapstructure:"description_brief"`

	// full documentation in markdown
	Description string `json:"description" mapstructure:"description"`

	// named parsers
	Parsers []string `json:"parsers" mapstructure:"parsers"`

	// belongs to a class of queries
	QueryClass string `json:"query_class" mapstructure:"query_class"`

	// used for grouping queries
	Group string `json:"group" mapstructure:"group"`

	// used for grouping queries
	Model string `json:"model" mapstructure:"model"`

	// pattern default "-*" eg. "-someset", "-ts-2019*"
	IdxPattern string `json:"idx_pattern" mapstructure:"idx_pattern"`

	// query object
	Query *es.Obj `json:"query,omitempty" mapstructure:"query"`

	// used to describe input parameters
	Parameters []tm.Model `json:"parameters,omitempty" mapstructure:"parameters"`

	// query json
	QueryJson string `json:"query_json,omitempty" mapstructure:"query_json"`

	// if a query template is present it will take the place of
	// the query and query_json fields
	QueryTemplate string `json:"query_template,omitempty" mapstructure:"query_template"`

	// describes the query output
	ResultFields []tm.Model `json:"fields" mapstructure:"fields"`
}

Query

type Result

type Result struct {
	es.Result
	Source Query `json:"_source"`
}

Result returned from Elastic

type SearchResults

type SearchResults struct {
	es.SearchResults
	Hits struct {
		Total    int      `json:"total"`
		MaxScore float64  `json:"max_score"`
		Hits     []Result `json:"hits"`
	} `json:"hits"`
}

SearchResults

type SearchResultsAck

type SearchResultsAck struct {
	ack.Ack
	Payload SearchResults `json:"payload"`
}

SearchResultsAck