opensearch

package
v1.127.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2023 License: BSD-3-Clause Imports: 14 Imported by: 0

README

This package is a wrapper around the AWS OpenSearch package to provide a consistent way to:

  • log time series data (e.g. api-logs)
  • store and search for documents

Time Series Logs

We use a time series for example to store API logs, writing a log record at the end of each API handler, with a start timestamp and duration as well as key values like HTTP method, path, request-id, URL parameters, response code etc.

The indexed logs are then mainly used from the OpenSearch dashboard to visualise the system usage and performance and we can setup alerts on conditions to detect failures and poor performance. One can also read the logs to asist trouble shooting.

The indexed logs can also be queried from our API to provide insights to account users, but the account users do not directly interact with OpenSearch.

We use a document store and search to provide quick text searches from the API. In this case a copy of a document (e.g. an order) is stored in OpenSearch when the order is created or updated. Updates overwrite the existing document, so there will only be one copy of each document.

When a user is looking for an order, the API provides an end-point to search orders e.g. for "lcd screen", then the API does an OpenSearch query in the orders index, get N results and then read those orders from the orders table in the database (not OpenSearch) and return those results to the user.

We therefore use OpenSearch only for searching and returning a list of document ids, then read the documents from the database. A document is typically an "order" but also anything else that we need to do free text searches on.

Documentation

Index

Constants

View Source 
const TimeFormat = "2006-01-02T15:04:05Z07:00"
View Source 
const Timeout = "25s" // API Gateway times out after 30s, so return before that

Variables

This section is empty.

Functions

func ValidIndexName 

func ValidIndexName(s string) bool

Types

type Cause 

type Cause struct {
	Reason string `json:"reason,omitempty"`
	Type   string `json:"type,omitempty"`
}

type Config 

type Config struct {
	Addresses []string `json:"addresses" doc:"List of server addresses. Requires at least one, e.g. \"https://localhost:9200\" for local testing"`
	Username  string   `json:"username" doc:"User name for HTTP basic auth. Defaults to admin for local testing."`
	Password  string   `json:"password" doc:"User password for HTTP basic auth. Defaults to admin for local testing."`
}

func (*Config) Validate 

func (c *Config) Validate() error

type CreateResponse 

type CreateResponse struct {
	Error *Error `json:"error,omitempty"`
}

type DocumentStore 

type DocumentStore struct {
	// contains filtered or unexported fields
}

func (*DocumentStore) Delete 

func (ds *DocumentStore) Delete(id string) (err error)

func (*DocumentStore) Get 

func (ds *DocumentStore) Get(id string) (res *GetResponseBody, err error)

func (*DocumentStore) Search 

func (ds *DocumentStore) Search(query Query, limit int64) (res *SearchResponseHits, err error)

Search Return: 

docs will be a slice of the DocumentStore data type

func (*DocumentStore) Write 

func (ds *DocumentStore) Write(id string, data interface{}) error

data must be of type specified in Writer.TimeSeries(tmpl)

type Error 

type Error struct {
	Reason    string  `json:"reason,omitempty"`
	RootCause []Cause `json:"root_cause,omitempty"`
}

type FilterQuery 

type FilterQuery struct {
	// one of:
	Match             *QueryMatch      `json:"match,omitempty"`
	Term              *QueryTerm       `json:"term,omitempty"`
	Range             *QueryRange      `json:"range,omitempty"`
	MultiMatch        *QueryMultiMatch `json:"multi_match,omitempty"`
	Bool              *QueryBool       `json:"bool,omitempty"`
	QueryString       *QueryString     `json:"query_string,omitempty"`
	SimpleQueryString *QueryString     `json:"simple_query_string,omitempty"`
	Wildcard          *QueryWildcard   `json:"wildcard,omitempty"` // https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html
}

type GetResponseBody 

type GetResponseBody struct {
	Index       string                 `json:"_index"` // name of index
	Type        string                 `json:"_type"`  // _doc
	ID          string                 `json:"_id"`
	Version     int                    `json:"_version"`
	SeqNo       int                    `json:"_seq_no"`
	PrimaryTerm int                    `json:"_primary_term"`
	Found       bool                   `json:"found"`
	Source      map[string]interface{} `json:"_source"` // the document of itemType
}

GetResponseBody Example: 

{
	"_index": "go-utils-opensearch-docs-test",
	"_type": "_doc",
	"_id": "836c6443-5b0e-489b-aa0f-712ebed96841",
	"_version": 1,
	"_seq_no": 6,
	"_primary_term": 1,
	"found": true,
	"_source": { ... }
 }

type HitDoc 

type HitDoc struct {
	Index  string                 `json:"_index"` // name of index
	Type   string                 `json:"_type"`  // _doc
	ID     string                 `json:"_id"`
	Score  float64                `json:"_score"`  //
	Source map[string]interface{} `json:"_source"` // the document of itemType
}

type IndexInfoSettings 

type IndexInfoSettings struct {
	Index IndexSettings `json:"index"`
}

type IndexResponse 

type IndexResponse struct {
	Error       *Error  `json:"error,omitempty"`
	Result      string  `json:"result,omitempty"` // e.g. "created"
	Index       string  `json:"_index,omitempty"`
	ID          string  `json:"_id,omitempty"`
	Version     int     `json:"_version,omitempty"`
	SeqNo       int     `json:"_seq_no,omitempty"`
	Shards      *Shards `json:"_shards,omitempty"`
	Type        string  `json:"_type,omitempty"`
	PrimaryTerm int     `json:"_primary_term,omitempty"`
}

type IndexSettings 

type IndexSettings struct {
	UUID             string `json:"uuid"`
	CreationDate     string `json:"creation_date"`
	NumberOfShards   string `json:"number_of_shards"`
	NumberOfReplicas string `json:"number_of_replicas"`
	ProviderName     string `json:"provided_name"` // e.g. "go-utils-audit-test-20211103"
}

type Mapping added in v1.117.0 

type Mapping struct {
	TotalFields TotalFields `json:"total_fields,omitempty"`
}

type MappingFieldProperties 

type MappingFieldProperties struct {
	Keyword *MappingKeyword `json:"keyword"`
}

type MappingKeyword 

type MappingKeyword struct {
	Type        string `json:"type"`         // ="keyword"
	IgnoreAbove int    `json:"ignore_above"` // e.g. 256
}

type MappingProperty 

type MappingProperty struct {
	Type       string                            `json:"type,omitempty"` // empty for sub-structs described with properties
	Enabled    bool                              `json:"enabled,omitempty"`
	Fields     map[string]MappingFieldProperties `json:"fields,omitempty"`
	Properties map[string]MappingProperty        `json:"properties,omitempty"`
}

type Mappings 

type Mappings struct {
	Properties map[string]MappingProperty `json:"properties,omitempty"`
}

type Query 

type Query struct {
	Match             *QueryMatch      `json:"match,omitempty" doc:"<field>:<value>"`
	Term              *QueryTerm       `json:"term,omitempty"`
	Range             *QueryRange      `json:"range,omitempty"`
	MultiMatch        *QueryMultiMatch `json:"multi_match,omitempty"`
	Bool              *QueryBool       `json:"bool,omitempty"`
	QueryString       *QueryString     `json:"query_string,omitempty"`
	SimpleQueryString *QueryString     `json:"simple_query_string,omitempty"`
}

https://opensearch.org/docs/latest/opensearch/query-dsl/bool/

type QueryBool 

type QueryBool struct {
	Must               []FilterQuery `json:"must,omitempty"`     // List of things that must appear in matching documents and will contribute to the score.
	Filter             []FilterQuery `json:"filter,omitempty"`   // List of things that must appear in matching documents. However, unlike must the score of the query will be ignored. Filter clauses are executed in filter context, meaning that scoring is ignored and clauses are considered for caching
	Should             []Query       `json:"should,omitempty"`   // List of things that should appear in the matching document.
	MustNot            []FilterQuery `json:"must_not,omitempty"` // List of things that must not appear in the matching documents. Clauses are executed in filter context meaning that scoring is ignored and clauses are considered for caching. Because scoring is ignored, a score of 0 for all documents is returned
	MinimumShouldMatch int64         `json:"minimum_should_match"`
}

type QueryExpr 

type QueryExpr map[string]string

type QueryMatch 

type QueryMatch map[string]string

type QueryMultiMatch 

type QueryMultiMatch struct {
	Query    string              `json:"query"` // Full value match opensearch in selected fields
	Fields   []string            `json:"fields,omitempty" doc:"List of fields"`
	Type     QueryMultiMatchType `json:"type,omitempty"`
	Operator string              `json:"operator,omitempty"`
}

type QueryMultiMatchType 

type QueryMultiMatchType string
const (
	QueryMultiMatchTypePhrase      QueryMultiMatchType = "phrase"
	QueryMultiMatchTypeCrossFields QueryMultiMatchType = "cross_fields"
)

type QueryRange 

type QueryRange map[string]QueryExpr

type QueryString 

type QueryString struct {
	Query           string   `json:"query"` // Text opensearch with partial matches, using asterisk for optional or question mark for required wildcards before and/or after text
	Fields          []string `json:"fields,omitempty" doc:"List of fields"`
	DefaultOperator string   `json:"default_operator,omitempty"`
}

type QueryTerm 

type QueryTerm map[string]string

type QueryValue 

type QueryValue struct {
	Query          string `json:"query"`
	Operator       string `json:"operator,omitempty"`  // defaults to "or", accepted values: or|and
	Fuzziness      string `json:"fuzziness,omitempty"` // https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#fuzziness
	ZeroTermsQuery string `json:"zero_terms_query,omitempty"`
}

type QueryWildcard 

type QueryWildcard map[string]string

type SearchRequestBody 

type SearchRequestBody struct {
	Sort    []map[string]string `json:"sort,omitempty"` // order and order_by
	Size    int64               `json:"size,omitempty"` // limit
	From    int64               `json:"from,omitempty"` // offset
	Query   Query               `json:"query"`
	Timeout string              `json:"timeout,omitempty"` // timeout for opensearch
}

type SearchResponseBody 

type SearchResponseBody struct {
	Took     int                  `json:"took"` // milliseconds
	TimedOut bool                 `json:"timed_out"`
	Shards   SearchResponseShards `json:"_shards"`
	Hits     SearchResponseHits   `json:"hits"`
}

SearchResponseBody example: 

{
	"took":872,
	"timed_out":false,
	"_shards":{
		"total":38,
		"successful":38,
		"skipped":0,
		"failed":0
	},
	"hits":{
		"total":{
			"value":0,
			"relation":"eq"
		},
		"max_score":null,
		"hits":[
			{
				"_index": "go-utils-audit-test-20211030",
				"_type": "_doc",
				"_id": "Tj9l5XwBWRiAneoYazic",
				"_score": 1.2039728,
				"_source": {
					"@timestamp": "2021-10-30T15:03:20.679481+02:00",
					"@end_time": "2021-10-30T15:03:20.469481+02:00",
					"@duration_ms": -210,
					"test1": "6",
					"test2": "ACC_00098",
					"test3": 10,
					"http": {
						"method": "GET",
						"path": "/accounts"
					},
					"http_method": "GET",
					"http_path": "/accounts"
				}
			},
		]
	}
}

type SearchResponseHits 

type SearchResponseHits struct {
	Total    SearchResponseHitsTotal `json:"total"`
	MaxScore *float64                `json:"max_score,omitempty"`
	Hits     []HitDoc                `json:"hits"`
}

type SearchResponseHitsTotal 

type SearchResponseHitsTotal struct {
	Value    int    `json:"value"`    // e.g. 0 when no docs matched
	Relation string `json:"relation"` // e.g. "eq"
}

type SearchResponseShards 

type SearchResponseShards struct {
	Total      int `json:"total"`
	Successful int `json:"successful"`
	Skipped    int `json:"skipped"`
	Failed     int `json:"failed"`
}

type Settings 

type Settings struct {
	Index *SettingsIndex `json:"index,omitempty"`
}

Settings Mapping configures an index in OpenSearch

type SettingsIndex 

type SettingsIndex struct {
	NumberOfShards   int      `json:"number_of_shards,omitempty"`
	NumberOfReplicas int      `json:"number_of_replicas,omitempty"`
	Mapping          *Mapping `json:"mapping,omitempty"`
}

type Shards 

type Shards struct {
	Total      int `json:"total"`
	Successful int `json:"successful"`
	Failed     int `json:"failed"`
}

type TimeSeries 

type TimeSeries struct {
	// contains filtered or unexported fields
}

func (*TimeSeries) DelOldTimeSeries 

func (ts *TimeSeries) DelOldTimeSeries(olderThanDays int) ([]string, error)

DelOldTimeSeries parameters: 

indexName is index prefix before dash-date, e.g. "api-api_logs" then will look for "api-api_logs-<date>"

returns 

list of indices to delete with err==nil if deleted successfully

func (*TimeSeries) Get 

func (ts *TimeSeries) Get(id string) (res *GetResponseBody, err error)

Get takes the id returned in Search() The id is uuid assigned by OpenSearch when documents are added with Write(). The document value type is the same as that of tmpl specified when you created the TimeSeries(..., tmpl)

func (*TimeSeries) Search 

func (ts *TimeSeries) Search(query Query, sort []map[string]string, limit int64, offset int64) (openSearchResult *SearchResponseHits, err error)

Search returns docs indexed on OpenSearch document ID which cat be used in Get(id) The docs value type is the same as that of tmpl specified when you created the TimeSeries(..., tmpl) So you can safely type assert e.g. 

	type myType struct {...}
	ts := opensearch.TimeSeries(..., myType{})
	docs,totalCount,err := ts.Search(...)
	if err == nil {
		for id,docValue := range docs {
			doc := docValue.(myType)
			...
		}
	}
docs will be a slice of the TimeSeries data type

func (*TimeSeries) Write 

func (ts *TimeSeries) Write(startTime, endTime time.Time, data interface{}) error

data must be of type specified in Writer.TimeSeries(tmpl)

type TimeSeriesHeader 

type TimeSeriesHeader struct {
	StartTime  time.Time `json:"@timestamp"`
	EndTime    time.Time `json:"@end_time"`
	DurationMs int64     `json:"@duration_ms"`
}

TimeSeriesHeader embed this into your log struct

type TotalFields added in v1.117.0 

type TotalFields struct {
	Limit int `json:"limit,omitempty"`
}

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

func New 

func New(config Config) (Writer, error)

func (*Writer) NewDocumentStore 

func (w *Writer) NewDocumentStore(name string, tmpl interface{}) (DocumentStore, error)

NewDocumentStore purpose: 

create a document store index to write e.g. orders then allow one to opensearch them

parameters: 

name must be the complete openSearch index name e.g. "uafrica-v3-orders"
tmpl must be your document data struct consisting of public fields as:
	Xxx string `json:"<name>" search:"keyword|text|long|date"`	(can later add more types)
	Xxx time.Time `json:"<name>"`								assumes type "date" for opensearch
	Xxx int `json:"<name>"`										assumes type "long" for opensearch, specify keyword if required

func (*Writer) NewTimeSeries 

func (w *Writer) NewTimeSeries(name string, tmpl interface{}) (TimeSeries, error)

NewTimeSeries purpose: 

create a time series to write e.g. api api_logs

parameters: 

name must be the openSearch index name prefix without the date, e.g. "uafrica-v3-api-api_logs"
	the actual indices in openSearch will be called "<indexName>-<ccyymmdd>" e.g. "uafrica-v3-api-api_logs-20210102"
tmpl must be your log data struct consisting of public fields as:
	Xxx string `json:"<name>" search:"keyword|text|long|date"`	(can later add more types)
	Xxx time.Time `json:"<name>"`								assumes type "date" for opensearch
	Xxx int `json:"<name>"`										assumes type "long" for opensearch, specify keyword if required

func (Writer) Write 

func (writer Writer) Write(indexName string, id string, doc interface{}) (*IndexResponse, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.