incite

package module
v0.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 16, 2021 License: MIT Imports: 16 Imported by: 0

Documentation

Index

Examples

Constants

View Source 
const (
	// QueryConcurrencyQuotaLimit contains the CloudWatch Logs Query
	// Concurrency service quota limit as documented at
	// https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html.
	//
	// The documented service quota may increase over time, in which case
	// this value should be updated to match the documentation.
	QueryConcurrencyQuotaLimit = 10

	// DefaultParallel is the default maximum number of parallel
	// CloudWatch Logs Insights queries a QueryManager will attempt to
	// run at any one time.
	//
	// The default value is set to slightly less than the service quota
	// limit to leave some concurrency available for other users even if
	// the QueryManager is at maximum capacity.
	DefaultParallel = QueryConcurrencyQuotaLimit - 2

	// DefaultLimit is the default result count limit applied to query
	// operations if no value is explicitly set.
	DefaultLimit = 1000

	// MaxLimit is the maximum result count limit a query operation may
	// specify.
	MaxLimit = 10000
)
View Source 
const TimeLayout = "2006-01-02 15:04:05.000"

TimeLayout is a Go time layout which defines the format of the time values returned in the timestamp fields of CloudWatch Logs Insights queries.

TimeLayout defines the format by showing how the Go reference time of 

Mon Jan 2 15:04:05 -0700 MST 2006

would be formatted if it were the value. TimeLayout can be used with time.Parse to parse timestamp fields, such as @timestamp and @ingestionTime, which are returned in CloudWatch Logs Insights query results.

Variables

View Source 
var DefaultRPS = map[CloudWatchLogsAction]int{
	StartQuery:      RPSQuotaLimits[StartQuery] - 2,
	StopQuery:       RPSQuotaLimits[StopQuery] - 2,
	GetQueryResults: RPSQuotaLimits[GetQueryResults] - 2,
}

DefaultRPS specifies the maximum number of requests per second which the QueryManager may make to the CloudWatch Logs web service for each CloudWatch Logs action.

View Source 
var (
	// ErrClosed is the error returned by a read or query operation
	// when the underlying stream or query manager has been closed.
	ErrClosed = errors.New("incite: operation on a closed object")
)
View Source 
var NopLogger = nopLogger(0)

NopLogger is a Logger that ignores any messages sent to it.

View Source 
var RPSQuotaLimits = map[CloudWatchLogsAction]int{
	StartQuery:      5,
	StopQuery:       5,
	GetQueryResults: 5,
}

RPSQuotaLimits contains the CloudWatch Logs service quota limits for number of requests per second for each CloudWatch Logs API action before the request fails due to a throttling error as documented in the AWS service limits system.

The documented service quotas may increase over time, in which case the map values should be updated to match the increases.

Functions

func Unmarshal 

func Unmarshal(data []Result, v interface{}) error

Unmarshal converts the CloudWatch Logs Insights result data into the user-defined type indicated by v, and stores the result in the value pointed to by v.

The argument v must contain a non-nil pointer whose ultimate target is a slice, array, or interface value. If v ultimately targets a interface{}, it is treated as if it targets a []map[string]string.

The element type of the array or slice must target a map type, struct type. As special cases, elements of type interface{} and Result are also allowed. If the element type targets a map, the maps keys must be strings and its value type must target a string type, interface{}, or any type that implements encoding.TextUnmarshaler.

To unmarshal data into an array or slice of maps, Unmarshal uses the ResultField name as the map key and the ResultField value as its value. If the map value targets an encoding.TextUnmarshaler, the value's UnmarshalText value is used to unmarshal the value. If the map value targets a string type, the ResultField's value is directly inserted as the field value in the map. As a special case, if the map value targets interface{}, Unmarshal first tries to unmarshal the value as JSON using json.Unmarshal, and falls back to the plain string value if JSON unmarshaling fails.

To unmarshal data into a struct type, Unmarshal uses the following rules top-level rules:

• A struct field with an "incite" tag receives the value of the ResultField field named in the tag. Unmarshaling of the field value is done according to rules discussed below. If the tag is "-" the field is ignored. If the field type does not ultimately target a struct field unmarshalable type, an InvalidUnmarshalError is returned.

• A struct field with a "json" tag receives the the value of the ResultField field named in the tag using the json.Unmarshal function with the ResultField value as the input JSON and the struct field address as the target. If the tag is "-" the field is ignored. The field type is not checked for validity.

• The "incite" tag takes precedence over the "json" tag so they should not be used together on the same struct field.

• A struct field with no "incite" or "json" tag receives the value of the ResultField field sharing the same case-sensitive name as the struct field, but only if the field type ultimately targets a struct field unmarshablable type. Otherwise the field is ignored.

The following types are considered struct field unmarshalable types: 

bool
int8, int16, int32, int64, int
uint8, uint16, uint32, uint64, uint
float32, float64
interface{}
[]byte
Any map, struct, slice, or array type

A struct field targeting interface{} or any map, struct, slice, or array type is assumed to contain valid JSON and unmarshaled using json.Unmarshal. Any other field is decoded from its string representation using the intuitive approach. As a special case, if a CloudWatch Logs timestamp field (@timestamp or @ingestionTime) is named in an "incite" tag, it may only target a time.Time or string value. If it targets a time.Time, the value is decoded using TimeLayout with the time.Parse function.

If a target type rule is violated, Unmarshal returns InvalidUnmarshalError. If a result field value cannot be decoded, Unmarshal stops unmarshaling and returns UnmarshalResultFieldValueError.

The value pointed to by v may have changed even if Unmarshal returns an error.

Example (Interface) 
package main

import (
	"fmt"

	"github.com/gogama/incite"
)

func main() {
	// An interface{} is treated as []map[string]string. The Object
	// key's value is not deserialized from JSON, it remains a string.
	data := []incite.Result{
		[]incite.ResultField{{Field: "@ptr", Value: `abc123`}, {Field: "Object", Value: `{"key":"value"}`}},
	}
	var v interface{}
	_ = incite.Unmarshal(data, &v) // Error ignored for simplicity.
	fmt.Println(v)
}

Output:

[map[@ptr:abc123 Object:{"key":"value"}]]
Example (MapStringInterface) 
package main

import (
	"fmt"

	"github.com/gogama/incite"
)

func main() {
	// As a special case, the data are unmarshalled fuzzily if the target
	// is a map[string]interface{}. If a value is valid JSON it is
	// unmarshalled as JSON, otherwise it is kept as a string. Here the
	// Object and QuotedString fields are contain valid JSON so they
	// unmarshal as a map and string, respectively. UnquotedString is
	// not valid JSON and stays as a string.
	data := []incite.Result{
		[]incite.ResultField{
			{Field: "Object", Value: `{"key":"value"}`},
			{Field: "QuotedString", Value: `"hello"`},
			{Field: "UnquotedString", Value: `world`},
		},
	}
	var v []map[string]interface{}
	_ = incite.Unmarshal(data, &v) // Error ignored for simplicity.
	fmt.Println(v)
}

Output:

[map[Object:map[key:value] QuotedString:hello UnquotedString:world]]
Example (MapStringString) 
package main

import (
	"fmt"

	"github.com/gogama/incite"
)

func main() {
	data := []incite.Result{
		[]incite.ResultField{{Field: "@ptr", Value: "foo"}, {Field: "@message", Value: "bar"}},
	}
	var v []map[string]string
	_ = incite.Unmarshal(data, &v) // Error ignored for simplicity.
	fmt.Println(v)
}

Output:

[map[@message:bar @ptr:foo]]

Types

type CloudWatchLogsAction 

type CloudWatchLogsAction int

CloudWatchLogsAction represents a single enumerated CloudWatch Logs action. 

const (
	// StartQuery indicates the CloudWatch Logs StartQuery action.
	StartQuery CloudWatchLogsAction = iota
	// StopQuery indicates the CloudWatch Logs StopQuery action.
	StopQuery
	// GetQueryResults indicates the CloudWatchLogs GetQueryResults action.
	GetQueryResults
)

type CloudWatchLogsActions 

type CloudWatchLogsActions interface {
	CloudWatchLogsQueryStarter
	CloudWatchLogsQueryStopper
	CloudWatchLogsQueryResultsGetter
}

CloudWatchLogsActions provides access to the CloudWatch Logs actions which QueryManager needs in order to run Insights queries using the CloudWatch Logs service.

This interface is compatible with the AWS SDK for Go (v1)'s cloudwatchlogsiface.CloudWatchLogsAPI interface and *cloudwatchlogs.CloudWatchLogs type, so you may use either of these AWS SDK for Go types to provide the CloudWatch Logs capabilities.

type CloudWatchLogsQueryResultsGetter 

type CloudWatchLogsQueryResultsGetter interface {
	GetQueryResultsWithContext(context.Context, *cloudwatchlogs.GetQueryResultsInput, ...request.Option) (*cloudwatchlogs.GetQueryResultsOutput, error)
}

type CloudWatchLogsQueryStarter 

type CloudWatchLogsQueryStarter interface {
	StartQueryWithContext(context.Context, *cloudwatchlogs.StartQueryInput, ...request.Option) (*cloudwatchlogs.StartQueryOutput, error)
}

type CloudWatchLogsQueryStopper 

type CloudWatchLogsQueryStopper interface {
	StopQueryWithContext(context.Context, *cloudwatchlogs.StopQueryInput, ...request.Option) (*cloudwatchlogs.StopQueryOutput, error)
}

type Config 

type Config struct {
	// Actions provides the CloudWatch Logs capabilities the QueryManager
	// needs to execute Insights queries against the CloudWatch Logs
	// service. If this value is nil then NewQueryManager panics.
	//
	// Normally Actions should be set to the value of an AWS SDK for Go
	// (v1) CloudWatch Logs client: both the cloudwatchlogsiface.CloudWatchLogsAPI
	// interface and the *cloudwatchlogs.CloudWatchLogs type are
	// compatible with the CloudWatchLogsActions interface. Use a
	// properly configured instance of one of these types to set the
	// value of the Actions field.
	Actions CloudWatchLogsActions

	// Parallel optionally specifies the maximum number of parallel
	// CloudWatch Logs Insights queries which the QueryManager may run
	// at one time. The purpose of Parallel is to avoid starving other
	// humans or systems using CloudWatch Logs Insights in the same AWS
	// account and region.
	//
	// If set to a positive number then that exact number is used as the
	// parallelism factor. If set to zero or a negative number then
	// DefaultParallel is used instead.
	//
	// Parallel gives the upper limit on the number of Insights queries
	// the QueryManager may have open at any one time. The actual number
	// of Insights queries may be lower either because of throttling or
	// service limit exceptions from the CloudWatch Logs web service, or
	// because the QueryManager simply doesn't need all the parallel
	// capacity.
	//
	// Note that an Insights query is not necessarily one-to-one with a
	// Query operation on a QueryManager. If the Query operation is
	// chunked, the QueryManager may create Insights multiple queries in
	// the CloudWatch Logs web service to fulfil the chunked Query
	// operation.
	Parallel int

	// RPS optionally specifies the maximum number of requests to the
	// CloudWatch Logs web service which the QueryManager may make in
	// each one second period for each CloudWatch Logs action. The
	// purpose of RPS is to prevent the QueryManager or other humans or
	// systems using CloudWatch Logs in the same AWS account and region
	// from being throttled by the web service.
	//
	// If RPS has a missing, zero, or negative number for any required
	// CloudWatch Logs capability, the value specified in DefaultRPS is
	// used instead.
	RPS map[CloudWatchLogsAction]int

	// Logger optionally specifies a logging object to which the
	// QueryManager can send log messages about queries it is managing.
	// This value may be left nil to skip logging altogether.
	Logger Logger
}

Config provides the NewQueryManager function with the information it needs to construct a new QueryManager.

type InvalidUnmarshalError 

type InvalidUnmarshalError struct {
	Type      reflect.Type
	RowType   reflect.Type
	Field     string
	FieldType reflect.Type
	Message   string
}

An InvalidUnmarshalError occurs when a value with an invalid type is passed to to Unmarshal.

func (*InvalidUnmarshalError) Error 

func (e *InvalidUnmarshalError) Error() string

type Logger 

type Logger interface {
	Printf(format string, v ...interface{})
}

A Logger represents a logging object that which can receive log messages from a QueryManager and send them to an output sink.

Logger is compatible with *log.Logger. Therefore you may, for example, set log.DefaultLogger(), or any other *log.Logger, as the logger field in a Config.

type QueryManager 

type QueryManager interface {
	io.Closer
	Queryer
	StatsGetter
}

QueryManager executes one or more CloudWatch Logs Insights queries, optionally executing simultaneous queries in parallel.

QueryManager's job is to hide the complexity of the CloudWatch Logs Insights API, taking care of mundane details such as starting and polling jobs in the CloudWatch Logs Insights service, breaking queries into smaller time chunks (if desired), de-duplicating and providing preview results (if desired), retrying transient request failures, and managing resources to try to stay within the CloudWatch Logs service quota limits.

Use NewQueryManager to create a QueryManager, and be sure to close it when you no longer need its services, since every QueryManager consumes some compute resources just by existing.

Calling the Query method will return a result Stream from which the query results can be read as they become available. Use the Unmarshal to unmarshal the bare results into other structured types.

Calling the Close method will immediately cancel all running queries] started with the Query, as if the query's Stream had been explicitly closed.

Calling GetStats will return the running sum of all statistics for all queries run within the QueryManager since it was created.

func NewQueryManager 

func NewQueryManager(cfg Config) QueryManager

NewQueryManager returns a new query manager with the given configuration.

type QuerySpec 

type QuerySpec struct {
	// Text contains the actual text of the CloudWatch Insights query,
	// following the query syntax documented at
	// https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html.
	//
	// To limit the number of results returned by the query, add the
	// `limit` command to your query text. Note that if the QuerySpec
	// specifies a chunked query then the limit will apply to the
	// results obtained from each chunk, not to the global query.
	//
	// Text may not contain an empty or blank string. Beyond checking
	// for blank text, Incite does not attempt to parse Text and simply
	// forwards it to the CloudWatch Logs service. Care must be taken to
	// specify query text compatible with the Chunk and Preview fields
	// or the results may be misleading.
	Text string

	// Start specifies the beginning of the time range to query,
	// inclusive of Start itself.
	//
	// Start must be strictly before End, and must represent a whole
	// number of seconds (it cannot have sub-second granularity).
	Start time.Time

	// End specifies the end of the time range to query, exclusive of
	// End itself.
	//
	// End must be strictly after Start, and must represent a whole
	// number of seconds (it cannot have sub-second granularity).
	End time.Time

	// Groups lists the names of the CloudWatch Logs log groups to be
	// queried. It may not be empty.
	Groups []string

	// Limit optionally specifies the maximum number of results to be
	// returned by the query.
	//
	// If Limit is zero or negative, the value DefaultLimit is used
	// instead. If Limit exceeds MaxLimit, the query operation will fail
	// with an error.
	//
	// In a chunked query, Limit applies to each chunk separately, so
	// up to (n × Limit) final results may be returned, where n is the
	// number of chunks.
	//
	// Note that as of 2021-07-15, the CloudWatch Logs StartQuery API
	// seems to ignore the `limit` command in the query text, so if you
	// want to apply a limit you must use the Limit field.
	Limit int64

	// Chunk optionally requests a chunked query and indicates the chunk
	// size.
	//
	// If Chunk is zero, negative, or greater than the difference between
	// End and Start, the query is not chunked. If Chunk is positive and
	// less than the difference between End and Start, the query is
	// broken into n chunks, where n is (End-Start)/Chunk, rounded up to
	// the nearest integer value.
	//
	// In a chunked query, each chunk is sent to the CloudWatch Logs
	// service as a separate Insights query. This can help large queries
	// complete before the CloudWatch Logs query timeout of 15 minutes,
	// and can increase performance because chunks can be run in parallel.
	// However, the following considerations should be taken into account:
	//
	// • In a chunked query, Limit applies separately to each chunk. So
	// a query with 50 chunks and a limit of 50 could produce up to 2500
	// final results.
	//
	// • If Text contains a sort command, the sort will only apply
	// within each individual chunk. If the QuerySpec is executed by a
	// QueryManager configured with a parallelism factor above 1, then
	// the results may appear be out of order since the order of
	// completion of chunks is not guaranteed.
	//
	// • If Text contains a stats command, the statistical aggregation
	// will be applied to each chunk in a chunked query, meaning up to n
	// versions of each aggregate data point may be returned, one per
	// chunk, necessitating further aggregation on the client side.
	//
	// • In general if you use chunking with query text which implies
	// any kind of server-side aggregation, you may need to perform
	// custom post-processing on the results.
	Chunk time.Duration

	// Preview optionally requests preview results from a running query.
	//
	// If Preview is true, intermediate results for the query are
	// sent to the results stream as soon as they are available. This
	// can result in increased responsiveness for the end-user of your
	// application but requires care since not all intermediate results
	// are valid members of the final result set.
	//
	// When Preview is true, the query result Stream may produce some
	// intermediate results which it later determines are invalid
	// because they shouldn't be final members of the result set. For
	// each such invalid result, an extra trivial Result will be sent to
	// the result Stream with the following structure:
	//
	// 	incite.Result{
	// 		{ Field: "@ptr", Value: "<Unique @ptr of the earlier invalid result>" },
	//		{ Field: "@deleted", Value: "true" },
	// 	}
	//
	// The presence of the "@deleted" field can be used to identify and
	// delete the earlier invalid result.
	//
	// The Preview option has no effect if the results returned from
	// CloudWatch Logs do not contain an @ptr field, which can happen,
	// for example, if the query Text contains a `stats` command.
	Preview bool

	// Priority optionally allows a query operation to be given a higher
	// or lower priority with regard to other query operations managed
	// by the same QueryManager. This can help the QueryManager manage
	// finite resources to stay within CloudWatch Logs service quota
	// limits.
	//
	// A lower number indicates a higher priority. The default zero
	// value is appropriate for many cases.
	//
	// The Priority field may be set to any valid int value. A query
	// whose Priority number is lower is allocated CloudWatch Logs
	// service query capacity in preference to a query whose Priority
	// number is higher, but only with the same QueryManager.
	Priority int

	// Hint optionally indicates the rough expected size of the result
	// set, which can help the QueryManager do a better job allocating
	// memory needed to manage and hold query results. Leave it zero
	// if you don't know the expected result size or aren't worried
	// about optimizing memory consumption.
	Hint uint16
}

QuerySpec specifies the parameter for a query operation either using the global Query function or a QueryManager.

type Queryer 

type Queryer interface {
	Query(QuerySpec) (Stream, error)
}

The Queryer interface provides the ability to run a CloudWatch Logs Insights query. Use NewQueryManager to create a QueryManager, which contains this interface.

type Reader 

type Reader interface {
	Read(p []Result) (n int, err error)
}

Reader provides a basic Read method to allow reading CloudWatch Logs Insights query results as a stream.

Read reads up to len(p) CloudWatch Logs Insights results into p. It returns the number of results read (0 <= n <= len(p)) and any error encountered. Even if Read returns n < len(p), it may use all of p as scratch space during the call. If some data are available but fewer than len(p) results, Read conventionally returns what is available instead of waiting for more.

When Read encounters an error or end-of-file condition after successfully reading n > 0 results, it returns the number of results read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call. An instance of this general case is that a Reader returning a non-zero number of results at the end of the input stream may return either err == EOF or err == nil. The next Read should return 0, EOF.

Callers should always process the n > 0 results returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some results and also both of the allowed EOF behaviors.

Implementations of Read are discouraged from returning a zero result count with a nil error, except when len(p) == 0. Callers should treat a return of 0 and nil as indicating that nothing happened; in particular it does not indicate EOF.

Implementations must not retain p.

As a convenience, the ReadAll function may be used to read all remaining results available in a reader.

type Result 

type Result []ResultField

Result represents a single result row from a CloudWatch Logs Insights query.

func Query 

func Query(caps CloudWatchLogsActions, q interface{}) ([]Result, error)

Query is sweet sweet sugar to perform a synchronous CloudWatch Logs Insights query and get back all the results without needing to construct a QueryManager.

Unlike NewQueryManager, which defaults to DefaultParallel, Query uses a parallelism factor of 1. This means that if q represents a chunked query, then the chunks will be run serially.

This function is intended for quick prototyping and simple scripting and command-line interface use cases. More complex applications, especially applications running concurrent queries against the same region from multiple goroutines, should construct and configure a QueryManager explicitly.

The input query may be either a QuerySpec or a bare string containing the text of an Insights query. If the query is a bare string, then it is treated like a zero-value QuerySpec which has had its Text member set to the string.

func ReadAll 

func ReadAll(r Reader) ([]Result, error)

ReadAll reads from s until an error or EOF and returns the data it read. A successful call returns err == nil, nor err == EOF. Because ReadAll is defined to read from r until EOF, it does not treat an EOF from Read as an error to be reported.

type ResultField 

type ResultField struct {
	Field string
	Value string
}

ResultField represents a single field name/field value pair within a Result.

type StartQueryError added in v0.9.1 

type StartQueryError struct {
	Text  string
	Start time.Time
	End   time.Time
	Cause error
}

func (*StartQueryError) Error added in v0.9.1 

func (err *StartQueryError) Error() string

func (*StartQueryError) Unwrap added in v0.9.1 

func (err *StartQueryError) Unwrap() error

type Stats 

type Stats struct {
	// BytesScanned represents the total number of bytes of log events
	// scanned.
	BytesScanned float64
	// RecordsMatched counts the number of log events that matched the
	// query or queries.
	RecordsMatched float64
	// RecordsScanned counts the number of log events scanned during the
	// query or queries.
	RecordsScanned float64
}

Stats contains metadata returned by CloudWatch Logs about the amount of data scanned and number of result records matched during one or more Insights queries.

type StatsGetter 

type StatsGetter interface {
	GetStats() Stats
}

StatsGetter provides access to the Insights query statistics returned by the CloudWatch Logs API.

Both Stream and QueryManager contain the StatsGetter interface. Call the GetStats method on a Stream to get the query statistics for the stream's query. Call the GetStats method on a QueryManager to get the query statistics for all queries run within the QueryManager.

type Stream 

type Stream interface {
	io.Closer
	Reader
	StatsGetter
}

Stream provides access to the result stream from a query operation either using a QueryManager or the global Query function.

Use the Close method if you need to prematurely cancel the query operation, releasing the local (in-process) and remote (in the CloudWatch Logs service) resources it consumes.

Use the Read method to read query results from the stream. The Read method returns io.EOF when the entire results stream has been consumed. At this point the query is over and all local and remote resources have been released, so it is not necessary to close the stream explicitly.

Use the GetStats method to obtain the Insights statistics pertaining to the query. Note that the results from the GetStats method may change over time as new results are pulled from the CloudWatch Logs web service, but will stop changing after the Read method returns io.EOF. If the query was chunked, the stats will be summed across multiple chunks.

type TerminalQueryStatusError added in v0.9.1 

type TerminalQueryStatusError struct {
	QueryID string
	Status  string
	Text    string
}

func (*TerminalQueryStatusError) Error added in v0.9.1 

func (err *TerminalQueryStatusError) Error() string

type UnexpectedQueryError added in v0.9.1 

type UnexpectedQueryError struct {
	QueryID string
	Text    string
	Cause   error
}

func (*UnexpectedQueryError) Error added in v0.9.1 

func (err *UnexpectedQueryError) Error() string

func (*UnexpectedQueryError) Unwrap added in v0.9.1 

func (err *UnexpectedQueryError) Unwrap() error

type UnmarshalResultFieldValueError 

type UnmarshalResultFieldValueError struct {
	ResultField
	Cause       error
	ResultIndex int
	FieldIndex  int
}

An UnmarshalResultFieldValueError describes a failure to unmarshal a specific ResultField value within a specific Result.

func (*UnmarshalResultFieldValueError) Error 

func (e *UnmarshalResultFieldValueError) Error() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.