structs

package
v0.0.0-...-6884e6a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 21, 2024 License: AGPL-3.0 Imports: 27 Imported by: 37

Documentation

Index

Constants

View Source 
const (
	SFMTop = iota
	SFMRare
)
View Source 
const (
	REMPhrase   = iota //Rename with a phrase
	REMRegex           //Rename fields with similar names using a wildcard
	REMOverride        //Rename to a existing field
)
View Source 
const (
	LSMBySum  = iota // If only a single aggregation is specified, the score is based on the sum of the values in the aggregation
	LSMByFreq        // Otherwise the score is based on the frequency of the by field's val
)
View Source 
const (
	VEMNumericExpr   = iota // Only NumricExpr is valid
	VEMStringExpr           // Only StringExpr is valid
	VEMConditionExpr        // Only ConditionExpr is valud
	VEMBooleanExpr          // Only BooleanExpr is valid
)
View Source 
const (
	SEMRawString     = iota // only used when mode is RawString
	SEMRawStringList        // only used when mode is RawStringList
	SEMField                // only used when mode is Field
	SEMConcatExpr           // only used when mode is Concat
	SEMTextExpr             // only used when mode is TextExpr
)
View Source 
const (
	NEMNumber      = iota // only used when mode is a Number
	NEMLenString          // only used when mode is a str (used for len())
	NEMNumberField        // only used when mode is Field (Field can be evaluated to a float)
	NEMLenField           // only used when mode is Field (Field can not be evaluated to a float, used for len())
	NEMNumericExpr        // only used when mode is a NumericExpr
)
View Source 
const (
	TS_Type8 = iota + 1
	TS_Type16
	TS_Type32
	TS_Type64
)
View Source 
const (
	InvalidCmd = iota
	SegmentStatsCmd
	GroupByCmd
	RRCCmd
)
View Source 
const MAX_SEGMETA_FSIZE = 10_000_000 // 10 MB
View Source 
const SIZE_OF_BSUM = 18
View Source 
const SIZE_OF_BlockInfo = 14 // 2 + 4 + 8 bytes
View Source 
const SIZE_OF_MBSUM = 10 // 2 + 4 + 4

Variables

This section is empty.

Functions

func CheckUnsupportedFunctions 

func CheckUnsupportedFunctions(post *QueryAggregators) error

func CompareSortValueSlices 

func CompareSortValueSlices(a []SortValue, b []SortValue, ascending []int) (int, error)

The `ascending` slice should have the same length as `a` and `b`. Moreover, each element of `ascending` should be either +1 or -1; +1 means higher values get sorted higher, and -1 means lower values get sorted higher.

func ExtractRangeFilterFromSearch 

func ExtractRangeFilterFromSearch(leftSearch *SearchExpressionInput, filterOp FilterOperator, rightSearch *SearchExpressionInput, qid uint64) (map[string]string, FilterOperator, bool)

Given a left and right SearchInputs with filterOp, extract out range filters Returns a map from column to value, the final operator, and a bool telling if a range filter has been found for these. False value means the search inputs have no range filters (may have swapped from original if right has column and left and literal)

func GetAllColumnsFromCondition 

func GetAllColumnsFromCondition(cond *SearchCondition) (map[string]bool, bool)

Get all columns that occur across a list of *SearchQuery returns all columns and if any of the columns contains wildcards

func GetAllColumnsFromNode 

func GetAllColumnsFromNode(node *SearchNode) (map[string]bool, bool)

func GetBsuFnameFromSegKey 

func GetBsuFnameFromSegKey(segkey string) string

func GetBucketKey 

func GetBucketKey(BucketKey interface{}, keyIndex int) string

func GetRenameGroupByCols 

func GetRenameGroupByCols(aggGroupByCols []string, agg *QueryAggregators) []string

Rename field A to field B. If A and B are groupby columns, field B should be removed from groupby columns, and rename A to B

func MatchAndExtractGroups 

func MatchAndExtractGroups(str string, rexExp *regexp.Regexp) (map[string]string, error)

func RemoveUnusedGroupByCols 

func RemoveUnusedGroupByCols(aggs *QueryAggregators, aggGroupByCols []string) []string

For Rename or top/rare block, we may need to delete some groupby columns while processing them

Types

type ASTNode 

type ASTNode struct {
	AndFilterCondition       *Condition     // A condition to query. Condition must return true for log line to pass
	OrFilterCondition        *Condition     // Condition must return true for log line to pass
	ExclusionFilterCondition *Condition     // Condition must return false for log line to pass
	TimeRange                *dtu.TimeRange // Time range for node micro index / raw search
	ActiveFileSearch         bool           // Lookup unrotated segfiles
	BucketLimit              int
}

Node used to query data in a segment file. A line matches a node if it matches all conditions in AndFilterConditions, any in OrFilterCriteria, and none in ExclusionFilterCriteria

type Aggregation 

type Aggregation struct {
	AggregatorFunction utils.AggregateFunctions //aggregator function
	FuncConstant       float64
	GroupByFields      []string // group by fields will be sorted
}

func (Aggregation) IsAggregateFromAllTimeseries 

func (agg Aggregation) IsAggregateFromAllTimeseries() bool

func (Aggregation) ShallowClone 

func (agg Aggregation) ShallowClone() *Aggregation

type AggregationResult 

type AggregationResult struct {
	IsDateHistogram bool            // is this a date histogram
	Results         []*BucketResult // histogram results
}

type AggregationType 

type AggregationType uint8

Defines the different types of bucketing aggregations for processing of results 

const (
	TimeHistogram AggregationType = iota
	GroupBy
)

type AllSegStatsJSON 

type AllSegStatsJSON struct {
	AllSegStats map[string]*SegStatsJSON
}

type AllSegStoreSummary 

type AllSegStoreSummary struct {
	TotalSegmentCount                  uint64
	TotalMetricsSegmentCount           uint64
	TotalTableCount                    uint64
	InMemoryCMICount                   uint64
	InMemorySearchMetadataCount        uint64
	InMemoryBlockMicroIndexSizeMB      uint64
	InMemorySsmSizeMB                  uint64
	InMemoryMetricsSearchMetadataCount uint64
	InMemoryMetricsBSumSizeMB          uint64
}

func (*AllSegStoreSummary) DecrementTotalMetricsSegmentCount 

func (sum *AllSegStoreSummary) DecrementTotalMetricsSegmentCount()

func (*AllSegStoreSummary) DecrementTotalSegKeyCount 

func (sum *AllSegStoreSummary) DecrementTotalSegKeyCount()

func (*AllSegStoreSummary) DecrementTotalTableCount 

func (sum *AllSegStoreSummary) DecrementTotalTableCount()

func (*AllSegStoreSummary) IncrementTotalMetricsSegmentCount 

func (sum *AllSegStoreSummary) IncrementTotalMetricsSegmentCount()

func (*AllSegStoreSummary) IncrementTotalSegmentCount 

func (sum *AllSegStoreSummary) IncrementTotalSegmentCount()

func (*AllSegStoreSummary) IncrementTotalTableCount 

func (sum *AllSegStoreSummary) IncrementTotalTableCount()

func (*AllSegStoreSummary) SetInMemoryBlockMicroIndexCount 

func (sum *AllSegStoreSummary) SetInMemoryBlockMicroIndexCount(count uint64)

func (*AllSegStoreSummary) SetInMemoryBlockMicroIndexSizeMB 

func (sum *AllSegStoreSummary) SetInMemoryBlockMicroIndexSizeMB(size uint64)

func (*AllSegStoreSummary) SetInMemoryMetricsSearchmetadataCount 

func (sum *AllSegStoreSummary) SetInMemoryMetricsSearchmetadataCount(val uint64)

func (*AllSegStoreSummary) SetInMemoryMetricsSsmSizeMB 

func (sum *AllSegStoreSummary) SetInMemoryMetricsSsmSizeMB(val uint64)

func (*AllSegStoreSummary) SetInMemorySearchmetadataCount 

func (sum *AllSegStoreSummary) SetInMemorySearchmetadataCount(count uint64)

func (*AllSegStoreSummary) SetInMemorySsmSizeMB 

func (sum *AllSegStoreSummary) SetInMemorySsmSizeMB(val uint64)

type AvgStat 

type AvgStat struct {
	Count int64
	Sum   float64
}

type BinOptions 

type BinOptions struct {
	SpanOptions *SpanOptions
}

type BlockInfo 

type BlockInfo struct {
	BlkNum    uint16
	BlkLen    uint32
	BlkOffset int64
}

type BlockMetadataHolder 

type BlockMetadataHolder struct {
	BlkNum            uint16
	ColumnBlockOffset map[string]int64
	ColumnBlockLen    map[string]uint32
}

type BlockSearchHelper 

type BlockSearchHelper struct {
	// contains filtered or unexported fields
}

a helper struct to keep track to the blocks search status. This struct will be used to re-use slices of matched time stamps it is important to know that the slice is re-used, so callers would need to copy the values if needed else, will leak memory

func InitAllBlockSearchHelpers 

func InitAllBlockSearchHelpers(fileParallelism int64) []*BlockSearchHelper

func InitBlockSearchHelper 

func InitBlockSearchHelper() *BlockSearchHelper

func (*BlockSearchHelper) AddMatchedRecord 

func (h *BlockSearchHelper) AddMatchedRecord(recNum uint)

func (*BlockSearchHelper) ClearBit 

func (h *BlockSearchHelper) ClearBit(recNum uint)

func (*BlockSearchHelper) DoesRecordMatch 

func (h *BlockSearchHelper) DoesRecordMatch(recNum uint) bool

func (*BlockSearchHelper) GetAllMatchedRecords 

func (h *BlockSearchHelper) GetAllMatchedRecords() *pqmr.PQMatchResults

func (*BlockSearchHelper) ResetBlockHelper 

func (h *BlockSearchHelper) ResetBlockHelper()

keep allocated slice

type BlockSummary 

type BlockSummary struct {
	HighTs   uint64
	LowTs    uint64
	RecCount uint16
}

If new member is added to BlockSum, make sure to reset it's value in resetwipblock()

func (*BlockSummary) Copy 

func (bs *BlockSummary) Copy() *BlockSummary

returns a deep copy of the block summary

func (*BlockSummary) GetSize 

func (bs *BlockSummary) GetSize() uint64

type BlockTracker 

type BlockTracker struct {
	// contains filtered or unexported fields
}

Helper struct to keep track of which blocks to check

func InitEntireFileBlockTracker 

func InitEntireFileBlockTracker() *BlockTracker

func InitExclusionBlockTracker 

func InitExclusionBlockTracker(spqmr *pqmr.SegmentPQMRResults) *BlockTracker

func (*BlockTracker) ShouldProcessBlock 

func (bt *BlockTracker) ShouldProcessBlock(blkNum uint16) bool

type BoolExpr 

type BoolExpr struct {
	IsTerminal bool

	// Only used when IsTerminal is true.
	LeftValue  *ValueExpr
	RightValue *ValueExpr
	ValueOp    string       // Only = or != for strings; can also be <, <=, >, >= for numbers.
	ValueList  []*ValueExpr //Use for in(<value>, <list>)

	// Only used when IsTerminal is false. For a unary BoolOp, RightExpr should be nil.
	LeftBool  *BoolExpr
	RightBool *BoolExpr
	BoolOp    BoolOperator
}

These structs are used to organize boolean, string, and numeric expressions. The expressions can contain fields. To evaluate one of these expressions, use GetFields() to find all the fields used directly or indirectly by the expression, then create a fieldToValue map specifying the value for each of these fields, and call Evaluate(fieldToValue).

func (*BoolExpr) Evaluate 

func (self *BoolExpr) Evaluate(fieldToValue map[string]utils.CValueEnclosure) (bool, error)

Evaluate this BoolExpr to a boolean, replacing each field in the expression with the value specified by fieldToValue. Each field listed by GetFields() must be in fieldToValue.

func (*BoolExpr) GetFields 

func (self *BoolExpr) GetFields() []string

func (*BoolExpr) GetNullFields 

func (self *BoolExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

type BoolOperator 

type BoolOperator uint8
const (
	BoolOpNot BoolOperator = iota
	BoolOpAnd
	BoolOpOr
)

type BucketHolder 

type BucketHolder struct {
	GroupByValues []string
	MeasureVal    map[string]interface{}
}

type BucketResult 

type BucketResult struct {
	ElemCount   uint64                           // total number of elements in bucket
	StatRes     map[string]utils.CValueEnclosure // results of statistic functions
	BucketKey   interface{}                      // bucket key
	GroupByKeys []string
}

func (*BucketResult) GetBucketValueForGivenField 

func (br *BucketResult) GetBucketValueForGivenField(fieldName string) (interface{}, int, bool)

GetBucketValueForGivenField returns the value of the bucket for the given field name. It checks for the field in the statistic results and group by keys. If the field is not found, it returns nil. The first return value is the value of the field. The second return value is the index of the field in the group by keys. The third return value is a boolean indicating if the field was found in the Statistic results. For Statistic results, the index is -1. and Bool is true. For GroupBy keys, the index is the index of the field in the group by keys; given that the Bucket key is a List Type. and Bool is false. Otherwise, the index is -1 and Bool is false.

func (*BucketResult) SetBucketValueForGivenField 

func (br *BucketResult) SetBucketValueForGivenField(fieldName string, value interface{}, index int, isStatRes bool) error

Can only be used for GroupBy keys. SetBucketValueForGivenField sets the value of the bucket for the given field name. The Value must be of type string or []string. If the value is of type []string, it is converted to a string.

type Cluster 

type Cluster struct {
	Field     string
	Threshold float64
	Match     string // termlist, termset, ngramset
	Delims    string
}

type CmiContainer 

type CmiContainer struct {
	CmiType uint8
	Loaded  bool
	Bf      *bloom.BloomFilter
	Ranges  map[string]*Numbers
}

a holder struct for holding a cmi for a single block. Based on CmiType, either Bf or Ranges will be defined

type ColSizeInfo 

type ColSizeInfo struct {
	CmiSize uint64 `json:"cmiSize"`
	CsgSize uint64 `json:"csgSize"`
}

type ColumnsRequest 

type ColumnsRequest struct {
	RenameColumns            map[string]string // maps original column name to new column name
	ExcludeColumns           []string          // names of columns to exclude
	IncludeColumns           []string          // names of columns to include
	IncludeValues            []*IncludeValue   // values of columns to include. Maps column name to index in column
	RenameAggregationColumns map[string]string // values of aggregations to rename
	Logfmt                   bool              // true if logfmt request
}

type ConcatAtom 

type ConcatAtom struct {
	IsField  bool
	Value    string
	TextExpr *TextExpr
}

type ConcatExpr 

type ConcatExpr struct {
	Atoms []*ConcatAtom
}

func (*ConcatExpr) Evaluate 

func (self *ConcatExpr) Evaluate(fieldToValue map[string]utils.CValueEnclosure) (string, error)

Concatenate all the atoms in this ConcatExpr, replacing all fields with the values specified by fieldToValue. Each field listed by GetFields() must be in fieldToValue.

func (*ConcatExpr) GetFields 

func (self *ConcatExpr) GetFields() []string

func (*ConcatExpr) GetNullFields 

func (self *ConcatExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

type Condition 

type Condition struct {
	FilterCriteria []*FilterCriteria // raw conditions to check
	NestedNodes    []*ASTNode        // nested conditions to check
}

A condition struct defines the FilterConditions and ASTNodes that exist as a part of a single condition

func (*Condition) JoinCondition 

func (c *Condition) JoinCondition(add *Condition)

type ConditionExpr 

type ConditionExpr struct {
	Op                  string //if, case, coalesce
	BoolExpr            *BoolExpr
	TrueValue           *ValueExpr //if bool expr is true, take this value
	FalseValue          *ValueExpr
	ConditionValuePairs []*ConditionValuePair
	ValueList           []*ValueExpr
}

func (*ConditionExpr) EvaluateCondition 

func (self *ConditionExpr) EvaluateCondition(fieldToValue map[string]utils.CValueEnclosure) (string, error)

Field may come from BoolExpr or ValueExpr

func (*ConditionExpr) GetFields 

func (self *ConditionExpr) GetFields() []string

Append all the fields in ConditionExpr

func (*ConditionExpr) GetNullFields 

func (self *ConditionExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

type ConditionValuePair 

type ConditionValuePair struct {
	Condition *BoolExpr
	Value     *ValueExpr
}

type Data 

type Data struct {
	ResultType parser.ValueType `json:"resultType"`
	Result     []Result         `json:"result,omitempty"`
}

type DedupExpr 

type DedupExpr struct {
	Limit              uint64
	FieldList          []string // Must have FieldList
	DedupOptions       *DedupOptions
	DedupSortEles      []*SortElement
	DedupSortAscending []int // Derived from DedupSortEles.SortByAsc values.

	// DedupCombinations maps combinations to a map mapping the record index
	// (of all included records for this combination) to the sort values for
	// that record. For example, if Limit is 3, each inner map will have at
	// most 3 entries and will store the index and sort values of the top 3
	// records for that combination.
	DedupCombinations map[string]map[int][]SortValue
	PrevCombination   string

	DedupRecords         map[string]map[string]interface{}
	NumProcessedSegments uint64
	// contains filtered or unexported fields
}

func (*DedupExpr) AcquireProcessedSegmentsLock 

func (self *DedupExpr) AcquireProcessedSegmentsLock()

func (*DedupExpr) ReleaseProcessedSegmentsLock 

func (self *DedupExpr) ReleaseProcessedSegmentsLock()

type DedupOptions 

type DedupOptions struct {
	Consecutive bool
	KeepEmpty   bool
	KeepEvents  bool
}

type Downsampler 

type Downsampler struct {
	Interval   int
	Unit       string
	CFlag      bool
	Aggregator Aggregation
}

func (*Downsampler) GetIntervalTimeInSeconds 

func (ds *Downsampler) GetIntervalTimeInSeconds() uint32

type EvalFuncChecker 

type EvalFuncChecker struct{}

func (EvalFuncChecker) IsUnsupported 

func (c EvalFuncChecker) IsUnsupported(funcName string) bool

type Expression 

type Expression struct {
	LeftInput    *ExpressionInput   // left expression input for operator
	ExpressionOp ArithmeticOperator // operator, used if complex expression that relates keys
	RightInput   *ExpressionInput   // right expression input for operator
}

expressions are used for SegReaders to parse and search segment files If expressionOp == nil, only leftInput is not nil else, only one of left/right ExpressionInput literal or columnName will be non empty

  • right expressionInput may not exist if op only needs one input (i.e. NOT_NULL)

i.e field2 * 0.2, is Expression{leftInput=ExpressionInput{columnName=field2}, op=Multiply, rightInput=ExpressionInput{literal=0.2}} for just literal 39, Expression{leftInput=ExpressionInput{literal=39}}

func (*Expression) IsTimeExpression 

func (exp *Expression) IsTimeExpression() bool

type ExpressionFilter 

type ExpressionFilter struct {
	LeftInput      *FilterInput         // left input to filterOperator
	FilterOperator utils.FilterOperator // how to logField in logline (i.e logField=filterString, logField >= filterValue)
	RightInput     *FilterInput         // right input to filterOperator
}

ExpressionFilter denotes a single expression to search for in a log record

func (*ExpressionFilter) GetAllColumns 

func (e *ExpressionFilter) GetAllColumns() map[string]bool

func (*ExpressionFilter) IsTimeRange 

func (e *ExpressionFilter) IsTimeRange() bool

type ExpressionInput 

type ExpressionInput struct {
	ColumnValue *DtypeEnclosure // column value: "0", "abc", "abcd*", "0.213"
	ColumnName  string          // column name for expression: "col1", "col2", ... "colN"
}

only one field will be non-nil literal can either be a string or a json.Number

type FileType 

type FileType int
const (
	Cmi    FileType = iota // columnar micro index
	Csg                    // columnar seg file
	Bsu                    // block summary
	Sid                    // segment info
	Pqmr                   // segment persistent query matched results
	Rollup                 // rollup files
	Sst                    // Segment column stats files
)

type FilterBucket 

type FilterBucket struct {
	FilterCriteria []*FilterCondition
	CatchAllBucket bool
	Count          uint64 // limit number of resulting histograms
}

type FilterCondition 

type FilterCondition struct {
	Criteria *FilterCriteria
	Name     string
}

type FilterCriteria 

type FilterCriteria struct {
	MatchFilter      *MatchFilter      // match filter to check multiple words in a column
	ExpressionFilter *ExpressionFilter // expression filter to check a single expression in a column
}

Top level filter criteria condition that define either a MatchFilter or ExpressionFilter. Only one will be defined, never both

func (*FilterCriteria) GetAllColumns 

func (f *FilterCriteria) GetAllColumns() map[string]bool

func (*FilterCriteria) IsTimeRange 

func (f *FilterCriteria) IsTimeRange() bool

type FilterInput 

type FilterInput struct {
	SubTree       *ASTNode    // root of ASTNode
	SubtreeResult string      // result of processing subtree
	Expression    *Expression // expression for filterInput
}

Input for filter operator can either be the result of a ASTNode or an expression either subtree or expression is defined, but never both and never neither

type FilterStringExpr 

type FilterStringExpr struct {
	StringValue  string
	EvalBoolExpr *BoolExpr
	SearchNode   interface{} // type: *ast.Node while parsing, later evaluated to ASTNode
}

type FormatResultsRequest 

type FormatResultsRequest struct {
	MVSeparator   string         // separator for multi-value fields. Default= "OR"
	MaxResults    uint64         // max number of results to return
	EmptyString   string         // string to return if no results are found. Default= "NOT()"
	RowColOptions *RowColOptions // options for row column
}

formats the results into a single result and places that result into a new field called search.

type Function 

type Function struct {
	MathFunction  utils.MathFunctions
	RangeFunction utils.RangeFunctions //range function to apply, only one of these will be non nil
	ValueList     []string
	TimeWindow    float64 //E.g: rate(metrics[1m]), extract 1m and convert to seconds
	Step          float64 //E.g: rate(metrics[5m:1m]), extract 1m and convert to seconds
	TimeFunction  utils.TimeFunctions
}

func (Function) ShallowClone 

func (metricFunc Function) ShallowClone() *Function

type GroupByRequest 

type GroupByRequest struct {
	MeasureOperations []*MeasureAggregator
	GroupByColumns    []string
	AggName           string // name of aggregation
	BucketCount       int
}

type HostMetaData 

type HostMetaData struct {
	MetaFileName string `json:"metaFileName"`
}

type IncludeValue 

type IncludeValue struct {
	Index   int    //index of value in original column
	ColName string //original column name
	Label   string //new label of value in record
}

type Label 

type Label struct {
	Name, Value string
}

type LetColumnsRequest 

type LetColumnsRequest struct {
	MultiColsRequest     *MultiColLetRequest
	SingleColRequest     *SingleColLetRequest
	ValueColRequest      *ValueExpr
	RexColRequest        *RexExpr
	StatisticColRequest  *StatisticExpr
	RenameColRequest     *RenameExpr
	DedupColRequest      *DedupExpr
	SortColRequest       *SortExpr
	NewColName           string
	MultiValueColRequest *MultiValueColLetRequest
	FormatResults        *FormatResultsRequest // formats the results into a single result and places that result into a new field called search.
}

Only NewColName and one of the other fields should have a value

type LimitExpr 

type LimitExpr struct {
	IsTop          bool // True: keeps the N highest scoring distinct values of the split-by field
	Num            int
	LimitScoreMode LimitScoreMode
}

type LimitScoreMode 

type LimitScoreMode uint8

type MBlockSummary 

type MBlockSummary struct {
	Blknum uint16
	HighTs uint32
	LowTs  uint32
}

NOTE: Change the value oF SIZE_OF_MBSUM each time this struct is updated

func (*MBlockSummary) FlushSummary 

func (mbs *MBlockSummary) FlushSummary(fName string) ([]byte, error)

Format of block summary file [version - 1 byte][blk num - 2 bytes][high ts - 4 bytes][low ts - 4 bytes]

func (*MBlockSummary) Reset 

func (mbs *MBlockSummary) Reset()

func (*MBlockSummary) UpdateTimeRange 

func (mbs *MBlockSummary) UpdateTimeRange(ts uint32)

type MatchDictArrayRequest 

type MatchDictArrayRequest struct {
	MatchKey   []byte
	MatchValue *utils.DtypeEnclosure
}

type MatchFilter 

type MatchFilter struct {
	MatchColumn    string                 // column to search for
	MatchWords     [][]byte               // all words to search for
	MatchOperator  utils.LogicalOperator  // how to combine matchWords
	MatchPhrase    []byte                 //whole string to search for in case of MatchPhrase query
	MatchDictArray *MatchDictArrayRequest //array to search for in case of jaeger query
	MatchType      MatchFilterType
	NegateMatch    bool
	RegexpString   string // Do not manually set this. Use SetRegexp(). This is only public to allow for GOB encoding MatchFilter.
	// contains filtered or unexported fields
}

MatchFilter searches for all words in matchWords in the column matchColumn The matchOperator defines if all or any of the matchWords need to be present

func (*MatchFilter) GetAllBlockBloomKeysToSearch 

func (match *MatchFilter) GetAllBlockBloomKeysToSearch() (map[string]bool, bool, LogicalOperator)

func (*MatchFilter) GetQueryInfo 

func (m *MatchFilter) GetQueryInfo() *QueryInfo

func (*MatchFilter) GetRegexp 

func (m *MatchFilter) GetRegexp() (*regexp.Regexp, error)

func (*MatchFilter) IsMatchAll 

func (mf *MatchFilter) IsMatchAll() bool

we expect a matchColumn == * AND matchWords == *

func (*MatchFilter) SetRegexp 

func (m *MatchFilter) SetRegexp(compileRegex *regexp.Regexp)

type MatchFilterType 

type MatchFilterType uint8
const (
	MATCH_WORDS MatchFilterType = iota + 1
	MATCH_PHRASE
	MATCH_DICT_ARRAY
)

type MathEvaluator 

type MathEvaluator struct {
	MathCol         string              `json:"mathCol,omitempty"`
	MathFunc        utils.MathFunctions `json:"mathFunc,omitempty"`
	StrEnc          string              `json:"strEnc,omitempty"`
	ValueColRequest *ValueExpr          `json:"valueCol,omitempty"`
}

type MeasureAggregator 

type MeasureAggregator struct {
	MeasureCol         string                   `json:"measureCol,omitempty"`
	MeasureFunc        utils.AggregateFunctions `json:"measureFunc,omitempty"`
	StrEnc             string                   `json:"strEnc,omitempty"`
	ValueColRequest    *ValueExpr               `json:"valueColRequest,omitempty"`
	OverrodeMeasureAgg *MeasureAggregator       `json:"overrideFunc,omitempty"`
	Param              string
}

func (*MeasureAggregator) String 

func (ma *MeasureAggregator) String() string

type MemoryTracker 

type MemoryTracker struct {
	TotalAllocatableBytes     uint64 // total bytes that can be allocated. This should not include CmiRuntimeAllocatedBytes
	CmiInMemoryAllocatedBytes uint64
	CmiRuntimeAllocatedBytes  uint64
	SegSearchRequestedBytes   uint64
	SegWriterUsageBytes       uint64
	SegStoreSummary           *AllSegStoreSummary
	SsmInMemoryAllocatedBytes uint64
	MetricsSegmentMaxSize     uint64
}

type MetricQueryAgg 

type MetricQueryAgg struct {
	AggBlockType    MetricQueryAggBlockType
	AggregatorBlock *Aggregation
	FunctionBlock   *Function
	Next            *MetricQueryAgg
}

type MetricQueryAggBlockType 

type MetricQueryAggBlockType int
const (
	AggregatorBlock MetricQueryAggBlockType = iota + 1
	FunctionBlock
)

type MetricsMeta 

type MetricsMeta struct {
	MSegmentDir        string          `json:"mSegmentDir"`
	NumBlocks          uint16          `json:"numBlocks"`
	BytesReceivedCount uint64          `json:"bytesReceivedCount"`
	OnDiskBytes        uint64          `json:"onDiskBytes"`
	TagKeys            map[string]bool `json:"tagKeys"`
	EarliestEpochSec   uint32          `json:"earliestEpochSec"`
	LatestEpochSec     uint32          `json:"latestEpochSec"`
	TTreeDir           string          `json:"TTreeDir"`
	DatapointCount     uint64          `json:"approximateDatapointCount"`
	OrgId              uint64          `json:"orgid"`
}

type MetricsQuery 

type MetricsQuery struct {
	MetricName             string            // metric name to query for.
	MetricOperator         utils.TagOperator // operator to apply on metric name
	MetricNameRegexPattern string            // regex pattern to apply on metric name
	QueryHash              uint64            // hash of the query
	HashedMName            uint64
	PqlQueryType           parser.ValueType // promql query type
	Aggregator             Aggregation
	Function               Function
	Downsampler            Downsampler
	TagsFilters            []*TagsFilter    // all tags filters to apply
	TagIndicesToKeep       map[int]struct{} // indices of tags to keep in the result
	SelectAllSeries        bool             //flag to select all series - for promQl

	MQueryAggs *MetricQueryAgg

	OrgId uint64 // organization id

	ExitAfterTagsSearch bool // flag to exit after raw tags search
	TagValueSearchOnly  bool // flag to search only tag values
	GetAllLabels        bool // flag to get all label sets for each time series
	Groupby             bool // flag to group by tags
	GroupByMetricName   bool // flag to group by metric name
	// contains filtered or unexported fields
}

Struct to represent a single metrics query request.

func (*MetricsQuery) GetNumStarFilters 

func (mq *MetricsQuery) GetNumStarFilters() int

func (*MetricsQuery) GetNumValueFilters 

func (mq *MetricsQuery) GetNumValueFilters() uint32

func (*MetricsQuery) IsRegexOnMetricName 

func (mQuery *MetricsQuery) IsRegexOnMetricName() bool

func (*MetricsQuery) ReorderTagFilters 

func (mq *MetricsQuery) ReorderTagFilters()

Fixes the order of tags filters to be in the following order: 1. other tag filters 2. * tag filters

type MetricsQueryProcessingMetrics 

type MetricsQueryProcessingMetrics struct {
	UpdateLock                 *sync.Mutex
	NumMetricsSegmentsSearched uint64
	NumTSOFilesLoaded          uint64
	NumTSGFilesLoaded          uint64
	NumSeriesSearched          uint64
	TimeLoadingTSOFiles        time.Duration
	TimeLoadingTSGFiles        time.Duration
}

func (*MetricsQueryProcessingMetrics) IncrementNumMetricsSegmentsSearched 

func (qm *MetricsQueryProcessingMetrics) IncrementNumMetricsSegmentsSearched(records uint64)

func (*MetricsQueryProcessingMetrics) IncrementNumSeriesSearched 

func (qm *MetricsQueryProcessingMetrics) IncrementNumSeriesSearched(records uint64)

func (*MetricsQueryProcessingMetrics) IncrementNumTSGFilesLoaded 

func (qm *MetricsQueryProcessingMetrics) IncrementNumTSGFilesLoaded(records uint64)

func (*MetricsQueryProcessingMetrics) IncrementNumTSOFilesLoaded 

func (qm *MetricsQueryProcessingMetrics) IncrementNumTSOFilesLoaded(records uint64)

func (*MetricsQueryProcessingMetrics) SetTimeLoadingTSGFiles 

func (qm *MetricsQueryProcessingMetrics) SetTimeLoadingTSGFiles(ttime time.Duration)

func (*MetricsQueryProcessingMetrics) SetTimeLoadingTSOFiles 

func (qm *MetricsQueryProcessingMetrics) SetTimeLoadingTSOFiles(ttime time.Duration)

type MetricsQueryRequest 

type MetricsQueryRequest struct {
	TimeRange    dtu.MetricsTimeRange
	MetricsQuery MetricsQuery
}

Struct to represent the metrics query request and its corresponding timerange

type MetricsQueryResponse 

type MetricsQueryResponse struct {
	MetricName string             `json:"metric"`
	Tags       map[string]string  `json:"tags"`
	Dps        map[uint32]float64 `json:"dps"`
}

type MetricsQueryResponsePromQl 

type MetricsQueryResponsePromQl struct {
	Status    string   `json:"status"` //success/error
	Data      Data     `json:"data"`
	ErrorType string   `json:"errorType"`
	Error     string   `json:"error"`
	Warnings  []string `json:"warnings"`
}

type MetricsSearchRequest 

type MetricsSearchRequest struct {
	MetricsKeyBaseDir    string
	BlocksToSearch       map[uint16]bool
	BlkWorkerParallelism uint
	QueryType            SegType
	AllTagKeys           map[string]bool
	UnrotatedMetricNames map[string]bool
}

type MultiColLetRequest 

type MultiColLetRequest struct {
	LeftCName  string
	Oper       utils.LogicalAndArithmeticOperator
	RightCName string
}

type MultiValueColLetRequest 

type MultiValueColLetRequest struct {
	Command         string // name of the command: makemv, mvcombine, mvexpand, etc.
	ColName         string
	DelimiterString string // delimiter string to split the column value. default is " " (single space)
	IsRegex         bool
	AllowEmpty      bool // if true, empty strings are allowed in the split values. default is false
	Setsv           bool // if true, split values are combined into a single value. default is false
}

type NodeAggregation 

type NodeAggregation struct {
	AggregationFunctions utils.AggregateFunctions // function to apply on results of children (e.g. min, max)
	AggregationColumns   []string                 // column names to aggregate on (i.e avg over which column name?)
}

type NodeResult 

type NodeResult struct {
	AllRecords                []*utils.RecordResultContainer
	ErrList                   []error
	Histogram                 map[string]*AggregationResult
	TotalResults              *QueryCount
	VectorResultValue         float64
	RenameColumns             map[string]string
	SegEncToKey               map[uint16]string
	TotalRRCCount             uint64
	MeasureFunctions          []string        `json:"measureFunctions,omitempty"`
	MeasureResults            []*BucketHolder `json:"measure,omitempty"`
	GroupByCols               []string        `json:"groupByCols,omitempty"`
	Qtype                     string          `json:"qtype,omitempty"`
	BucketCount               int             `json:"bucketCount,omitempty"`
	PerformAggsOnRecs         bool            // if true, perform aggregations on records that are returned from rrcreader.go
	RecsAggsType              PipeCommandType // To determine Whether it is GroupByType or MeasureAggsType
	GroupByRequest            *GroupByRequest
	MeasureOperations         []*MeasureAggregator
	NextQueryAgg              *QueryAggregators
	RecsAggsBlockResults      interface{}              // Evaluates to *blockresults.BlockResults
	RecsAggsColumnKeysMap     map[string][]interface{} // map of column name to column keys for GroupBy Recs
	RecsAggsProcessedSegments uint64
	RecsRunningSegStats       []*SegStats
	TransactionEventRecords   map[string]map[string]interface{}
	TransactionsProcessed     map[string]map[string]interface{}
	ColumnsOrder              map[string]int
}

A helper struct to keep track of errors and results together In cases of partial failures, both logLines and errList can be defined

func (*NodeResult) ApplyScroll 

func (nr *NodeResult) ApplyScroll(scroll int)

type Numbers 

type Numbers struct {
	Min_uint64  uint64             `json:"Min_uint64"`
	Max_uint64  uint64             `json:"Max_uint64"`
	Min_int64   int64              `json:"Min_int64"`
	Max_int64   int64              `json:"Max_int64"`
	Min_float64 float64            `json:"Min_float64"`
	Max_float64 float64            `json:"Max_float64"`
	NumType     utils.RangeNumType `json:"NumType"`
}

func (*Numbers) Copy 

func (n *Numbers) Copy() *Numbers

type NumericExpr 

type NumericExpr struct {
	NumericExprMode NumericExprMode

	IsTerminal bool

	// Only used when IsTerminal is true.
	ValueIsField bool
	Value        string

	// Only used when IsTerminal is false.
	Op    string // Including arithmetic, mathematical and text functions ops
	Left  *NumericExpr
	Right *NumericExpr
	Val   *StringExpr
}

func (*NumericExpr) Evaluate 

func (self *NumericExpr) Evaluate(fieldToValue map[string]utils.CValueEnclosure) (float64, error)

Evaluate this NumericExpr to a float, replacing each field in the expression with the value specified by fieldToValue. Each field listed by GetFields() must be in fieldToValue.

func (*NumericExpr) GetFields 

func (self *NumericExpr) GetFields() []string

func (*NumericExpr) GetNullFields 

func (self *NumericExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

type NumericExprMode 

type NumericExprMode uint8

type NumericStats 

type NumericStats struct {
	Min   utils.NumTypeEnclosure `json:"min,omitempty"`
	Max   utils.NumTypeEnclosure `json:"max,omitempty"`
	Sum   utils.NumTypeEnclosure `json:"sum,omitempty"`
	Dtype utils.SS_DTYPE         `json:"Dtype,omitempty"` // Dtype shared across min,max, and sum
}

func (*NumericStats) Merge 

func (ss *NumericStats) Merge(other *NumericStats)

type OTSDBMetricsQueryExpDownsampler 

type OTSDBMetricsQueryExpDownsampler struct {
	Interval   string `json:"interval"`
	Aggregator string `json:"aggregator"`
}

type OTSDBMetricsQueryExpFilter 

type OTSDBMetricsQueryExpFilter struct {
	Tags []OTSDBMetricsQueryExpTags `json:"tags"`
	Id   string                     `json:"id"`
}

type OTSDBMetricsQueryExpMetric 

type OTSDBMetricsQueryExpMetric struct {
	Id         string            `json:"id"`
	MetricName string            `json:"metric"`
	Filter     string            `json:"filter"`
	Aggregator string            `json:"aggregator"`
	FillPolicy map[string]string `json:"fillPolicy"`
}

type OTSDBMetricsQueryExpOutput 

type OTSDBMetricsQueryExpOutput struct {
	Id    string `json:"id"`
	Alias string `json:"alias"`
}

type OTSDBMetricsQueryExpRequest 

type OTSDBMetricsQueryExpRequest struct {
	Time        OTSDBMetricsQueryExpTime       `json:"time"`
	Filters     []OTSDBMetricsQueryExpFilter   `json:"filters"`
	Metrics     []OTSDBMetricsQueryExpMetric   `json:"metrics"`
	Expressions []OTSDBMetricsQueryExpressions `json:"expressions"`
	Outputs     []OTSDBMetricsQueryExpOutput   `json:"outputs"`
}

type OTSDBMetricsQueryExpTags 

type OTSDBMetricsQueryExpTags struct {
	Type    string `json:"type"`
	Tagk    string `json:"tagk"`
	Filter  string `json:"filter"`
	GroupBy bool   `json:"groupBy"`
}

type OTSDBMetricsQueryExpTime 

type OTSDBMetricsQueryExpTime struct {
	Start       interface{}                     `json:"start"`
	End         interface{}                     `json:"end"`
	Timezone    string                          `json:"timezone"`
	Aggregator  string                          `json:"aggregator"`
	Downsampler OTSDBMetricsQueryExpDownsampler `json:"downsampler"`
}

type OTSDBMetricsQueryExpressions 

type OTSDBMetricsQueryExpressions struct {
	Id  string `json:"id"`
	Exp string `json:"exp"`
}

type OrderByAggregator 

type OrderByAggregator struct {
	ColumnName    string
	IncreaseOrder bool
}

type OutputTransforms 

type OutputTransforms struct {
	HarcodedCol            []string
	RenameHardcodedColumns map[string]string
	OutputColumns          *ColumnsRequest    // post processing on output columns
	LetColumns             *LetColumnsRequest // let columns processing on output columns
	FilterRows             *BoolExpr          // discard rows failing some condition
	MaxRows                uint64             // if 0, get all results; else, get at most this many
	RowsAdded              uint64             // number of rows added to the result. This is used in conjunction with MaxRows.
}

type PipeCommandType 

type PipeCommandType uint8
const (
	OutputTransformType PipeCommandType = iota + 1
	MeasureAggsType
	GroupByType
	TransactionType
	VectorArithmeticExprType
)

type QueryAggregators 

type QueryAggregators struct {
	PipeCommandType      PipeCommandType
	OutputTransforms     *OutputTransforms
	MeasureOperations    []*MeasureAggregator
	MathOperations       []*MathEvaluator
	VectorArithmeticExpr *NumericExpr
	TimeHistogram        *TimeBucket     // Request for time histograms
	GroupByRequest       *GroupByRequest // groupby aggregation request
	Sort                 *SortRequest    // how to sort resulting data
	EarlyExit            bool            // should query early exit
	BucketLimit          int
	ShowRequest          *ShowRequest
	TableName            string
	TransactionArguments *TransactionArguments
	StatsOptions         *StatsOptions
	Next                 *QueryAggregators
	Limit                int
}

func InitDefaultQueryAggregations 

func InitDefaultQueryAggregations() *QueryAggregators

Init default query aggregators. By default, a descending sort is added

func (*QueryAggregators) CanLimitBuckets 

func (qa *QueryAggregators) CanLimitBuckets() bool

func (*QueryAggregators) GetSortLimit 

func (qa *QueryAggregators) GetSortLimit() uint64

func (*QueryAggregators) HasDedupBlock 

func (qa *QueryAggregators) HasDedupBlock() bool

func (*QueryAggregators) HasDedupBlockInChain 

func (qa *QueryAggregators) HasDedupBlockInChain() bool

func (*QueryAggregators) HasGroupByOrMeasureAggsInBlock 

func (qa *QueryAggregators) HasGroupByOrMeasureAggsInBlock() bool

func (*QueryAggregators) HasGroupByOrMeasureAggsInChain 

func (qa *QueryAggregators) HasGroupByOrMeasureAggsInChain() bool

func (*QueryAggregators) HasQueryAggergatorBlock 

func (qa *QueryAggregators) HasQueryAggergatorBlock() bool

To determine whether it contains certain specific AggregatorBlocks, such as: Rename Block, Rex Block, FilterRows, MaxRows...

func (*QueryAggregators) HasQueryAggergatorBlockInChain 

func (qa *QueryAggregators) HasQueryAggergatorBlockInChain() bool

func (*QueryAggregators) HasRexBlockInChainWithStats 

func (qa *QueryAggregators) HasRexBlockInChainWithStats() bool

func (*QueryAggregators) HasRexBlockInQA 

func (qa *QueryAggregators) HasRexBlockInQA() bool

func (*QueryAggregators) HasSortBlock 

func (qa *QueryAggregators) HasSortBlock() bool

func (*QueryAggregators) HasSortBlockInChain 

func (qa *QueryAggregators) HasSortBlockInChain() bool

func (*QueryAggregators) HasTransactionArguments 

func (qa *QueryAggregators) HasTransactionArguments() bool

func (*QueryAggregators) HasTransactionArgumentsInChain 

func (qa *QueryAggregators) HasTransactionArgumentsInChain() bool

func (*QueryAggregators) HasValueColRequest 

func (qa *QueryAggregators) HasValueColRequest() bool

To determine whether it contains ValueColRequest

func (*QueryAggregators) HasValuesFunc 

func (qa *QueryAggregators) HasValuesFunc() bool

To determine whether it contains Aggregate Func: Values()

func (*QueryAggregators) IsAggsEmpty 

func (qa *QueryAggregators) IsAggsEmpty() bool

func (*QueryAggregators) IsStatisticBlockEmpty 

func (qa *QueryAggregators) IsStatisticBlockEmpty() bool

func (*QueryAggregators) UsedByTimechart 

func (qa *QueryAggregators) UsedByTimechart() bool

type QueryArithmetic 

type QueryArithmetic struct {
	OperationId uint64
	LHS         uint64
	RHS         uint64
	LHSExpr     *QueryArithmetic
	RHSExpr     *QueryArithmetic
	ConstantOp  bool
	Operation   utils.LogicalAndArithmeticOperator
	ReturnBool  bool // If a comparison operator, return 0/1 rather than filtering.
	Constant    float64
	// maps groupid to a map of ts to value. This aggregates DsResults based on the aggregation function
	Results        map[string]map[uint32]float64
	OperatedState  bool //true if operation has been executed
	VectorMatching *VectorMatching
}

Struct to represent the metrics arithmetic request and its corresponding timerange

type QueryContext 

type QueryContext struct {
	TableInfo *TableInfo
	SizeLimit uint64
	Scroll    int
	Orgid     uint64
}

New struct for passin query params

func InitQueryContext 

func InitQueryContext(indexRequest string, sizeLimit uint64, scroll int, orgid uint64, es bool) *QueryContext

func InitQueryContextWithTableInfo 

func InitQueryContextWithTableInfo(ti *TableInfo, sizeLimit uint64, scroll int, orgid uint64, es bool) *QueryContext

func (*QueryContext) GetNumTables 

func (qc *QueryContext) GetNumTables() int

gets the number of tables that will be queried

type QueryCount 

type QueryCount struct {
	TotalCount uint64 // total number of
	Op         utils.FilterOperator
	EarlyExit  bool // if early exit was requested or not
}

type QueryInfo 

type QueryInfo struct {
	ColName string
	KValDte []byte          // only non-nil for `MatchDictArray` requests
	QValDte *DtypeEnclosure // column value to use for raw check. May be nil if query is match filter
}

type QueryProcessingMetrics 

type QueryProcessingMetrics struct {
	NumRecordsMatched     uint64
	NumBlocksWithMatch    uint64
	NumBlocksToRawSearch  uint64
	NumBlocksInSegment    uint64
	NumRecordsToRawSearch uint64
	NumRecordsUnmatched   uint64
}

func (*QueryProcessingMetrics) IncrementNumBlocksToRawSearch 

func (qm *QueryProcessingMetrics) IncrementNumBlocksToRawSearch(records uint64)

func (*QueryProcessingMetrics) IncrementNumBlocksWithMatch 

func (qm *QueryProcessingMetrics) IncrementNumBlocksWithMatch(nBlocks uint64)

func (*QueryProcessingMetrics) IncrementNumRecordsNoMatch 

func (qm *QueryProcessingMetrics) IncrementNumRecordsNoMatch(nBlocks uint64)

func (*QueryProcessingMetrics) IncrementNumRecordsWithMatch 

func (qm *QueryProcessingMetrics) IncrementNumRecordsWithMatch(nBlocks uint64)

func (*QueryProcessingMetrics) SetNumBlocksInSegFile 

func (qm *QueryProcessingMetrics) SetNumBlocksInSegFile(records uint64)

func (*QueryProcessingMetrics) SetNumBlocksToRawSearch 

func (qm *QueryProcessingMetrics) SetNumBlocksToRawSearch(records uint64)

func (*QueryProcessingMetrics) SetNumRecordsMatched 

func (qm *QueryProcessingMetrics) SetNumRecordsMatched(records uint64)

func (*QueryProcessingMetrics) SetNumRecordsToRawSearch 

func (qm *QueryProcessingMetrics) SetNumRecordsToRawSearch(records uint64)

func (*QueryProcessingMetrics) SetNumRecordsUnmatched 

func (qm *QueryProcessingMetrics) SetNumRecordsUnmatched(records uint64)

type QueryType 

type QueryType uint8

func (QueryType) String 

func (qtype QueryType) String() string

type RangeBucket 

type RangeBucket struct {
	BucketKey     string  // column name to group
	Interval      float64 // interval of request
	MinBucketSize uint64  // minimum count for bucket to show up
	Count         uint64  // max number of unique histograms to return
}

type RangeStat 

type RangeStat struct {
	Min float64
	Max float64
}

type RenameExpr 

type RenameExpr struct {
	RenameExprMode  RenameExprMode
	OriginalPattern string
	NewPattern      string
}

func (*RenameExpr) CheckIfMatch 

func (self *RenameExpr) CheckIfMatch(colName string) bool

Check if colName match the OriginalPattern

func (*RenameExpr) Evaluate 

func (self *RenameExpr) Evaluate(fieldToValue map[string]utils.CValueEnclosure, fieldName string) (string, error)

func (*RenameExpr) GetFields 

func (self *RenameExpr) GetFields() []string

func (*RenameExpr) GetNullFields 

func (self *RenameExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

func (*RenameExpr) ProcessRenameRegexExpression 

func (self *RenameExpr) ProcessRenameRegexExpression(colName string) (string, error)

Check if colName matches the specified pattern and replace wildcards to generate a new colName.

func (*RenameExpr) RemoveBucketHolderGroupByColumnsByIndex 

func (self *RenameExpr) RemoveBucketHolderGroupByColumnsByIndex(bucketHolder *BucketHolder, groupByCols []string, indexToRemove []int)

Remove unused GroupByVals in Bucket Holder

func (*RenameExpr) RemoveBucketResGroupByColumnsByIndex 

func (self *RenameExpr) RemoveBucketResGroupByColumnsByIndex(bucketResult *BucketResult, indexToRemove []int)

func (*RenameExpr) RemoveColsByIndex 

func (self *RenameExpr) RemoveColsByIndex(strs []string, indexToRemove []int) []string

type RenameExprMode 

type RenameExprMode uint8

type Result 

type Result struct {
	Metric map[string]string `json:"metric"`
	Value  []interface{}     `json:"values"`
}

type RexExpr 

type RexExpr struct {
	Pattern     string
	FieldName   string
	RexColNames []string
}

func (*RexExpr) Evaluate 

func (self *RexExpr) Evaluate(fieldToValue map[string]utils.CValueEnclosure, rexExp *regexp.Regexp) (map[string]string, error)

func (*RexExpr) GetFields 

func (self *RexExpr) GetFields() []string

func (*RexExpr) GetNullFields 

func (self *RexExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

type RowColOptions 

type RowColOptions struct {
	RowPrefix       string // prefix for row. Default= "("
	ColumnPrefix    string // prefix for column. Default= "("
	ColumnSeparator string // separator for column. Default= "AND"
	ColumnEnd       string // end for column. Default= ")"
	RowSeparator    string // separator for row. Default= "OR"
	RowEnd          string // end for row. Default= ")"
}

type SPathExpr 

type SPathExpr struct {
	InputColName    string // default is set to _raw
	Path            string // the path to the field from which the values need to be extracted.
	IsPathFieldName bool   // If true, the path is the field name and the value is the field value
	OutputColName   string // the name of the column in the output table to which the extracted values will be written. By Default it is set the same as the path.
}

To extract information from the structured data formats XML and JSON.

type SearchCondition 

type SearchCondition struct {
	SearchQueries []*SearchQuery
	SearchNode    []*SearchNode
}

func (*SearchCondition) AddQueryInfo 

func (c *SearchCondition) AddQueryInfo() SearchNodeType

type SearchExpression 

type SearchExpression struct {
	LeftSearchInput  *SearchExpressionInput
	FilterOp         FilterOperator
	RightSearchInput *SearchExpressionInput
	SearchInfo       *SearchInfo
}

A flattened expression used for searching leftSearchInput will always be defined, rightSearchInput may not be depending on filterOp

func (*SearchExpression) GetAllBlockBloomKeysToSearch 

func (searchExp *SearchExpression) GetAllBlockBloomKeysToSearch() (map[string]bool, bool, error)

returns a map with keys, a boolean, and error the map will contain only non wildcarded keys, if bool is true, the searchExpression contained a wildcard

func (*SearchExpression) GetExpressionType 

func (searchExp *SearchExpression) GetExpressionType() SearchQueryType

func (*SearchExpression) GetQueryInfo 

func (se *SearchExpression) GetQueryInfo() *QueryInfo

func (*SearchExpression) IsMatchAll 

func (searchExp *SearchExpression) IsMatchAll() bool

func (*SearchExpression) IsTimeRangeFilter 

func (ef *SearchExpression) IsTimeRangeFilter() bool

type SearchExpressionInput 

type SearchExpressionInput struct {
	ColumnName      string          // columnName to search for
	ComplexRelation *Expression     // complex relations that have columns defined in both sides
	ColumnValue     *DtypeEnclosure // column value: "0", "abc", "abcd*", "0.213"
}

A flattened expression input used for searching TODO: flatten SearchExpressionInput with just []byte input

type SearchInfo 

type SearchInfo struct {
	ColEncoding []byte
	QValDte     *DtypeEnclosure
}

type SearchMetadataHolder 

type SearchMetadataHolder struct {
	BlockSummaries     []*BlockSummary
	BlockSummariesFile string
	SearchTotalMemory  uint64 // total memory that this search would take, BlockSummaries + raw search buffers
}

type SearchNode 

type SearchNode struct {
	AndSearchConditions       *SearchCondition
	OrSearchConditions        *SearchCondition
	ExclusionSearchConditions *SearchCondition
	NodeType                  SearchNodeType // type of search request
}

func (*SearchNode) AddQueryInfoForNode 

func (r *SearchNode) AddQueryInfoForNode() SearchNodeType

func (*SearchNode) GetAllColumnsToSearch 

func (node *SearchNode) GetAllColumnsToSearch() (map[string]bool, bool)

type SearchNodeType 

type SearchNodeType uint8

******************************************* ******************************************* ******************************************* **************** NOTE ********************* any time you add a new element in the searchnode structs or in their sub structs, make sure to adjust the snhasher.go code to update the hashids, else PQS will incorrectly compute the hash id

Also DO NOT change the order of the struct fields, if you do then you have to adjust the order in snhasher.go as well, however in that case the first time when the new code will run, it will create new pqid values for existing queries

******************************************* ******************************************* ******************************************* 

const (
	MatchAllQuery    SearchNodeType = iota // query only needs to know a record's time range, no raw values
	ColumnValueQuery                       // query needs to know >0 non-timestamp column values
	InvalidQuery                           // an invalid query (e.g. invalid column name)
)

func EditQueryTypeForInvalidColumn 

func EditQueryTypeForInvalidColumn(originalType SearchNodeType) SearchNodeType

type SearchQuery 

type SearchQuery struct {
	ExpressionFilter *SearchExpression
	MatchFilter      *MatchFilter
	SearchType       SearchQueryType // type of query
	QueryInfo        *QueryInfo      // query info
}

A Search query is either an expression or match filter Never will both be defined

func GetSearchQueryFromFilterCriteria 

func GetSearchQueryFromFilterCriteria(criteria *FilterCriteria, qid uint64) *SearchQuery

func (*SearchQuery) ExtractRangeFilterFromQuery 

func (query *SearchQuery) ExtractRangeFilterFromQuery(qid uint64) (map[string]string, FilterOperator, bool)

func (*SearchQuery) GetAllBlockBloomKeysToSearch 

func (query *SearchQuery) GetAllBlockBloomKeysToSearch() (map[string]bool, bool, LogicalOperator)

returns map[string]bool, bool, LogicalOperator map is all non-wildcard block bloom keys, bool is if any keyword contained a wildcard, LogicalOperator is if any/all of map keys need to exist

func (*SearchQuery) GetAllColumnsInQuery 

func (query *SearchQuery) GetAllColumnsInQuery() (map[string]bool, bool)

extract all columns from SearchQuery returns a map[string]bool, where key is the column name returns a bool that indicates whether a full wildcard is present (only "*")

func (*SearchQuery) GetQueryInfo 

func (n *SearchQuery) GetQueryInfo() SearchNodeType

func (*SearchQuery) GetQueryType 

func (q *SearchQuery) GetQueryType() SearchNodeType

func (*SearchQuery) IsMatchAll 

func (q *SearchQuery) IsMatchAll() bool

type SearchQueryType 

type SearchQueryType uint8
const (
	MatchAll                   SearchQueryType = iota // match all data
	MatchWords                                        // match words in a single column
	MatchWordsAllColumns                              // match words in any column
	SimpleExpression                                  // simple expression has one column name an operator and a value to compare
	RegexExpression                                   // regex expression has one column and a regex string column value
	RegexExpressionAllColumns                         // apply RegexExpression on all columns
	SimpleExpressionAllColumns                        // apply SimpleExpression on all columns
	ComplexExpression                                 // complex expression relates multiple columns
	MatchDictArraySingleColumn
	MatchDictArrayAllColumns
)

type SegMeta 

type SegMeta struct {
	SegmentKey         string                  `json:"segmentKey"`
	EarliestEpochMS    uint64                  `json:"earliestEpochMs,omitempty"`
	LatestEpochMS      uint64                  `json:"latestEpochMs,omitempty"`
	SegbaseDir         string                  `json:"segbaseDir,omitempty"`
	VirtualTableName   string                  `json:"virtualTableName"`
	RecordCount        int                     `json:"recordCount,omitempty"`
	BytesReceivedCount uint64                  `json:"bytesReceivedCount,omitempty"`
	OnDiskBytes        uint64                  `json:"onDiskBytes,omitempty"`
	ColumnNames        map[string]*ColSizeInfo `json:"columnNames,omitempty"`
	AllPQIDs           map[string]bool         `json:"pqids,omitempty"`
	NumBlocks          uint16                  `json:"numBlocks,omitempty"`
	OrgId              uint64                  `json:"orgid,omitempty"`
}

type SegSetData 

type SegSetData struct {
	InUse          bool   `json:"-"` // is this seg file being used? This will tell cleaner if it can delete it or not
	AccessTime     int64  `json:"AccessTime"`
	Size           uint64 `json:"Size"` // size in bytes of file
	SegSetFileName string `json:"SegSetFileName"`
}

type SegSetDataWithStrFname 

type SegSetDataWithStrFname struct {
	Size       uint64 `json:"Size"`
	IsPresent  bool   `json:"IsPresent"`
	AccessTime int64  `json:"AccessTime"`
	LatestTime uint64 `json:"LatestTime"`
	SegSetFile string `json:"SegSetFile"`
}

type SegSetFile 

type SegSetFile struct {
	SegKey string `json:"SegKey"`
	// identifier for file. If csg, xxhash of column name, if for block micro files empty string, if spqmr, then qid
	Identifier string   `json:"Identifier"`
	FileType   FileType `json:"FileType"`
}

type SegStats 

type SegStats struct {
	IsNumeric   bool
	Count       uint64
	Hll         *hyperloglog.Sketch
	NumStats    *NumericStats
	StringStats *StringStats
	Records     []*utils.CValueEnclosure
}

func (*SegStats) Init 

func (ss *SegStats) Init(rawSegStatJson []byte) error

init SegStats from raw bytes of SegStatsJSON

func (*SegStats) Merge 

func (ss *SegStats) Merge(other *SegStats)

func (*SegStats) ToJSON 

func (ss *SegStats) ToJSON() (*SegStatsJSON, error)

convert SegStats to SegStatsJSON

type SegStatsJSON 

type SegStatsJSON struct {
	IsNumeric bool
	Count     uint64
	RawHll    []byte
	NumStats  *NumericStats
}

json exportable struct for segstats

func (*SegStatsJSON) ToStats 

func (ssj *SegStatsJSON) ToStats() (*SegStats, error)

type SegType 

type SegType uint8
const (
	UNKNOWN SegType = iota
	RAW_SEARCH
	PQS
	UNROTATED_PQS
	UNROTATED_RAW_SEARCH
	SEGMENT_STATS_SEARCH
	UNROTATED_SEGMENT_STATS_SEARCH
	METRICS_SEARCH
	UNROTATED_METRICS_SEARCH
)

func (SegType) String 

func (s SegType) String() string

type SegmentSearchRequest 

type SegmentSearchRequest struct {
	SegmentKey         string
	SearchMetadata     *SearchMetadataHolder
	AllBlocksToSearch  map[uint16]*BlockMetadataHolder // maps all blocks needed to search to the BlockMetadataHolder needed to read
	VirtualTableName   string
	AllPossibleColumns map[string]bool // all possible columns for the segKey
	LatestEpochMS      uint64          // latest epoch time - used for query planning
	SType              SegType
	CmiPassedCnames    map[uint16]map[string]bool // maps blkNum -> colName -> true that have passed the cmi check
	HasMatchedRrc      bool                       // flag to denote matches, so that we decide whether to send a websocket update
}

a struct for raw search to apply search on specific blocks within a file

func (*SegmentSearchRequest) CleanSearchMetadata 

func (ssr *SegmentSearchRequest) CleanSearchMetadata()

function used to nil out block sum and colmeta

func (*SegmentSearchRequest) GetMaxSearchMemorySize 

func (ssr *SegmentSearchRequest) GetMaxSearchMemorySize(sNode *SearchNode, parallelismPerFile int64, bitsetMinSize uint16) uint64

even if only one block will be searched and parallelism=10, we will spawn 10 buffers, although 9 wont be used TODO: more accurate block summaries and colmeta sizing

func (*SegmentSearchRequest) JoinColumnInfo 

func (ssr *SegmentSearchRequest) JoinColumnInfo(toJoin *SegmentSearchRequest)

merges toJoin.SearchColumns with ssr.SearchColumns

func (*SegmentSearchRequest) JoinRequest 

func (ssr *SegmentSearchRequest) JoinRequest(toJoin *SegmentSearchRequest, op LogicalOperator)

* 

Logical operator only dictates how the block numbers should be resolved

the CMIPassed names will always be unioned.

*

type ShowColumns 

type ShowColumns struct {
	InTable string
}

type ShowFilter 

type ShowFilter struct {
	Like string
}

type ShowRequest 

type ShowRequest struct {
	ShowTables     bool
	ShowFilter     *ShowFilter
	ShowTable      string
	ColumnsRequest *ShowColumns
}

type SingleAgg 

type SingleAgg struct {
	MeasureOperations []*MeasureAggregator
}

type SingleColLetRequest 

type SingleColLetRequest struct {
	CName string
	Oper  utils.LogicalAndArithmeticOperator
	Value *utils.DtypeEnclosure
}

type SortElement 

type SortElement struct {
	SortByAsc bool
	Op        string
	Field     string
}

type SortExpr 

type SortExpr struct {
	SortEles []*SortElement
	Limit    uint64

	SortAscending        []int
	SortRecords          map[string]map[string]interface{}
	NumProcessedSegments uint64
	// contains filtered or unexported fields
}

func (*SortExpr) AcquireProcessedSegmentsLock 

func (self *SortExpr) AcquireProcessedSegmentsLock()

func (*SortExpr) ReleaseProcessedSegmentsLock 

func (self *SortExpr) ReleaseProcessedSegmentsLock()

type SortRequest 

type SortRequest struct {
	ColName   string // column name to sort on
	Ascending bool   // if true, result is in ascending order. Else, result is in descending order
}

How to sort results

type SortValue 

type SortValue struct {
	Val         string
	InterpretAs string // Should be "ip", "num", "str", "auto", or ""
}

func (*SortValue) Compare 

func (self *SortValue) Compare(other *SortValue) (int, error)

type SpanLength 

type SpanLength struct {
	Num       int
	TimeScalr utils.TimeUnit
}

type SpanOptions 

type SpanOptions struct {
	DefaultSettings bool
	SpanLength      *SpanLength
}

type SplitByClause 

type SplitByClause struct {
	Field     string
	TcOptions *TcOptions
}

type StatisticExpr 

type StatisticExpr struct {
	StatisticFunctionMode StatisticFunctionMode
	Limit                 string
	StatisticOptions      *StatisticOptions
	FieldList             []string //Must have FieldList
	ByClause              []string
}

func (*StatisticExpr) GetGroupByCols 

func (self *StatisticExpr) GetGroupByCols() []string

func (*StatisticExpr) OverrideGroupByCol 

func (self *StatisticExpr) OverrideGroupByCol(bucketResult *BucketResult, resTotal uint64) error

func (*StatisticExpr) RemoveFieldsNotInExprForBucketRes 

func (self *StatisticExpr) RemoveFieldsNotInExprForBucketRes(bucketResult *BucketResult) error

Only display fields which in StatisticExpr

func (*StatisticExpr) SetCountToStatRes 

func (self *StatisticExpr) SetCountToStatRes(statRes map[string]utils.CValueEnclosure, elemCount uint64)

func (*StatisticExpr) SetPercToStatRes 

func (self *StatisticExpr) SetPercToStatRes(statRes map[string]utils.CValueEnclosure, elemCount uint64, resTotal uint64)

func (*StatisticExpr) SortBucketResult 

func (self *StatisticExpr) SortBucketResult(results *[]*BucketResult) error

type StatisticFunctionMode 

type StatisticFunctionMode uint8

type StatisticOptions 

type StatisticOptions struct {
	CountField   string
	OtherStr     string
	PercentField string
	ShowCount    bool
	ShowPerc     bool
	UseOther     bool
}

type StatsFuncChecker 

type StatsFuncChecker struct{}

func (StatsFuncChecker) IsUnsupported 

func (c StatsFuncChecker) IsUnsupported(funcName utils.AggregateFunctions) bool

type StatsOptions 

type StatsOptions struct {
	Delim          string
	Partitions     uint64
	DedupSplitvals bool
	Allnum         bool
}

type StringExpr 

type StringExpr struct {
	StringExprMode StringExprMode
	RawString      string      // only used when mode is RawString
	StringList     []string    // only used when mode is RawStringList
	FieldName      string      // only used when mode is Field
	ConcatExpr     *ConcatExpr // only used when mode is Concat
	TextExpr       *TextExpr   // only used when mode is TextExpr
}

func (*StringExpr) Evaluate 

func (self *StringExpr) Evaluate(fieldToValue map[string]utils.CValueEnclosure) (string, error)

func (*StringExpr) GetFields 

func (self *StringExpr) GetFields() []string

func (*StringExpr) GetNullFields 

func (self *StringExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

type StringExprMode 

type StringExprMode uint8

type StringStats 

type StringStats struct {
	StrSet map[string]struct{}
}

type TMLimitResult 

type TMLimitResult struct {
	ValIsInLimit     map[string]bool
	GroupValScoreMap map[string]*utils.CValueEnclosure
	Hll              *hyperloglog.Sketch
	StrSet           map[string]struct{}
	OtherCValArr     []*utils.CValueEnclosure
}

This structure is used to store values which are not within limit. And These values will be merged into the 'other' category.

type TS_TYPE 

type TS_TYPE uint8

type TableInfo 

type TableInfo struct {
	// contains filtered or unexported fields
}

Helper struct to keep track of raw and expanded tables

func InitTableInfo 

func InitTableInfo(rawRequest string, orgid uint64, es bool) *TableInfo

func (*TableInfo) GetKibanaIndices 

func (ti *TableInfo) GetKibanaIndices() []string

func (*TableInfo) GetNumIndices 

func (ti *TableInfo) GetNumIndices() int

func (*TableInfo) GetQueryTables 

func (ti *TableInfo) GetQueryTables() []string

func (*TableInfo) GetRawRequest 

func (ti *TableInfo) GetRawRequest() string

func (*TableInfo) String 

func (ti *TableInfo) String() string

type TagValueIndex 

type TagValueIndex struct {
	// contains filtered or unexported fields
}

type TagValueType 

type TagValueType string
const (
	StarValue   TagValueType = "*"
	ValueString TagValueType = "string"
)

Values for TagValueType

type TagsFilter 

type TagsFilter struct {
	TagKey          string
	RawTagValue     interface{} //change it to utils.DtypeEnclosure later
	HashTagValue    uint64
	TagOperator     utils.TagOperator
	LogicalOperator utils.LogicalOperator
	NotInitialGroup bool
}

Represents a single tag filter for a metric query

func (*TagsFilter) IsRegex 

func (tf *TagsFilter) IsRegex() bool

type TcOptions 

type TcOptions struct {
	BinOptions *BinOptions
	UseNull    bool
	UseOther   bool
	NullStr    string
	OtherStr   string
}

type TextExpr 

type TextExpr struct {
	IsTerminal  bool
	Op          string //lower, ltrim, rtrim
	Param       *StringExpr
	StrToRemove string
	Delimiter   *StringExpr
	ValueList   []*StringExpr
	StartIndex  *NumericExpr
	EndIndex    *NumericExpr
	LengthExpr  *NumericExpr
	Val         *ValueExpr
	Condition   *BoolExpr  // To filter out values that do not meet the criteria within a multivalue field
	InferTypes  bool       // To specify that the mv_to_json_array function should attempt to infer JSON data types when it converts field values into array elements.
	Cluster     *Cluster   // generates a cluster label
	SPathExpr   *SPathExpr // To extract information from the structured data formats XML and JSON.
}

func (*TextExpr) EvaluateText 

func (self *TextExpr) EvaluateText(fieldToValue map[string]utils.CValueEnclosure) (string, error)

func (*TextExpr) GetFields 

func (self *TextExpr) GetFields() []string

func (*TextExpr) GetNullFields 

func (self *TextExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

type TimeBucket 

type TimeBucket struct {
	IntervalMillis uint64 // size of each histogram bucket in millis
	StartTime      uint64 // start time of histogram
	EndTime        uint64 // end time of histogram
	AggName        string // name of aggregation
	Timechart      *TimechartExpr
}

type TimechartExpr 

type TimechartExpr struct {
	TcOptions  *TcOptions
	BinOptions *BinOptions
	SingleAgg  *SingleAgg
	ByField    string // group by this field inside each time range bucket (timechart)
	LimitExpr  *LimitExpr
}

type TransactionArguments 

type TransactionArguments struct {
	SortedRecordsSlice    []map[string]interface{}
	OpenTransactionsState map[string]*TransactionGroupState
	OpenTransactionEvents map[string][]map[string]interface{}
	Fields                []string
	StartsWith            *FilterStringExpr
	EndsWith              *FilterStringExpr
}

type TransactionGroupState 

type TransactionGroupState struct {
	Key       string
	Open      bool
	RecInden  string
	Timestamp uint64
}

type ValueBucket 

type ValueBucket struct {
	BucketKey     string // columnName for which to create buckets on
	MinBucketSize uint64 // min count per bucket
	Count         uint64 // limit number of resulting histograms
	CatchAll      bool
	ValueOrder    *ValueOrder // how to return filter buckets
}

type ValueExpr 

type ValueExpr struct {
	ValueExprMode ValueExprMode

	FloatValue    float64
	NumericExpr   *NumericExpr
	StringExpr    *StringExpr
	ConditionExpr *ConditionExpr
	BooleanExpr   *BoolExpr
}

See ValueExprMode type definition for which fields are valid for each mode.

func (*ValueExpr) EvaluateToFloat 

func (self *ValueExpr) EvaluateToFloat(fieldToValue map[string]utils.CValueEnclosure) (float64, error)

Try evaluating this ValueExpr to a float value, replacing each field in the expression with the value specified by fieldToValue. Each field listed by GetFields() must be in fieldToValue.

A ValueExpr can be evaluated to a string or float, so if this fails you may want to call ValueExpr.EvaluateToString().

func (*ValueExpr) EvaluateToString 

func (self *ValueExpr) EvaluateToString(fieldToValue map[string]utils.CValueEnclosure) (string, error)

Try evaluating this ValueExpr to a string value, replacing each field in the expression with the value specified by fieldToValue. Each field listed by GetFields() must be in fieldToValue.

A ValueExpr can be evaluated to a string or float, so if this fails you may want to call ValueExpr.EvaluateToFloat().

func (*ValueExpr) EvaluateValueExprAsString 

func (self *ValueExpr) EvaluateValueExprAsString(fieldToValue map[string]utils.CValueEnclosure) (string, error)

In this case, if we can not evaluate numeric expr to a float, we should evaluate it as a str

func (*ValueExpr) GetFields 

func (self *ValueExpr) GetFields() []string

func (*ValueExpr) GetNullFields 

func (self *ValueExpr) GetNullFields(fieldToValue map[string]utils.CValueEnclosure) ([]string, error)

type ValueExprMode 

type ValueExprMode uint8

type ValueOrder 

type ValueOrder struct {
	Ascending bool // should results be given in ascending order or descending. (true = ascending, false = descending)
}

type VectorMatchCardinality 

type VectorMatchCardinality int

VectorMatchCardinality describes the cardinality relationship of two Vectors in a binary operation. 

const (
	CardOneToOne VectorMatchCardinality = iota
	CardManyToOne
	CardOneToMany
	CardManyToMany
)

type VectorMatching 

type VectorMatching struct {
	// The cardinality of the two Vectors.
	Cardinality VectorMatchCardinality
	// MatchingLabels contains the labels which define equality of a pair of
	// elements from the Vectors.
	MatchingLabels []string
	// On includes the given label names from matching,
	// rather than excluding them.
	On bool
}

VectorMatching describes how elements from two Vectors in a binary operation are supposed to be matched.

type VtableCounts 

type VtableCounts struct {
	BytesCount       uint64
	RecordCount      uint64
	OnDiskBytesCount uint64
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.