Documentation
¶
Overview ¶
Builtin functions are a library of functions natively available in qlbridge expression evaluation although adding your own is easy.
Index ¶
- func FilterEval(ctx expr.EvalContext, vals []value.Value) (value.Value, bool)
- func FilterMatchEval(ctx expr.EvalContext, vals []value.Value) (value.Value, bool)
- func FiltersFromArgs(filterVals []value.Value) []string
- func HostEval(ctx expr.EvalContext, args []value.Value) (value.Value, bool)
- func HostsEval(ctx expr.EvalContext, args []value.Value) (value.Value, bool)
- func LoadAllBuiltins()
- func UrlWithQueryEval(include []*regexp.Regexp) expr.EvaluatorFunc
- type All
- type Any
- type ArrayIndex
- type ArraySlice
- type Avg
- type Cast
- type Contains
- type Count
- type DayOfWeek
- type Domain
- type Domains
- type Email
- type EmailDomain
- type EmailName
- type EncodeB64Decode
- type EncodeB64Encode
- type Eq
- type Exists
- type Filter
- type FilterMatch
- type Ge
- type Gt
- type HasPrefix
- type HasSuffix
- type HashMd5
- type HashSha1
- type HashSha256
- type HashSha512
- type HashSip
- type Host
- type Hosts
- type HourOfDay
- type HourOfWeek
- type Join
- type JsonPath
- type Le
- type Length
- type LowerCase
- type Lt
- type MapFunc
- type MapInvert
- type MapKeys
- type MapTime
- type MapValues
- type Match
- type Mm
- type Ne
- type Not
- type Now
- type OneOf
- type Pow
- type Qs
- type QsDeprecate
- type Replace
- type Split
- type Sqrt
- type StrFromTime
- type StringIndex
- type Strip
- type SubString
- type Sum
- type TimeSeconds
- type TimeTrunc
- type TitleCase
- type ToBool
- type ToDate
- type ToDateIn
- type ToInt
- type ToNumber
- type ToString
- type ToTimestamp
- type Unsign
- type UpperCase
- type UrlDecode
- type UrlMain
- type UrlMinusQs
- type UrlPath
- type UrlWithQuery
- type UserAgent
- type UserAgentMap
- type UuidGenerate
- type Yy
- type YyMm
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func FilterEval ¶
func FilterMatchEval ¶
func FiltersFromArgs ¶
FilterFromArgs given set of values
func LoadAllBuiltins ¶
func LoadAllBuiltins()
func UrlWithQueryEval ¶
func UrlWithQueryEval(include []*regexp.Regexp) expr.EvaluatorFunc
UrlWithQueryEval pass reg-expressions to match qs args. Must match one regexp or else the qs param is dropped.
Types ¶
type All ¶
type All struct{}
All Answers True/False if all of the arguments evaluate to truish (javascripty) type definintion of true. Non-Nil, non-Error, values.
Rules for if True:
int != 0 string != "" boolean natively supported true/false time != time.IsZero()
Examples:
all("hello",2, true) => true
all("hello",0,true) => false
all("",2, true) => false
type Any ¶
type Any struct{}
Any Answers True/False if any of the arguments evaluate to truish (javascripty) type definintion of true
Rules for if True:
int != 0 string != "" boolean natively supported true/false time != time.IsZero()
Examples:
any(item,item2) => true, true any(not_field) => false, true
type ArrayIndex ¶
type ArrayIndex struct{}
ArrayIndex array.index choose the nth element of an array
// given context input of "items" = [1,2,3] array.index(items, 1) => 1, true array.index(items, 5) => nil, false array.index(items, -1) => 3, true
func (*ArrayIndex) Type ¶
func (m *ArrayIndex) Type() value.ValueType
Type unknown - returns single value from SliceValue array
func (*ArrayIndex) Validate ¶
func (m *ArrayIndex) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type ArraySlice ¶
type ArraySlice struct{}
array.slice slice element m -> n of a slice. First arg must be a slice.
// given context of "items" = [1,2,3,4,5] array.slice(items, 1, 3) => [2,3], true array.slice(items, 2) => [3,4,5], true array.slice(items, -2) => [4,5], true
func (*ArraySlice) Validate ¶
func (m *ArraySlice) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
Validate must be at least 2 args, max of 3
type Avg ¶
type Avg struct{}
Avg average of values. Note, this function DOES NOT persist state doesn't aggregate across multiple calls. That would be responsibility of write context.
avg(1,2,3) => 2.0, true
avg("hello") => math.NaN, false
type Cast ¶
type Cast struct{}
Cast type coercion, cast to an explicit type.
cast(identity AS <type>) => 5.0 cast(reg_date AS string) => "2014/01/12"
Types: [char, string, int, float]
type Contains ¶
type Contains struct{}
Contains does first arg string contain 2nd arg?
contains("alabama","red") => false
type Count ¶
type Count struct{}
Count Return int value 1 if non-nil/zero. This should be renamed Increment and in general is a horrible, horrible function that needs to be replaced with occurrences of value, ignores the value and ensures it is non null
count(anyvalue) => 1, true count(not_number) => -- 0, false
type DayOfWeek ¶
type DayOfWeek struct{}
DayOfWeek day of week [0-6]
dayofweek() => 3, true
dayofweek("2016/07/04") => 5, true
type Domain ¶
type Domain struct{}
Extract Domain from a Value, or Values (must be urlish), doesn't do much/any validation. if input is a list of strings, only first is evaluated, for plural see domains()
domain("http://www.lytics.io/index.html") => "lytics.io"
type Domains ¶
type Domains struct{}
Domains Extract Domains from a Value, or Values (must be urlish), doesn't do much/any validation
domains("http://www.lytics.io/index.html") => []string{"lytics.io"}
type Email ¶
type Email struct{}
email a string, parses email and makes sure it is valid
email("Bob <bob@bob.com>") => bob@bob.com, true
email("Bob <bob>") => "", false
type EmailDomain ¶
type EmailDomain struct{}
emaildomain parses email and returns domain
emaildomain("Bob <bob@bob.com>") => bob.com
func (*EmailDomain) Validate ¶
func (m *EmailDomain) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type EmailName ¶
type EmailName struct{}
emailname a string, parses email
emailname("Bob <bob@bob.com>") => Bob
type EncodeB64Decode ¶
type EncodeB64Decode struct{}
Base 64 encoding function
encoding.b64decode("aGVsbG8gd29ybGQ=") => "hello world"
func (*EncodeB64Decode) Validate ¶
func (m *EncodeB64Decode) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type EncodeB64Encode ¶
type EncodeB64Encode struct{}
Base 64 encoding function
encoding.b64encode("hello world=") => aGVsbG8gd29ybGQ=
func (*EncodeB64Encode) Validate ¶
func (m *EncodeB64Encode) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type Eq ¶
type Eq struct{}
Equal function? returns true if items are equal
// given context {"name":"wil","event":"stuff", "int4": 4}
eq(int4,5) => false
type Exists ¶
type Exists struct{}
Exists Answers True/False if the field exists and is non null
exists(real_field) => true
exists("value") => true
exists("") => false
exists(empty_field) => false
exists(2) => true
exists(todate(date_field)) => true
type Filter ¶
type Filter struct{}
Filter Filter OUT Values that match specified list of match filter criteria
Operates on MapValue (map[string]interface{}), StringsValue ([]string), or string takes N Filter Criteria supports Matching: "filter**" // matches "filter_x", "filterstuff"
Filter a map of values by key to remove certain keys
filter(match("topic_"),key_to_filter, key2_to_filter) => {"goodkey": 22}, true
Filter out VALUES (not keys) from a list of []string{} for a specific value
filter(split("apples,oranges",","),"ora*") => ["apples"], true
Filter out values for single strings
filter("apples","app*") => []string{}, true
type FilterMatch ¶
type FilterMatch struct{}
FilterMatch Filter IN Values that match specified list of match filter criteria
Operates on MapValue (map[string]interface{}), StringsValue ([]string), or string takes N Filter Criteria
Wildcard Matching: "abcd*" // matches "abcd_x", "abcdstuff"
Filter a map of values by key to only keep certain keys
filtermatch(match("topic_"),key_to_filter, key2_to_filter) => {"goodkey": 22}, true
Filter in VALUES (not keys) from a list of []string{} for a specific value
filtermatch(split("apples,oranges",","),"ora*") => ["oranges"], true
Filter in values for single strings
filtermatch("apples","app*") => []string{"apple"}, true
func (*FilterMatch) Validate ¶
func (m *FilterMatch) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type Gt ¶
type Gt struct{}
Gt GreaterThan is left hand > right hand. Must be able to convert items to Floats.
gt(5,6) => true, true
type HasPrefix ¶
type HasPrefix struct{}
HasPrefix string evaluation to see if string begins with
hasprefix("apples","ap") => true
hasprefix("apples","o") => false
type HasSuffix ¶
type HasSuffix struct{}
HasSuffix string evaluation to see if string ends with
hassuffix("apples","es") => true
hassuffix("apples","e") => false
type HashMd5 ¶
type HashMd5 struct{}
HashMd5Func Hash a value to MD5 string
hash.md5("/blog/index.html") => abc345xyz
type HashSha1 ¶
type HashSha1 struct{}
HashSha1Func Hash a value to SHA256 string
hash.sha1("/blog/index.html") => abc345xyz
type HashSha256 ¶
type HashSha256 struct{}
HashSha256Func Hash a value to SHA256 string
hash.sha256("/blog/index.html") => abc345xyz
func (*HashSha256) Validate ¶
func (m *HashSha256) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type HashSha512 ¶
type HashSha512 struct{}
HashSha512Func Hash a value to SHA512 string
hash.sha512("/blog/index.html") => abc345xyz
func (*HashSha512) Validate ¶
func (m *HashSha512) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type HashSip ¶
type HashSip struct{}
hash.sip() hash a value to a 64 bit int
hash.sip("/blog/index.html") => 1234
type Host ¶
type Host struct{}
Extract host from a String (must be urlish), doesn't do much/any validation In the event the value contains more than one input url, will ONLY evaluate first
host("http://www.lytics.io/index.html") => www.lytics.io
type Hosts ¶
type Hosts struct{}
Extract hosts from a Strings (must be urlish), doesn't do much/any validation
hosts("http://www.lytics.io", "http://www.activate.lytics.io") => www.lytics.io, www.activate.lytics.io
type HourOfDay ¶
type HourOfDay struct{}
hourofday hour of day [0-23]
hourofday(field) hourofday() // Uses message time
type HourOfWeek ¶
type HourOfWeek struct{}
hour of week [0-167]
func (*HourOfWeek) Validate ¶
func (m *HourOfWeek) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type Join ¶
type Join struct{}
Join items together (string concatenation)
join("apples","oranges",",") => "apples,oranges"
join(["apples","oranges"],",") => "apples,oranges"
join("apples","oranges","") => "applesoranges"
type JsonPath ¶
type JsonPath struct{}
JsonPath jmespath json parser http://jmespath.org/
json_field = `[{"name":"n1","ct":8,"b":true, "tags":["a","b"]},{"name":"n2","ct":10,"b": false, "tags":["a","b"]}]`
json.jmespath(json_field, "[?name == 'n1'].name | [0]") => "n1"
type Length ¶
type Length struct{}
len length of array types
len([1,2,3]) => 3, true len(not_a_field) => -- NilInt, false
type LowerCase ¶
type LowerCase struct{}
LowerCase take a string and lowercase it. must be able to convert to string.
string.lowercase("HELLO") => "hello", true
type MapFunc ¶
type MapFunc struct{}
Map Create a map from two values. If the right side value is nil then does not evaluate.
map(left, right) => map[string]value{left:right}
type MapInvert ¶
type MapInvert struct{}
MapInvert: Take a map and invert key/values
// given input:
tags = {"1":"news","2":"sports"}
mapinvert(tags) => map[string]string{"news":"1","sports":"2"}
type MapKeys ¶
type MapKeys struct{}
MapKeys: Take a map and extract array of keys
//given input:
{"tag.1":"news","tag.2":"sports"}
mapkeys(match("tag.")) => []string{"news","sports"}
type MapTime ¶
type MapTime struct{}
MapTime() Create a map[string]time of each key
maptime(field) => map[string]time{field_value:message_timestamp}
maptime(field, timestamp) => map[string]time{field_value:timestamp}
type MapValues ¶
type MapValues struct{}
MapValues: Take a map and extract array of values
// given input:
{"tag.1":"news","tag.2":"sports"}
mapvalue(match("tag.")) => []string{"1","2"}
type Match ¶
type Match struct{}
Match a simple pattern match on KEYS (not values) and build a map of all matched values. Matched portion is replaced with empty string. - May pass as many match strings as you want. - Must match on Prefix of key.
given input context of:
{"score_value":24,"event_click":true, "tag_apple": "apple", "label_orange": "orange"}
match("score_") => {"value":24}
match("amount_") => false
match("event_") => {"click":true}
match("label_","tag_") => {"apple":"apple","orange":"orange"}
type Mm ¶
type Mm struct{}
mm Get month as integer from date
@optional timestamp (if not, gets from context reader)
mm() => 01, true /// assuming message ts = jan 1
mm("2014-03-17") => 03, true
type Ne ¶
type Ne struct{}
Ne Not Equal function? returns true if items are equal
// given {"5s":"5","item4":4,"item4s":"4"}
ne(`5s`,5) => true, true
ne(`not_a_field`,5) => false, true
ne(`item4s`,5) => false, true
ne(`item4`,5) => false, true
type Not ¶
type Not struct{}
Not urnary negation function
not(eq(5,5)) => false, true
not(eq("false")) => false, true
type Now ¶
type Now struct{}
Now Get current time of Message (message time stamp) or else choose current server time if none is available in message context
type OneOf ¶
type OneOf struct{}
OneOf choose the first non-nil, non-zero, non-false fields
oneof(nil, 0, "hello") => 'hello'
type Pow ¶
type Pow struct{}
Pow exponents, raise x to the power of y
pow(5,2) => 25, true pow(3,2) => 9, true pow(not_number,2) => NilNumber, false
type Qs ¶
type Qs struct{}
Qs Extract qs param from a string (must be url valid)
qs("http://www.lytics.io/?utm_source=google","utm_source") => "google", true
type QsDeprecate ¶
type QsDeprecate struct{}
Qs Extract qs param from a string (must be url valid)
qs("http://www.lytics.io/?utm_source=google","utm_source") => "google", true
func (*QsDeprecate) Validate ¶
func (m *QsDeprecate) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type Replace ¶
type Replace struct{}
Replace a string(s). Replace occurences of 2nd arg In first with 3rd. 3rd arg "what to replace with" is optional
replace("/blog/index.html", "/blog","") => /index.html
replace("/blog/index.html", "/blog") => /index.html
replace("/blog/index.html", "/blog/archive/","/blog") => /blog/index.html
replace(item, "M")
type Split ¶
type Split struct{}
Split a string with given separator
split("apples,oranges", ",") => []string{"apples","oranges"}
type Sqrt ¶
type Sqrt struct{}
Sqrt square root function. Must be able to coerce to number.
sqrt(4) => 2, true sqrt(9) => 3, true sqrt(not_number) => 0, false
type StrFromTime ¶
type StrFromTime struct{}
StrFromTime extraces certain parts from a time, similar to Python's StrfTime See http://strftime.org/ for Strftime directives.
strftime("2015/07/04", "%B") => "July"
strftime("2015/07/04", "%B:%d") => "July:4"
strftime("1257894000", "%p") => "PM"
func (*StrFromTime) Validate ¶
func (m *StrFromTime) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type StringIndex ¶
type StringIndex struct{}
StringIndex a string, removing leading/trailing whitespace
string.index("apples, oranges ", ",") => 6
string.index("apples, oranges ", "X") => -1, false
func (*StringIndex) Type ¶
func (m *StringIndex) Type() value.ValueType
func (*StringIndex) Validate ¶
func (m *StringIndex) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type Strip ¶
type Strip struct{}
Strip a string, removing leading/trailing whitespace
strip(split("apples, oranges ",",")) => {"apples", "oranges"}
strip("apples ") => "apples"
type SubString ¶
type SubString struct{}
SubString from a given string, use integers to describe the start, [stop] of substring to extract.
string.substr("apples, oranges ", 0, 3) => "app", true
string.substr("apple", 3) => "le", true
string.substr("apple", 30, 500) => nil, false
type Sum ¶
type Sum struct{}
Sum function to add values. Note, this function DOES NOT persist state doesn't aggregate across multiple calls. That would be responsibility of write context.
sum(1, 2, 3) => 6 sum(1, "horse", 3) => nan, false
type TimeSeconds ¶
type TimeSeconds struct{}
TimeSeconds time in Seconds, parses a variety of formats looking for seconds See github.com/araddon/dateparse for formats supported on date parsing
seconds("M10:30") => 630
seconds("M100:30") => 6030
seconds("00:30") => 30
seconds("30") => 30
seconds(30) => 30
seconds("2015/07/04") => 1435968000
func (*TimeSeconds) Validate ¶
func (m *TimeSeconds) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type TimeTrunc ¶
type TimeTrunc struct {
// contains filtered or unexported fields
}
UnixDateTruncFunc converts a value.Value to a unix timestamp string. This is used for the BigQuery export since value.TimeValue returns a unix timestamp with milliseconds (ie "1438445529707") by default. This gets displayed in BigQuery as "47547-01-24 10:49:05 UTC", because they expect seconds instead of milliseconds. This function "truncates" the unix timestamp to seconds to the form "1438445529.707" i.e the milliseconds are placed after the decimal place. Inspired by the "DATE_TRUNC" function used in PostgresQL and RedShift: http://www.postgresql.org/docs/8.1/static/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC
unixtrunc("1438445529707") --> "1438445529"
unixtrunc("1438445529707", "seconds") --> "1438445529.707"
unixtrunc("1438445529707123456", "secondsmicro") --> "1438445529.707123"
type TitleCase ¶
type TitleCase struct{}
TitleCase take a string and uppercase it. must be able to convert to string.
string.uppercase("hello") => "HELLO", true
type ToDate ¶
type ToDate struct{}
todate: convert to Date
// uses lytics/datemath
todate("now-3m")
// uses araddon/dateparse util to recognize formats
todate(field)
// first parameter is the layout/format
todate("01/02/2006", field )
type ToDateIn ¶
type ToDateIn struct{}
todatein: convert to Date with timezon
// uses lytics/datemath
todate("now-3m", "America/Los_Angeles")
// uses araddon/dateparse util to recognize formats
todate(field, "America/Los_Angeles")
type ToInt ¶
type ToInt struct{}
ToInt Convert to Integer: Best attempt at converting to integer.
toint("5") => 5, true
toint("5.75") => 5, true
toint("5,555") => 5555, true
toint("$5") => 5, true
toint("5,555.00") => 5555, true
type ToNumber ¶
type ToNumber struct{}
ToNumber Convert to Number: Best attempt at converting to integer
tonumber("5") => 5.0
tonumber("5.75") => 5.75
tonumber("5,555") => 5555
tonumber("$5") => 5.00
tonumber("5,555.00") => 5555
type ToTimestamp ¶
type ToTimestamp struct{}
totimestamp: convert to date, then to unix Seconds
totimestamp() => int, true
totimestamp("Apr 7, 2014 4:58:55 PM") => 1396889935
func (*ToTimestamp) Validate ¶
func (m *ToTimestamp) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type Unsign ¶
type Unsign struct{}
Unsign converts a signed int to an unsigned int represented as a string for positive numbers this should simply convert the number to a string for negative numbers, converting to a uint adds the negative value to the max uint value (18446744073709551615).
unsign(-32847623329847) => 18446711226086221769
unsign(876) => 876
unsign("-70") => 18446744073709551546
type UpperCase ¶
type UpperCase struct{}
UpperCase take a string and uppercase it. must be able to convert to string.
string.uppercase("hello") => "HELLO", true
type UrlDecode ¶
type UrlDecode struct{}
url decode a string
urldecode("http://www.lytics.io/index.html") => http://www.lytics.io
In the event the value contains more than one input url, will ONLY evaluate first
type UrlMain ¶
type UrlMain struct{}
UrlMain remove the querystring and scheme from url
urlmain("http://www.lytics.io/?utm_source=google") => "www.lytics.io/", true
type UrlMinusQs ¶
type UrlMinusQs struct{}
UrlMinusQs removes a specific query parameter and its value from a url
urlminusqs("http://www.lytics.io/?q1=google&q2=123", "q1") => "http://www.lytics.io/?q2=123", true
func (*UrlMinusQs) Validate ¶
func (m *UrlMinusQs) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type UrlPath ¶
type UrlPath struct{}
UrlPath Extract url path from a String (must be urlish), doesn't do much/any validation
path("http://www.lytics.io/blog/index.html") => blog/index.html
In the event the value contains more than one input url, will ONLY evaluate first
type UrlWithQuery ¶
type UrlWithQuery struct{}
UrlWithQueryFunc strips a url and retains only url parameters that match the supplied regular expressions.
url.matchqs(url, re1, re2, ...) => url_withoutqs
func (*UrlWithQuery) Validate ¶
func (*UrlWithQuery) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type UserAgent ¶
type UserAgent struct{}
UserAgent Extract user agent features
useragent(user_agent_field,"mobile") => "true", true
type UserAgentMap ¶
type UserAgentMap struct{}
UserAgentMap Extract user agent features
useragent.map(user_agent_field) => {"mobile": "false","platform":"X11"}, true
func (*UserAgentMap) Validate ¶
func (m *UserAgentMap) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type UuidGenerate ¶
type UuidGenerate struct{}
uuid generates a new uuid
uuid() => "...."
func (*UuidGenerate) Validate ¶
func (m *UuidGenerate) Validate(n *expr.FuncNode) (expr.EvaluatorFunc, error)
type Yy ¶
type Yy struct{}
Yy Get year in integer from field, must be able to convert to date
yy() => 15, true // assuming it is 2015
yy("2014-03-01") => 14, true
Source Files