dsl

package
v0.0.0-...-724e02d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 23, 2017 License: Apache-2.0 Imports: 13 Imported by: 3

Documentation

Overview

Cicerone - a lager connoisseur

lager emits logs chug pretty prints logs in real time cicerone sifts through those logs and analyzes them

The dsl package has a number of different nouns:

- Entry: Cicerone's representation of a log line - Entries: an ordered list of entries, can be filtered and grouped - GroupedEntries: a collection of Entries grouped by an arbitrary key - EntryPair: a set of two entries typically used to represent a period of time between two events of interest - EntryPairs: a collection of EntryPair. From this one can generate:

  • DTStats: a set of statistics describing a collection of EntryPairs
  • Durations: an array of time.Durations with some convenience helpers

- TimelineDescription: a collection of TimelinePoints used to construct a timeline - Timeline: combines a TimelineDescription with an Entries -- represents the timeline associated with a particular object flowing through the logs - Timelines: a pile of logs will have several timelines in them. These are collected into a Timelines object. - Matchers: matchers take an Entry and return a boolean - Getters: getters take an Entry and pull data out of it

Using these nouns and their attendant verbs one can use Cicerone to quickly slice and dice a collection of log lines. The resulting data can then be visualized with the viz package.

Index

Constants

This section is empty.

Variables

View Source 
var GetIndex = GetterFunc(func(entry Entry) (interface{}, bool) {
	return entry.Index, true
})

GetJob returns the Job associated with an entry

View Source 
var GetJob = GetterFunc(func(entry Entry) (interface{}, bool) {
	return entry.Job, true
})

GetJob returns the Job associated with an entry

View Source 
var GetLogLevel = GetterFunc(func(entry Entry) (interface{}, bool) {
	return entry.LogLevel, true
})

GetLogLevel returns the LogLevel associated with an entry

View Source 
var GetMessage = GetterFunc(func(entry Entry) (interface{}, bool) {
	return entry.Message, true
})

GetMessage returns the message associated with an entry

View Source 
var GetSession = GetterFunc(func(entry Entry) (interface{}, bool) {
	return entry.Session, true
})

GetSession returns the session associated with an entry

View Source 
var GetSource = GetterFunc(func(entry Entry) (interface{}, bool) {
	return entry.Source, true
})

GetSource returns the source (typically process-name) associated with an entry

View Source 
var GetVM = GetterFunc(func(entry Entry) (interface{}, bool) {
	return entry.VM(), true
})

GetVM returns the VM associated with an entry

View Source 
var NoOpTransformation = func(d interface{}) (interface{}, bool) {
	return d, true
}
View Source 
var TrimTransformation = func(d interface{}) (interface{}, bool) {
	bs, err := json.Marshal(d)
	if err != nil {
		return nil, false
	}
	return strings.Trim(string(bs), `"`), true
}

Functions

This section is empty.

Types

type DTStats 

type DTStats struct {
	Name      string
	Min       time.Duration
	Max       time.Duration
	Mean      time.Duration
	N         int
	MinWinner EntryPair
	MaxWinner EntryPair
}

DTStats bndles up statistics extracted from a collection of EntryPairs

func (DTStats) String 

func (d DTStats) String() string

Printing out a DTStats is useful - it will emit the Annotation associated with the most extreme EntryPair outliers in the sample

type DTStatsSlice 

type DTStatsSlice []DTStats

DTStatsSLice is a collection of DTStats

func (DTStatsSlice) String 

func (d DTStatsSlice) String() string

String() joins the Strings() of the underlying DTStats

type Durations 

type Durations []time.Duration

Duration wraps []time.Duration and adds a number of convenience methods

func (Durations) CountInRange 

func (d Durations) CountInRange(min time.Duration, max time.Duration) int

CountInRange returns the number of durations between min and max It's used to construct histograms

func (Durations) Max 

func (d Durations) Max() time.Duration

Max returns the largest duration in the list

func (Durations) Min 

func (d Durations) Min() time.Duration

Min returns the smallest duration in the list

type Entries 

type Entries []Entry

Entries is a list of invidiual Entry(ies)

func (Entries) ConstructTimeline 

func (e Entries) ConstructTimeline(description TimelineDescription, zeroEntry Entry) Timeline

ConstructTimeline takes a TimelineDescription and a Zeroth entry and returns a Timeline The Zeroth entry is used to compute the starting time the Timeline

func (Entries) Filter 

func (e Entries) Filter(matcher Matcher) Entries

Filter returns the list of Entries that match the passed-in Matcher

func (Entries) First 

func (e Entries) First(matcher Matcher) (Entry, bool)

First returns the *first* Entry that satisfies the passed in Matcher. The second return value tells the caller if an entry was found or not

func (Entries) GroupBy 

func (e Entries) GroupBy(getter Getter) *GroupedEntries

GroupBy groups all Entries by the passed in Getter it returns a GroupedEntries object The values returned by the Getter correpond to the Keys in the returned GroupedEntries object

func (Entries) Len 

func (e Entries) Len() int

func (Entries) Less 

func (e Entries) Less(i, j int) bool

func (Entries) Swap 

func (e Entries) Swap(i, j int)

func (Entries) WriteLagerFormatTo 

func (e Entries) WriteLagerFormatTo(w io.Writer) error

WriteLagerFormatTo emits lager-formatted entries to the passed in writer

type Entry 

type Entry struct {
	chug.LogEntry
	Job   string
	Index int
}

An Entry represtents a Cicerone log line

func NewEntryFromChugLog 

func NewEntryFromChugLog(chugEntry chug.Entry) (Entry, error)

func (Entry) IsZero 

func (e Entry) IsZero() bool

IsZero returns true if the Entry is the zero Entry

func (Entry) LagerFormat 

func (e Entry) LagerFormat() lager.LogFormat

func (Entry) VM 

func (e Entry) VM() string

VM returns a unique idenfitier of the machine that emitted the log line

This corresponds to "job/index"

func (Entry) WriteLagerFormatTo 

func (e Entry) WriteLagerFormatTo(w io.Writer) error

WriteLagerFormatTo emits lager formatted output to the passed-in writer

type EntryPair 

type EntryPair struct {
	FirstEntry  Entry
	SecondEntry Entry
	Annotation  interface{}
}

EntryPair represents a span in time between two Entries It includes an arbitrary annotation

func (EntryPair) DT 

func (e EntryPair) DT() time.Duration

DT returns the time.Duration between the two events in the EntryPair

func (EntryPair) String 

func (e EntryPair) String() string

type EntryPairs 

type EntryPairs []EntryPair

EntryPairs is a slice of EntryPair

func (EntryPairs) DTStats 

func (e EntryPairs) DTStats() DTStats

DTStats returns a DTStats rollup summarizing the distribution of time intervals in the slice of EntryPairs

func (EntryPairs) Durations 

func (e EntryPairs) Durations() Durations

Durations returns a Durations slice of time.Duration composed - it's a collection of the time intervals in the slice of EntryPairs

func (EntryPairs) FilterByDurationGreaterThan 

func (e EntryPairs) FilterByDurationGreaterThan(cutoff time.Duration) EntryPairs

FilterByDurationGreaterThan returns a copy of the EntryPairs containing entries that exceed the passed in duration

func (EntryPairs) FilterByDurationLessThan 

func (e EntryPairs) FilterByDurationLessThan(cutoff time.Duration) EntryPairs

FilterByDurationGreaterThan returns a copy of the EntryPairs containing entries that exceed the passed in duration

func (EntryPairs) String 

func (e EntryPairs) String() string

type Getter 

type Getter interface {
	Get(entry Entry) (interface{}, bool)
}

Getter objects can return arbitrary data from a passed-in-Entry

func DataGetter 

func DataGetter(keys ...string) Getter

DataGetter returns a Getter that can extract data from an Entry's Data field DataGetter takes multiple keys. These are tried in order -- if a key is found in the Data field, the corresponding value is returned. A key can be a full-blown JSON path (e.g. `foo.bar.baz`) -- DataGetter will traverse the Data field as far as possible to fetch the corresponding value.

These behaviors combine well wtih entries.GroupBy. In particular, we are often inconsistent with how we name keys in our lager.Data -- sometimes this is intentional as a message passes from one layer of abstraction to another. For example, TaskGuid becomes Guid becomes Container.Handle as it flows from BBS=>Rep=>Executor=>Garden.

To group by TaskGuid one can 

entries.GroupBy(DataGetter("TaskGuid", "Guid", "Container.Handle"))

func TransformingGetter 

func TransformingGetter(transformations TransformationMap) Getter

type GetterFunc 

type GetterFunc func(Entry) (interface{}, bool)

GetterFunc makes it easy to create Getters from bare functions

func (GetterFunc) Get 

func (g GetterFunc) Get(entry Entry) (interface{}, bool)

Get satisfies the Getter interface

type GroupedEntries 

type GroupedEntries struct {
	Keys    []interface{}
	Entries []Entries
	// contains filtered or unexported fields
}

GroupedEntries represent an ordered collection of Grouped Entries.

groupedEntries.Keys is a slice of Keys (the value returned by the Getter passed to entries.GroupBy) groupedEntries.Entries is a slice of Entries for each Key

GroupedEntries uses parallel arrays to retain the ordering in which Keys are detected - the first Key in the GroupedEntries is the first Key that was detected by entried.GroupBy

func NewGroupedEntries 

func NewGroupedEntries() *GroupedEntries

NewGroupedEntries initializes a new GroupedEntries

func (*GroupedEntries) Append 

func (g *GroupedEntries) Append(key interface{}, entry Entry)

Append adds an entry to the given Key

func (*GroupedEntries) AppendEntries 

func (g *GroupedEntries) AppendEntries(key interface{}, entries Entries)

Append entries appends a slice of Entries to the given Key

func (*GroupedEntries) ConstructTimelines 

func (g *GroupedEntries) ConstructTimelines(description TimelineDescription) (Timelines, error)

ConstructTimelines creates a slice of Timelines by calling entries.ConstructTimeline on each Entries element in the group. The Key associated with the Entries element becomes the Annotation associated with the Timeline.

Note that Timelines aren't Key=>Timeline mappings. Instead GroupedEntries returns a *flat list* of Timelines with the Key parameter associated with the individual Timeline.

func (*GroupedEntries) EachGroup 

func (g *GroupedEntries) EachGroup(f func(interface{}, Entries) error) error

EachGroup is an iterator (think functional thoughts) that loops over all Keys and Entries in order 

	groupedEntries.EachGroup(func(key interface{}, entries Entries) error {
 	fmt.Printf("%s: %s\n", key, entries)
		return nil
	})

will print all entries in the group. Returning non-nil will cause the iterator to abort.

func (*GroupedEntries) Filter 

func (g *GroupedEntries) Filter(matcher Matcher) *GroupedEntries

Filter applies `entries.Filter` to each Entries element in the group. Entries that end up empty are removed from the group.

As a consequence: 

entries.Filter(matcher).GroupBy(getter)

and 

entries.GroupBy(getter).Filter(matcher)

are identical.

func (*GroupedEntries) Lookup 

func (g *GroupedEntries) Lookup(key interface{}) (Entries, bool)

Look up entries for a given key

func (*GroupedEntries) WriteLagerFormatTo 

func (g *GroupedEntries) WriteLagerFormatTo(w io.Writer) error

WriteLagerFormat emits lager formatted output for all Entries in the group.

type GroupedTimelines 

type GroupedTimelines struct {
	Keys      []interface{}
	Timelines []Timelines
	// contains filtered or unexported fields
}

GroupedTimelines represent an ordered collection of Grouped Timelines

func NewGroupedTimelines 

func NewGroupedTimelines() *GroupedTimelines

NewGroupedTimelines initializes a new GroupedTimelines

func (*GroupedTimelines) Append 

func (g *GroupedTimelines) Append(key interface{}, timeline Timeline)

Append adds an timeline to the given Key

func (*GroupedTimelines) AppendTimelines 

func (g *GroupedTimelines) AppendTimelines(key interface{}, timelines Timelines)

Append timelines appends a slice of Timeline to the given Key

func (*GroupedTimelines) Description 

func (g *GroupedTimelines) Description() TimelineDescription

Description returns the TimelineDescription associated with the Timelines in the group

func (*GroupedTimelines) EachGroup 

func (g *GroupedTimelines) EachGroup(f func(interface{}, Timelines) error) error

EachGroup is an iterator (think functional thoughts) that loops over all Keys and Timelines in order 

groupedTimelines.EachGroup(func(key interface{}, timelines Timelines) error {
    fmt.Printf("%s: %s\n", key, timelines)
    return nil
})

will print all timelines in the group. Returning non-nil will cause the iterator to abort.

func (*GroupedTimelines) Lookup 

func (g *GroupedTimelines) Lookup(key interface{}) (Timelines, bool)

Look up timelines for a given key

type Matcher 

type Matcher interface {
	Match(entry Entry) bool
}

Matcher objects can test an Entry, returning true/false

func And 

func And(matchers ...Matcher) Matcher

And combines Matchers - the output is the logical && of the output of each component Matcher

func False 

func False() Matcher

False is a trivial Matcher that always returns False

func MatchAfter 

func MatchAfter(t time.Time) Matcher

MatchAfter returns true if the Entry's timestamp is after the passed-in time

func MatchBefore 

func MatchBefore(t time.Time) Matcher

MatchBefore returns true if the Entry's timestamp is before the passed-in time

func MatchBetween 

func MatchBetween(after, before time.Time) Matcher

MatchBefore returns true if the Entry's timestamp is within the interval supplied

func MatchIndex 

func MatchIndex(index int) Matcher

MatchIndex matches true if the Entry's Index matches the passed-in integer

func MatchJob 

func MatchJob(job string) Matcher

MatchJob matches true if the Entry's Job matches the passed-in string (interpreted as a regular expression)

func MatchLogLevel 

func MatchLogLevel(logLevel lager.LogLevel) Matcher

MatchLogLEvel matches true if the Entry's LogLevel matches the passed-in lager.LogLevel

func MatchMessage 

func MatchMessage(message string) Matcher

MatchMessage matches true if the Entry's Message matches the passed-in string (interpreted as a regular expression)

func MatchSession 

func MatchSession(session string) Matcher

MatchSession matches true if the Entry's Session matches the passed-in string (interpreted as a regular expression)

func MatchSource 

func MatchSource(source string) Matcher

MatchSource matches true if the Entry's Source matches the passed-in string (interpreted as a regular expression)

func MatchVM 

func MatchVM(vm string) Matcher

MatchVM matches true if the Entry's VM matches the passed-in string (interpreted as a regular expression)

func Not 

func Not(matcher Matcher) Matcher

Not negates the passed-in Matcher

func Or 

func Or(matchers ...Matcher) Matcher

Or combines Matchers - the output is the logical || of the output of each component Matcher

func RegExpMatcher 

func RegExpMatcher(getter Getter, regExp string) Matcher

RegExpMatcher takes a Getter (presumed to return a string) and a regular expression (encoded as a string) RegExpMatcher returns true of the string returned by the Getter matches the passed-in regular expression.

func True 

func True() Matcher

True is a trivial Matcher that always returns True

type MatcherFunc 

type MatcherFunc func(Entry) bool

MatcherFunc makes it easy to create Matchers from bare functions

func (MatcherFunc) Match 

func (m MatcherFunc) Match(entry Entry) bool

Match satisifes the Matcher interface

type Timeline 

type Timeline struct {
	Annotation  interface{}
	Description TimelineDescription
	ZeroEntry   Entry
	Entries     Entries
}

Timeline encapsulates a slice of Entries and a corresponding TimelineDescription A Timeline can have an optional - arbitrary - Annotation A Timeline also has a ZeroEntry. This is used as a zero point to compute the time at which the first entry occurs. The Timeline is effectively comprised of two parallel slices: the TimelineDescription and the slice of Entries.

func (Timeline) BeginsAt 

func (t Timeline) BeginsAt() time.Time

BeginsAt returns the timestamp at which the first non-zero entry in the Timeline occurs Note: BeginsAt does not include the ZeroEntry.

func (Timeline) EndsAt 

func (t Timeline) EndsAt() time.Time

EndsAt returns the timestamp at which the last non-zero entry in the Timeline occurs

func (Timeline) EntryPair 

func (t Timeline) EntryPair(index int) (EntryPair, bool)

EntryPair returns the EntryPair at the passed-in index of the Timeline.

EntryPair is strict: only timeline points where both the preceding entry and requested entry are non-zero are returned.

As a concrete example, consider a Timeline constructed with a TimelineDescription of: 

description := {
	{"Object-Created", CreateMatcher},
	{"Object-Run", RunMatcher},
	{"Object-Destroyed", DestroyMatcher}
}

Then 

timeline.EntryPair(1)

Will return an EntryPair where the FirstEntry corresponds to the Object-Created Entry and the SecondEntry corresponds to the Object-Run event.

The annotation associated with the resulting EntryPair is set to the Annotation associated with this Timeline.

Finally, note that 

timeline.EntryPair(0)

returns the ZeroEntry as the FirstEntry in the pair.

func (Timeline) First 

func (t Timeline) First(matcher Matcher) (Entry, bool)

First returns the first entry in the timeline that matches the passed-in matcher

func (Timeline) IsComplete 

func (t Timeline) IsComplete() bool

IsComplete returns true if all events in the timeline are present

func (Timeline) String 

func (t Timeline) String() string

String() produces a textual representation of the timeline. The TimelinePoint name and elapsed time are printed for each TimelinePoint in the Timeline's TimelineDescription. If a TimelinePoint is missing a corresponding Entry the TimelinePoint's name is rendered in red.

type TimelineDescription 

type TimelineDescription []TimelinePoint

A TimelineDescription is an ordered list of TimelinePoints.

type TimelinePoint 

type TimelinePoint struct {
	Name    string
	Matcher Matcher
	Squash  float64
}

A TimelinePoint describes a point in a TimelineDescription

The Name is any string to associate with the TimelinePoint The Matcher is used to identify Entries that should be associated with the TimelinePoint

type Timelines 

type Timelines []Timeline

Timelines is a slice of Timeline It is assumed (though not enforced) that all Timeline entries share identical TimelineDescriptions

func (Timelines) CompleteTimelines 

func (t Timelines) CompleteTimelines() Timelines

CompleteTimelines returns the subset of Timelines that are complete.

func (Timelines) DTStatsSlice 

func (t Timelines) DTStatsSlice() DTStatsSlice

DTStatsSlice returns a collection of DTStats -- one for each TimelinePoint in the TimelineDescription

This allows one to efficiently identify (for example) which TimelinePoint contributes the most to the total elapsed time.

func (Timelines) Description 

func (t Timelines) Description() TimelineDescription

Description returns the TimelineDescription associated with the Timelines

func (Timelines) EndsAfter 

func (t Timelines) EndsAfter() time.Duration

StartsAfter finds the largest timeline.EndsAt().Sub(timeline.ZeroEntry.Timestamp) in the entire set of Timelines in the sample

func (Timelines) EntryPairs 

func (t Timelines) EntryPairs(index int) EntryPairs

EntryPairs fetches the EntryPair at the passed-in index for each Timeline then returns the corresponding list of EntryPairs It filters out EntryPairs with negative DTs.

See the documentation for timeline.EntryPair for more details

func (Timelines) GroupBy 

func (t Timelines) GroupBy(matcher Matcher, getter Getter) *GroupedTimelines

GroupBy returns GroupedTimelines - to compute the grouping key a matcher is used to pick out an entry in the timeline then a getter is used to fetch the key

func (Timelines) Len 

func (t Timelines) Len() int

Len returns the length of the Timelines slice

func (Timelines) MatchedEntryPairs 

func (t Timelines) MatchedEntryPairs(i, j int) (EntryPairs, EntryPairs)

MatchedEntryPairs fetches two EntryPairs, one for each of the passed-in indices for each Timeline. It ensures that each entry in the first EntryPairs list corresponds to the entry at the same index in the second list. It also ensures that no entries with negative DTs are in either list.

See the documentation for timeline.EntryPair for more details

func (Timelines) SortByEndTime 

func (t Timelines) SortByEndTime()

SortByEndTime sorts the Timelines in-place by the timestamp returned by timeline.EndsAt()

func (Timelines) SortByEntryAtIndex 

func (t Timelines) SortByEntryAtIndex(index int)

SortByEntryAtIndex sorts the Timelines in-place by the timestamp of the entry at the specifeid index in the TimelineDescription

func (Timelines) SortByStartTime 

func (t Timelines) SortByStartTime()

SortByStartTime sorts the Timelines in-place by the timestamp returned by timeline.BeginsAt()

func (Timelines) SortByVMForEntryAtIndex 

func (t Timelines) SortByVMForEntryAtIndex(index int)

SortByVMForEntryAtindex sorts the Timelines in-place by VM

Since a Timeline can be comprised of events that span multiple VMs one must specify the entry (by index in the TimelineDescription) that should be used to fetch the VM.

func (Timelines) StartsAfter 

func (t Timelines) StartsAfter() time.Duration

StartsAfter finds the smallest timeline.BeginsAt().Sub(timeline.ZeroEntry.Timestamp) in the entire set of Timelines in the sample

func (Timelines) String 

func (t Timelines) String() string

String() joins the strings of the constitutent Timeline entries

func (Timelines) Swap 

func (t Timelines) Swap(i, j int)

Swap swaps two timelines in place

func (Timelines) ToCSV 

func (t Timelines) ToCSV(w io.Writer)

ToCSV generates a CSV file from a timeline

type TransformationFunction 

type TransformationFunction func(interface{}) (interface{}, bool)

func TrimWithPrefixTransformation 

func TrimWithPrefixTransformation(prefix string) TransformationFunction

type TransformationMap 

type TransformationMap map[string]TransformationFunction

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.