summary

README

Summary Engineering Notes

Summary is intended for development use to enable systematic probe logging.

These logs are designed to be one-per-probe. Each test may have multiple probes, and multiple probes will likely share similar probe types. Probes will overwrite within each test, however, to ensure that only one probe type is logged per test.

State

A new State context is created each time probr is run, and is readily accessible anywhere in the code via summary.State.

SummaryStateStruct.LogProbeMeta

Adding entries to the an Probe's meta data requires the name of the test and a key-value pair to be inserted.

n := "name-of-the-current-test"
k = "arbitrary_key_name"
v = "string_value"
summary.State.LogProbeMeta(n, k, v)

SummaryStateStruct.LogPodName

The names of all pods should be tracked, so users may identify whether Probr is the source of any unexpected pods in their cluster.

if pd != nil {
  s.PodName = pd.GetObjectMeta().GetName()
  summary.State.LogPodName(s.PodName)
}

SummaryStateStruct.GetProbeLog

Many summaries will be made directly to probes. In order to do so, the probe must first be retrieved by name. In the example below, the probe is being stored alongside other test context information for easy access during execution.

	ctx.BeforeScenario(func(s *godog.Scenario) {
		ps.name = s.Name
		ps.probe = summary.State.GetProbeLog(NAME)
		coreengine.LogScenarioStart(s)
	})

SummaryStateStruct.ProbeComplete

After an probe has finished running every scenario, we should summary the final outcome of the probe.

s, o, err := g.Handler(g.Data)
summary.State.ProbeComplete(t.ProbeDescriptor.Name)

SummaryStateStruct.SetProbrStatus

After all probes have completed, we should set the final probr status. This step may not always be relevant, as it may be possible to nest it within other methods such as PrintSummary. This should be reevaluated after more feedback has been gathered regarding how Probr is being used.

summary.State.SetProbrStatus()

SummaryStateStruct.PrintSummary

Instead of logging the final status of Probr within a particular loglevel and formatting it using log, PrintSummary simply formats the status into JSON and prints it to the command line. This is currently the very last thing our CLI tool does prior to exiting.

summary.State.PrintSummary()
os.Exit(s)

Probes

Probe.CountPodCreated and Probe.CountPodDestroyed

Whenever Probr will create or destroy a pod, these counters should be called to update the probe accordingly.

Probe.AuditScenarioStep

This function should be used every time a step in a scenario completes. AuditScenarioStep will automatically form the name of the step from the name of the function that called it. The name value provided will establish which scenario the step is a part of. The error (or nil) provided will dictate whether the test passes or fails.

The description and payload values are arbitrary and are used only to assist auditors in their evaluation. A nil error will be recorded as a successful step.

s.audit.AuditScenarioStep( "description string", payloadObject, err) 

Documentation

Index

• type Probe
  • func (e *Probe) CountPodCreated(podName string)
  • func (e *Probe) CountPodDestroyed()
  • func (e *Probe) InitializeAuditor(name string, tags []*messages.Pickle_PickleTag) *ScenarioAudit
• type ProbeAudit
  • func (e *ProbeAudit) Write()
• type ScenarioAudit
  • func (p *ScenarioAudit) AuditScenarioStep(description string, payload interface{}, err error)
• type StepAudit
• type SummaryState
  • func (s *SummaryState) GetProbeLog(n string) *Probe
  • func (s *SummaryState) LogPodName(n string)
  • func (s *SummaryState) LogProbeMeta(name string, key string, value interface{})
  • func (s *SummaryState) PrintSummary()
  • func (s *SummaryState) ProbeComplete(name string)
  • func (s *SummaryState) SetProbrStatus()

Types

type Probe

type Probe struct {
	Meta               map[string]interface{}
	PodsCreated        int
	PodsDestroyed      int
	ScenariosAttempted int
	ScenariosSucceeded int
	ScenariosFailed    int
	Result             string
	// contains filtered or unexported fields
}

func (*Probe) CountPodCreated

func (e *Probe) CountPodCreated(podName string)

CountPodCreated increments pods_created for probe

func (*Probe) CountPodDestroyed

func (e *Probe) CountPodDestroyed()

CountPodDestroyed increments pods_destroyed for probe

func (*Probe) InitializeAuditor

func (e *Probe) InitializeAuditor(name string, tags []*messages.Pickle_PickleTag) *ScenarioAudit

type ProbeAudit

type ProbeAudit struct {
	Name               string
	PodsDestroyed      *int
	ScenariosAttempted *int
	ScenariosSucceeded *int
	ScenariosFailed    *int
	Result             *string
	Scenarios          map[int]*ScenarioAudit
	// contains filtered or unexported fields
}

func (*ProbeAudit) Write

func (e *ProbeAudit) Write()

type ScenarioAudit

type ScenarioAudit struct {
	Name   string
	Result string // Passed / Failed / Given Not Met
	Tags   []string
	Steps  map[int]*StepAudit
}

func (*ScenarioAudit) AuditScenarioStep

func (p *ScenarioAudit) AuditScenarioStep(description string, payload interface{}, err error)

auditScenarioStep sets description, payload, and pass/fail based on err parameter. This function should be deferred to catch panic behavior, otherwise the audit will not be logged on panic

type StepAudit

type StepAudit struct {
	Name        string
	Description string      // Long-form explanation of anything happening in the step
	Result      string      // Passed / Failed
	Error       string      // Log the error text
	Payload     interface{} // Handles any values that are sent across the network
}

type SummaryState

type SummaryState struct {
	Meta          map[string]interface{}
	Status        string
	ProbesPassed  int
	ProbesFailed  int
	ProbesSkipped int
	Probes        map[string]*Probe
}

var State SummaryState

func (*SummaryState) GetProbeLog

func (s *SummaryState) GetProbeLog(n string) *Probe

GetProbeLog initializes or returns existing log probe for the provided test name

func (*SummaryState) LogPodName

func (s *SummaryState) LogPodName(n string)

LogPodName adds pod names to a list for user's debugging purposes

func (*SummaryState) LogProbeMeta

func (s *SummaryState) LogProbeMeta(name string, key string, value interface{})

LogProbeMeta accepts a test name with a key and value to insert to the meta logs for that test. Overwrites key if already present.

func (*SummaryState) PrintSummary

func (s *SummaryState) PrintSummary()

PrintSummary will print the current Probes object state, formatted to JSON, if NoSummary is not "true"

func (*SummaryState) ProbeComplete

func (s *SummaryState) ProbeComplete(name string)

ProbeComplete takes an probe name and status then updates the summary & probe meta information

func (*SummaryState) SetProbrStatus

func (s *SummaryState) SetProbrStatus()

SetProbrStatus evaluates the current SummaryState state to set the Status