ninjalog

Documentation

Overview

Package ninjalog provides ninja_log parser

It support ninja log v5.

# ninja log v5
<start>	<end>	<restat>	<target>	<cmdhash>

where

<start> = start time since ninja starts in msec.
<end>   = end time since ninja starts in msec.
<restat> = restat time in epoch.
<target> = target (output) filename
<cmdhash> = hash of command line (?)

It assumes steps in the last build will be ascendent order of <end>.

It also supports metadata added by chromium's buildbot compile.py. metadata is added after

# end of ninja log

and written in json format.

Index

• func ClangTracesToInterleave(mainTraces []chrometrace.Trace, buildRoot string, granularity time.Duration) ([]chrometrace.Trace, error)
• func Dump(w io.Writer, steps []Step) error
• func Flow(steps []Step) [][]Step
• func MurmurHash64A(data []byte) uint64
• func ToTraces(steps [][]Step, pid int) []chrometrace.Trace
• func TotalTime(steps []Step) (startupTime, endTime, cpuTime time.Duration)
• func WeightedTime(steps []Step) map[string]time.Duration
• type ByDuration
  • func (s ByDuration) Less(i, j int) bool
• type ByEnd
  • func (s ByEnd) Less(i, j int) bool
• type ByWeightedTime
  • func (s ByWeightedTime) Less(i, j int) bool
• type Metadata
• type NinjaLog
  • func Parse(fname string, r io.Reader) (*NinjaLog, error)
• type Stat
  • func StatsByType(steps []Step, weighted map[string]time.Duration, typeOf func(Step) string) []Stat
• type Step
  • func Dedup(steps []Step) []Step
  • func Populate(steps []Step, commands []compdb.Command) []Step
  • func SlowestSteps(steps []Step, n int) []Step
  • func (s Step) AllOutputs() []string
  • func (s Step) Category() string
  • func (s Step) Duration() time.Duration
• type Steps
  • func (s Steps) Len() int
  • func (s Steps) Less(i, j int) bool
  • func (s Steps) Reverse()
  • func (s Steps) Swap(i, j int)

Functions

func ClangTracesToInterleave

func ClangTracesToInterleave(mainTraces []chrometrace.Trace, buildRoot string, granularity time.Duration) ([]chrometrace.Trace, error)

ClangTracesToInterleave returns all clang traces that can be interleaved directly into their corresponding build steps from `mainTraces`.

`buildRoot` should point to the directory where the Ninja build of `mainTrace` is executed, where this function will look for clang traces next to object files built by clang. Object files with no clang traces next to them are skipped.

Clang traces include events with very short durations, so `granularity` is provided to filter them and reduce the size of returned slice.

func Dump

func Dump(w io.Writer, steps []Step) error

Dump dumps steps as ninja log v5 format in w.

func Flow

func Flow(steps []Step) [][]Step

Flow returns concurrent steps by time. steps in every []Step will not have time overlap. steps will be sorted by start time.

func MurmurHash64A

func MurmurHash64A(data []byte) uint64

MurmurHash64A computes Murmur hash using the same exact algorithm and seed that is used by Ninja in .ninja_log. See https://github.com/ninja-build/ninja/blob/cc79afb/src/build_log.cc#L64

func ToTraces

func ToTraces(steps [][]Step, pid int) []chrometrace.Trace

ToTraces converts Flow outputs into trace log.

If a non-empty `criticalPath` is provided, steps on the critical path will have "critical_path" in their category to enable quick searching and highlighting.

func TotalTime

func TotalTime(steps []Step) (startupTime, endTime, cpuTime time.Duration)

TotalTime returns startup time and end time of ninja, and accumulated time of all tasks.

func WeightedTime

func WeightedTime(steps []Step) map[string]time.Duration

WeightedTime calculates weighted time, which is elapsed time with each segment divided by the number of tasks that were running in paralle. This makes it a much better approximation of how "important" a slow step was. For example, A link that is entirely or mostly serialized will have a weighted time that is the same or similar to its elapsed time. A compile that runs in parallel with 999 other compiles will have a weighted time that is tiny.

Types

type ByDuration

type ByDuration struct{ Steps }

ByDuration is used to sort by duration.

func (ByDuration) Less

func (s ByDuration) Less(i, j int) bool

type ByEnd

type ByEnd struct{ Steps }

ByEnd is used to sort by end time.

func (ByEnd) Less

func (s ByEnd) Less(i, j int) bool

type ByWeightedTime

type ByWeightedTime struct {
	Weighted map[string]time.Duration
	Steps
}

ByWeightedTime is used to sort by weighted time.

func (ByWeightedTime) Less

func (s ByWeightedTime) Less(i, j int) bool

type Metadata

type Metadata struct {
	// Platform is platform of buildbot.
	Platform string `json:"platform"`

	// Argv is argv of compile.py
	Argv []string `json:"argv"`

	// Cwd is current working directory of compile.py
	Cwd string `json:"cwd"`

	// Compiler is compiler used.
	Compiler string `json:"compiler"`

	// Cmdline is command line of ninja.
	Cmdline []string `json:"cmdline"`

	// Exit is exit status of ninja.
	Exit int `json:"exit"`

	// Env is environment variables.
	Env map[string]string `json:"env"`

	// CompilerProxyInfo is a path name of associated compiler_proxy.INFO log.
	CompilerProxyInfo string `json:"compiler_proxy_info"`

	// Raw is raw string for metadata.
	Raw string
	// Error is error message of parsing metadata.
	Error string
}

Metadata is data added by compile.py.

type NinjaLog

type NinjaLog struct {
	// Filename is a filename of ninja_log.
	Filename string

	// Start is start line of the last build in ninja_log file.
	Start int

	// Steps contains steps in the last build in ninja_log file.
	Steps []Step

	// Metadata is additional data found in ninja_log file.
	Metadata Metadata
}

NinjaLog is parsed data of ninja_log file.

func Parse

func Parse(fname string, r io.Reader) (*NinjaLog, error)

Parse parses .ninja_log file, with chromium's compile.py metadata.

type Stat

type Stat struct {
	// Type used to group this stat, this is determined by the grouping function
	// provided by the caller when this stat is calculated.
	Type string
	// Count of builds for this type.
	Count int32
	// Accumulative build time for this type.
	Time time.Duration
	// Accumulative weighted build time for this time.
	Weighted time.Duration
	// Build times of all actions grouped under this stat.
	Times []time.Duration
}

Stat represents statistics for build step.

func StatsByType

func StatsByType(steps []Step, weighted map[string]time.Duration, typeOf func(Step) string) []Stat

StatsByType summarizes build step statistics with weighted and typeOf. Order of the returned slice is undefined.

type Step

type Step struct {
	Start time.Duration
	End   time.Duration
	// modification time, but not convertable to absolute real time.
	// on POSIX, time_t is used, but on Windows different type is used.
	// htts://github.com/martine/ninja/blob/HEAD/src/timestamp.h
	Restat  int
	Out     string
	CmdHash uint64

	// other outs for the same CmdHash if dedup'ed.
	Outs []string
	// compilation database command
	Command *compdb.Command

	// Whether ths step is on the critical path.
	OnCriticalPath bool
	// TotalFloat is the amount of time this step can be delayed without affecting
	// the completion time of the build.
	//
	// https://en.wikipedia.org/wiki/Float_(project_management)
	TotalFloat time.Duration
	// Drag is the amount of time this step is adding to the total build time. All
	// non-critial steps have zero drag.
	//
	// https://en.wikipedia.org/wiki/Critical_path_drag
	Drag time.Duration
}

Step is one step in ninja_log file. time is measured from ninja start time.

func Dedup

func Dedup(steps []Step) []Step

Dedup dedupes steps. step may have the same cmd hash. Dedup only returns the first step for these steps. steps will be sorted by start time.

func Populate

func Populate(steps []Step, commands []compdb.Command) []Step

Populate the steps with information from the compilation database.

func SlowestSteps

func SlowestSteps(steps []Step, n int) []Step

SlowestSteps returns the `n` steps that took the longest time to finish.

Returned steps are sorted on build time in descending order (step takes the longest time to build is the first element).

func (Step) AllOutputs

func (s Step) AllOutputs() []string

AllOutputs return all outputs of this step.

func (Step) Category

func (s Step) Category() string

category returns a comma separated list of executed commands.

func (Step) Duration

func (s Step) Duration() time.Duration

Duration reports step's duration.

type Steps

type Steps []Step

Steps is a list of Step. It could be used to sort by start time.

func (Steps) Len

func (s Steps) Len() int

func (Steps) Less

func (s Steps) Less(i, j int) bool

func (Steps) Reverse

func (s Steps) Reverse()

Reverse reverses steps. It would be more efficient if steps is already sorted than using sort.Reverse.

func (Steps) Swap

func (s Steps) Swap(i, j int)