ninjalog

Documentation

Overview

Package ninjalog provides ninja_log parser

It support ninja log v5.

# ninja log v5
<start>	<end>	<restat>	<target>	<cmdhash>

where

<start> = start time since ninja starts in msec.
<end>   = end time since ninja starts in msec.
<restat> = restat time in epoch.
<target> = target (output) filename
<cmdhash> = hash of command line (?)

It assumes steps in the last build will be ascendent order of <end>.

It also supports metadata added by chromium's buildbot compile.py. metadata is added after

# end of ninja log

and written in json format.

Index

• func Dump(w io.Writer, steps []Step) error
• func Flow(steps []Step, sortByEnd bool) [][]Step
• func TotalTime(steps []Step) (startupTime, endTime, cpuTime time.Duration)
• func UploadTraceOnCriticalPath(ctx context.Context, projectID, traceName string, nlog *NinjaLog, ...) (rerr error)
• func WeightedTime(steps []Step) map[string]time.Duration
• func WriteNinjaLogToGCS(ctx context.Context, nlog *NinjaLog, bucket, filename string) (rerr error)
• type ByDuration
  • func (s ByDuration) Less(i, j int) bool
• type ByEnd
  • func (s ByEnd) Less(i, j int) bool
• type ByWeightedTime
  • func (s ByWeightedTime) Less(i, j int) bool
• type Metadata
• type NinjaLog
  • func Parse(fname string, r io.Reader) (*NinjaLog, error)
• type Stat
  • func StatsByType(steps []Step, weighted map[string]time.Duration, typeOf func(Step) string) []Stat
• type Step
  • func Dedup(steps []Step) []Step
  • func (s Step) Duration() time.Duration
• type Steps
  • func (s Steps) Len() int
  • func (s Steps) Less(i, j int) bool
  • func (s Steps) Reverse()
  • func (s Steps) Swap(i, j int)
• type Trace
  • func ToTraces(steps [][]Step, pid int) []Trace

Functions

func Dump

func Dump(w io.Writer, steps []Step) error

Dump dumps steps as ninja log v5 format in w.

func Flow

func Flow(steps []Step, sortByEnd bool) [][]Step

Flow returns concurrent steps by time. steps in every []Step will not have time overlap. steps will be sorted by start or end time depends on |sortByEnd|.

func TotalTime

func TotalTime(steps []Step) (startupTime, endTime, cpuTime time.Duration)

TotalTime returns startup time and end time of ninja, and accumulated time of all tasks.

func UploadTraceOnCriticalPath

func UploadTraceOnCriticalPath(ctx context.Context, projectID, traceName string, nlog *NinjaLog, invocationID uuid.UUID) (rerr error)

UploadTraceOnCriticalPath uploads build actions included in critical path of build in ninja log to Cloud Trace.

func WeightedTime

func WeightedTime(steps []Step) map[string]time.Duration

WeightedTime calculates weighted time, which is elapsed time with each segment divided by the number of tasks that were running in paralle. This makes it a much better approximation of how "important" a slow step was. For example, A link that is entirely or mostly serialized will have a weighted time that is the same or similar to its elapsed time. A compile that runs in parallel with 999 other compiles will have a weighted time that is tiny.

func WriteNinjaLogToGCS

func WriteNinjaLogToGCS(ctx context.Context, nlog *NinjaLog, bucket, filename string) (rerr error)

WriteNinjaLogToGCS upload ninja log to GCS in avro format.

Types

type ByDuration

type ByDuration struct{ Steps }

ByDuration is used to sort by duration.

func (ByDuration) Less

func (s ByDuration) Less(i, j int) bool

Less is used to sort a list by Steps Duration.

type ByEnd

type ByEnd struct{ Steps }

ByEnd is used to sort by end time.

func (ByEnd) Less

func (s ByEnd) Less(i, j int) bool

Less is used to sort a list by Steps End time.

type ByWeightedTime

type ByWeightedTime struct {
	Weighted map[string]time.Duration
	Steps
}

ByWeightedTime is used to sort by weighted time.

func (ByWeightedTime) Less

func (s ByWeightedTime) Less(i, j int) bool

Less is used to sort a list by Weighted duration.

type Metadata

type Metadata struct {
	// BuildID is identifier of build used in buildbucket api v2 (go/buildbucket-api-v2)
	// Deprecated: Use InvocationID to identify a build invocation.
	// This field is kept only for joining with buildbucket table.
	BuildID int64 `json:"build_id"`

	// InvocationID is identifier for the build invocation.
	// It must be a UUID generated by autoninja or chromium.compile() recipe
	// module API, and is expected to be used consistently in all build tools,
	// such as Siso, Reclient, RBE.
	InvocationID string `json:"invocation_id"`

	// User is the user who runs the build.
	// It is usually an email address.
	User string `json:"user"`

	// ExitCode is exit code of the ninja execution.
	ExitCode int `json:"exit_code"`

	// BuildDurationSec is the time spent on autoninja including the overheads
	// outside of the ninja execution. e.g. Reclient startup/shutdown, Siso shutdown
	BuildDurationSec int `json:"build_duration_sec"`

	// Platform is platform of buildbot.
	Platform string `json:"platform"`

	// Argv is argv of compile.py
	Argv []string `json:"argv"`

	// Cwd is current working directory of compile.py
	Cwd string `json:"cwd"`

	// Compiler is compiler used.
	Compiler string `json:"compiler"`

	// Exit is exit status of ninja.
	Exit int `json:"exit"`

	// Hostname is hostname of builder.
	Hostname string `json:"hostname"`

	// StepName is stepname to distinguish multiple compile steps in a build.
	StepName string `json:"step_name"`

	// CPUCore is the number of cpu cores.
	CPUCore int32 `json:"cpu_core"`

	// IsCloudtop is whether the host machine is a Cloudtop.
	IsCloudtop *bool `json:"is_cloudtop"`

	// GCEMachineType is the GCE machine type of the host machine.
	GCEMachineType string `json:"gce_machine_type"`

	// IsCog is whether the workspace is CoG.
	IsCog *bool `json:"is_cog"`

	// BuildConfigs is content of args.gn.
	BuildConfigs map[string]string `json:"build_configs"`

	// ExplicitBuildConfigKeys is a list of build config keys specified explicitly in
	// args.gn.
	ExplicitBuildConfigKeys []string `json:"explicit_build_config_keys"`

	// Env is environment variables.
	Env map[string]string `json:"env"`

	// Jobs is number of parallel process in a build.
	Jobs int `json:"jobs"`

	// Build target.
	Targets []string `json:"targets"`

	// Raw is raw string for metadata.
	Raw string
	// Error is error message of parsing metadata.
	Error string
}

Metadata is data added by compile.py or json sent from ninjalog uploader.

type NinjaLog

type NinjaLog struct {
	// Filename is a filename of ninja_log.
	Filename string

	// Start is start line of the last build in ninja_log file.
	Start int

	// Steps contains steps in the last build in ninja_log file.
	Steps []Step

	// Metadata is additional data found in ninja_log file.
	Metadata Metadata
}

NinjaLog is parsed data of ninja_log file.

func Parse

func Parse(fname string, r io.Reader) (*NinjaLog, error)

Parse parses .ninja_log file, with chromium's compile.py metadata.

type Stat

type Stat struct {
	Type     string
	Count    int
	Time     time.Duration
	Weighted time.Duration
}

Stat represents statistics for build step.

func StatsByType

func StatsByType(steps []Step, weighted map[string]time.Duration, typeOf func(Step) string) []Stat

StatsByType summarizes build step statistics with weighted and typeOf. Stats is sorted by Weighted, longer first.

type Step

type Step struct {
	Start time.Duration
	End   time.Duration
	// modification time, but not convertible to absolute real time.
	// on POSIX, time_t is used, but on Windows different type is used.
	// htts://github.com/martine/ninja/blob/master/src/timestamp.h
	Restat  int
	Out     string
	CmdHash string

	// other outs for the same CmdHash if dedup'ed.
	Outs []string
}

Step is one step in ninja_log file. time is measured from ninja start time.

func Dedup

func Dedup(steps []Step) []Step

Dedup dedupes steps. step may have the same cmd hash. Dedup only returns the first step for these steps. steps will be sorted by start time.

func (Step) Duration

func (s Step) Duration() time.Duration

Duration reports step's duration.

type Steps

type Steps []Step

Steps is a list of Step. It could be used to sort by start time.

func (Steps) Len

func (s Steps) Len() int

func (Steps) Less

func (s Steps) Less(i, j int) bool

func (Steps) Reverse

func (s Steps) Reverse()

Reverse reverses steps. It would be more efficient if steps is already sorted than using sort.Reverse.

func (Steps) Swap

func (s Steps) Swap(i, j int)

type Trace

type Trace struct {
	Name      string                 `json:"name"`
	Category  string                 `json:"cat"`
	EventType string                 `json:"ph"`
	Timestamp int                    `json:"ts"`  // microsecond
	Duration  int                    `json:"dur"` // microsecond
	ProcessID int                    `json:"pid"`
	ThreadID  int                    `json:"tid"`
	Args      map[string]interface{} `json:"args"`
}

Trace is an entry of trace format. https://code.google.com/p/trace-viewer/

func ToTraces

func ToTraces(steps [][]Step, pid int) []Trace

ToTraces converts Flow outputs into trace log.