replay

Documentation

Overview

Package replay implements collection and replaying of compaction benchmarking workloads. A workload is a collection of flushed and ingested sstables, along with the corresponding manifests describing the order and grouping with which they were applied. Replaying a workload flushes and ingests the same keys and sstables to reproduce the write workload for the purpose of evaluating compaction heuristics.

Index

• type Metrics
  • func (m *Metrics) Plots(width, height int) []Plot
  • func (m *Metrics) WriteBenchmarkString(name string, w io.Writer) error
• type PaceByFixedReadAmp
• type PaceByReferenceReadAmp
• type Pacer
• type Plot
• type Runner
  • func (r *Runner) Close() error
  • func (r *Runner) Run(ctx context.Context) error
  • func (r *Runner) Wait() (Metrics, error)
• type SampledMetric
  • func (m *SampledMetric) Max() int64
  • func (m *SampledMetric) Mean() float64
  • func (m *SampledMetric) Min() int64
  • func (m *SampledMetric) Plot(width, height int, scale float64) string
  • func (m *SampledMetric) PlotIncreasingPerSec(width, height int, scale float64) string
  • func (m *SampledMetric) Values(n int) []float64
• type Unpaced
• type WorkloadCollector
  • func NewWorkloadCollector(srcDir string) *WorkloadCollector
  • func (w *WorkloadCollector) Attach(opts *pebble.Options)
  • func (w *WorkloadCollector) IsRunning() bool
  • func (w *WorkloadCollector) Start(destFS vfs.FS, destPath string)
  • func (w *WorkloadCollector) Stop()
  • func (w *WorkloadCollector) WaitAndStop()

Types

type Metrics

type Metrics struct {
	CompactionCounts struct {
		Total       int64
		Default     int64
		DeleteOnly  int64
		ElisionOnly int64
		Move        int64
		Read        int64
		Rewrite     int64
		MultiLevel  int64
	}
	EstimatedDebt SampledMetric
	Final         *pebble.Metrics
	Ingest        struct {
		BytesIntoL0 uint64
		// BytesWeightedByLevel is calculated as the number of bytes ingested
		// into a level multiplied by the level's distance from the bottommost
		// level (L6), summed across all levels. It can be used to guage how
		// effective heuristics are at ingesting files into lower levels, saving
		// write amplification.
		BytesWeightedByLevel uint64
	}
	// PaceDuration is the time waiting for the pacer to allow the workload to
	// continue.
	PaceDuration time.Duration
	ReadAmp      SampledMetric
	// QuiesceDuration is the time between completing application of the workload and
	// compactions quiescing.
	QuiesceDuration time.Duration
	TombstoneCount  SampledMetric
	// TotalSize holds the total size of the database, sampled after each
	// workload step.
	TotalSize           SampledMetric
	TotalWriteAmp       float64
	WorkloadDuration    time.Duration
	WriteBytes          uint64
	WriteStalls         map[string]int
	WriteStallsDuration map[string]time.Duration
	WriteThroughput     SampledMetric
}

Metrics holds the various statistics on a replay run and its performance.

func (*Metrics) Plots

func (m *Metrics) Plots(width, height int) []Plot

Plots returns a slice of ascii plots describing metrics change over time.

func (*Metrics) WriteBenchmarkString

func (m *Metrics) WriteBenchmarkString(name string, w io.Writer) error

WriteBenchmarkString writes the metrics in the form of a series of 'Benchmark' lines understandable by benchstat.

type PaceByFixedReadAmp

type PaceByFixedReadAmp int

PaceByFixedReadAmp implements Pacer by applying each new write following a fixed read amplification.

type PaceByReferenceReadAmp

type PaceByReferenceReadAmp struct{}

PaceByReferenceReadAmp implements Pacer by applying each new write following the collected workloads read amplification.

type Pacer

type Pacer interface {
	// contains filtered or unexported methods
}

A Pacer paces replay of a workload, determining when to apply the next incoming write.

type Plot

type Plot struct {
	Name string
	Plot string
}

Plot holds an ascii plot and its name.

type Runner

type Runner struct {
	RunDir        string
	WorkloadFS    vfs.FS
	WorkloadPath  string
	Pacer         Pacer
	Opts          *pebble.Options
	MaxWriteBytes uint64
	// contains filtered or unexported fields
}

Runner runs a captured workload against a test database, collecting metrics on performance.

func (*Runner) Close

func (r *Runner) Close() error

Close closes remaining open resources, including the database. It must be called after Wait.

func (*Runner) Run

func (r *Runner) Run(ctx context.Context) error

Run begins executing the workload and returns.

The workload application will respect the provided context's cancellation.

func (*Runner) Wait

func (r *Runner) Wait() (Metrics, error)

Wait waits for the workload replay to complete. Wait returns once the entire workload has been replayed, and compactions have quiesced.

type SampledMetric

type SampledMetric struct {
	// contains filtered or unexported fields
}

SampledMetric holds a metric that is sampled at various points of workload replay. Samples are collected when a new step in the workload is applied to the database, and whenever a compaction completes.

func (*SampledMetric) Max

func (m *SampledMetric) Max() int64

Max calculates the maximum value of the metric.

func (*SampledMetric) Mean

func (m *SampledMetric) Mean() float64

Mean calculates the mean value of the metric.

func (*SampledMetric) Min

func (m *SampledMetric) Min() int64

Min calculates the mininum value of the metric.

func (*SampledMetric) Plot

func (m *SampledMetric) Plot(width, height int, scale float64) string

Plot returns an ASCII graph plot of the metric over time, with the provided width and height determining the size of the graph and the number of representable discrete x and y points. All values are first multiplied by the provided scale parameter before graphing.

func (*SampledMetric) PlotIncreasingPerSec

func (m *SampledMetric) PlotIncreasingPerSec(width, height int, scale float64) string

PlotIncreasingPerSec returns an ASCII graph plot of the increasing delta of a metric over time, per-second. The provided width and height determine the size of the graph and the number of representable discrete x and y points. All deltas are multiplied by the provided scale parameter and scaled to per-second before graphing.

func (*SampledMetric) Values

func (m *SampledMetric) Values(n int) []float64

Values returns the values of the metric, distributed across n discrete buckets that are equally spaced over time. If multiple values fall within a bucket, the latest recorded value is used. If no values fall within a bucket, the next recorded value is used.

type Unpaced

type Unpaced struct{}

Unpaced implements Pacer by applying each new write as soon as possible. It may be useful for examining performance under high read amplification.

type WorkloadCollector

type WorkloadCollector struct {
	// contains filtered or unexported fields
}

WorkloadCollector is designed to capture workloads by handling manifest files, flushed SSTs and ingested SSTs. The collector hooks into the pebble.EventListener and pebble.Cleaner in order keep track of file states.

func NewWorkloadCollector

func NewWorkloadCollector(srcDir string) *WorkloadCollector

NewWorkloadCollector is used externally to create a New WorkloadCollector.

func (*WorkloadCollector) Attach

func (w *WorkloadCollector) Attach(opts *pebble.Options)

Attach is used to set up the WorkloadCollector by attaching itself to pebble.Options EventListener and Cleaner.

func (*WorkloadCollector) IsRunning

func (w *WorkloadCollector) IsRunning() bool

IsRunning returns whether the WorkloadCollector is currently running.

func (*WorkloadCollector) Start

func (w *WorkloadCollector) Start(destFS vfs.FS, destPath string)

Start begins collecting a workload. All flushed and ingested sstables, plus corresponding manifests are copied to the provided destination path on the provided FS.

func (*WorkloadCollector) Stop

func (w *WorkloadCollector) Stop()

Stop stops collection of the workload.

func (*WorkloadCollector) WaitAndStop

func (w *WorkloadCollector) WaitAndStop()

WaitAndStop waits for all enqueued sstables to be copied over, and then calls Stop. Gracefully ensures that all sstables referenced in the collected manifest's latest version edit will exist in the copy directory.