pipeline

Documentation

Overview

Package pipeline implements an in-process pipeline to create a combined filetree and xrefs serving table from a stream of GraphStore-ordered entries.

Index

• func Run(ctx context.Context, entries <-chan *spb.Entry, db keyvalue.DB, opts *Options) error
• type Options

Functions

func Run

func Run(ctx context.Context, entries <-chan *spb.Entry, db keyvalue.DB, opts *Options) error

Run writes the xrefs and filetree serving tables to db based on the given entries (in GraphStore-order).

Types

type Options

type Options struct {
	// Verbose determines whether to emit extra, and possibly excessive, log messages.
	Verbose bool

	// MaxPageSize is maximum number of edges/cross-references that are allowed in
	// PagedEdgeSets, CrossReferences, EdgePages, and CrossReferences_Pages.  If
	// MaxPageSize <= 0, no paging is attempted.
	MaxPageSize int

	// CompressShards determines whether intermediate data written to disk should
	// be compressed.
	CompressShards bool

	// MaxShardSize is the maximum number of elements to keep in-memory before
	// flushing an intermediary data shard to disk.
	MaxShardSize int

	// IOBufferSize is the size of the reading/writing buffers for the temporary
	// file shards.
	IOBufferSize int
}

Options controls the behavior of pipeline.Run.