pipeline

Documentation

Overview

Package pipeline implements an in-process pipeline to create a combined filetree and xrefs serving table from a stream of GraphStore-ordered entries.

Index

• func Run(ctx context.Context, rd stream.EntryReader, db keyvalue.DB, opts *Options) error
• type KytheBeam
  • func FromEntries(s beam.Scope, entries beam.PCollection) *KytheBeam
  • func FromNodes(s beam.Scope, nodes beam.PCollection) *KytheBeam
  • func (k *KytheBeam) CorpusRoots() beam.PCollection
  • func (k *KytheBeam) CrossReferences() (sets, pages beam.PCollection)
  • func (k *KytheBeam) Decorations() beam.PCollection
  • func (k *KytheBeam) Directories() beam.PCollection
  • func (k *KytheBeam) Documents() beam.PCollection
  • func (k *KytheBeam) Edges() (beam.PCollection, beam.PCollection)
  • func (k *KytheBeam) Nodes() beam.PCollection
  • func (k *KytheBeam) References() beam.PCollection
  • func (k *KytheBeam) SplitCrossReferences() beam.PCollection
  • func (k *KytheBeam) SplitDecorations() beam.PCollection
  • func (k *KytheBeam) SplitEdges() beam.PCollection
• type Options

Functions

func Run

func Run(ctx context.Context, rd stream.EntryReader, db keyvalue.DB, opts *Options) error

Run writes the xrefs and filetree serving tables to db based on the given entries (in GraphStore-order).

Types

type KytheBeam

type KytheBeam struct {
	// contains filtered or unexported fields
}

KytheBeam controls the lifetime and generation of PCollections in the Kythe pipeline.

func FromEntries

func FromEntries(s beam.Scope, entries beam.PCollection) *KytheBeam

FromEntries creates a KytheBeam pipeline from an input collection of *spb.Entry messages.

func FromNodes

func FromNodes(s beam.Scope, nodes beam.PCollection) *KytheBeam

FromNodes creates a KytheBeam pipeline from an input collection of *spb.Nodes.

func (*KytheBeam) CorpusRoots

func (k *KytheBeam) CorpusRoots() beam.PCollection

CorpusRoots returns the single *srvpb.CorpusRoots key-value for the Kythe FileTree service. The beam.PCollection has elements of type KV<string, *srvpb.CorpusRoots>.

func (*KytheBeam) CrossReferences

func (k *KytheBeam) CrossReferences() (sets, pages beam.PCollection)

CrossReferences returns a Kythe file decorations table derived from the Kythe input graph. The beam.PCollections have elements of type KV<string, *srvpb.PagedCrossReferences> and KV<string, *srvpb.PagedCrossReferences_Page>, respectively.

func (*KytheBeam) Decorations

func (k *KytheBeam) Decorations() beam.PCollection

Decorations returns a Kythe file decorations table derived from the Kythe input graph. The beam.PCollection has elements of type KV<string, *srvpb.FileDecorations>.

func (*KytheBeam) Directories

func (k *KytheBeam) Directories() beam.PCollection

Directories returns a Kythe *srvpb.FileDirectory table for the Kythe FileTree service. The beam.PCollection has elements of type KV<string, *srvpb.FileDirectory>.

func (*KytheBeam) Documents

func (k *KytheBeam) Documents() beam.PCollection

Documents returns a Kythe documentation table derived from the Kythe input graph. The beam.PCollection has elements of type KV<string, *srvpb.Document>.

func (*KytheBeam) Edges

func (k *KytheBeam) Edges() (beam.PCollection, beam.PCollection)

Edges returns a Kythe edges table derived from the Kythe input graph. The beam.PCollections have elements of type KV<string, *srvpb.PagedEdgeSet> and KV<string, *srvpb.EdgePage>, respectively.

func (*KytheBeam) Nodes

func (k *KytheBeam) Nodes() beam.PCollection

Nodes returns all *scpb.Nodes from the Kythe input graph.

func (*KytheBeam) References

func (k *KytheBeam) References() beam.PCollection

References returns all derived *ppb.References from the Kythe input graph.

func (*KytheBeam) SplitCrossReferences

func (k *KytheBeam) SplitCrossReferences() beam.PCollection

SplitCrossReferences returns a columnar Kythe cross-references table derived from the Kythe input graph. The beam.PCollection has elements of type KV<[]byte, []byte>.

func (*KytheBeam) SplitDecorations

func (k *KytheBeam) SplitDecorations() beam.PCollection

SplitDecorations returns a columnar Kythe file decorations table derived from the Kythe input graph. The beam.PCollection has elements of type KV<[]byte, []byte>.

func (*KytheBeam) SplitEdges

func (k *KytheBeam) SplitEdges() beam.PCollection

SplitEdges returns a columnar Kythe edges table derived from the Kythe input graph. The beam.PCollection have elements of type KV<[]byte, []byte>.

type Options

type Options struct {
	// Verbose determines whether to emit extra, and possibly excessive, log messages.
	Verbose bool

	// MaxPageSize is maximum number of edges/cross-references that are allowed in
	// PagedEdgeSets, CrossReferences, EdgePages, and CrossReferences_Pages.  If
	// MaxPageSize <= 0, no paging is attempted.
	MaxPageSize int

	// CompressShards determines whether intermediate data written to disk should
	// be compressed.
	CompressShards bool

	// MaxShardSize is the maximum number of elements to keep in-memory before
	// flushing an intermediary data shard to disk.
	MaxShardSize int
}

Options controls the behavior of pipeline.Run.