pipeline

package
v0.0.36 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 3, 2019 License: Apache-2.0, NCSA Imports: 44 Imported by: 2

Documentation

Overview

Package pipeline implements an in-process pipeline to create a combined filetree and xrefs serving table from a stream of GraphStore-ordered entries.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Run

func Run(ctx context.Context, rd stream.EntryReader, db keyvalue.DB, opts *Options) error

Run writes the xrefs and filetree serving tables to db based on the given entries (in GraphStore-order).

Types

type KytheBeam added in v0.0.27

type KytheBeam struct {
	// contains filtered or unexported fields
}

KytheBeam controls the lifetime and generation of PCollections in the Kythe pipeline.

func FromEntries added in v0.0.27

func FromEntries(s beam.Scope, entries beam.PCollection) *KytheBeam

FromEntries creates a KytheBeam pipeline from an input collection of *spb.Entry messages.

func FromNodes added in v0.0.28

func FromNodes(s beam.Scope, nodes beam.PCollection) *KytheBeam

FromNodes creates a KytheBeam pipeline from an input collection of *spb.Nodes.

func (*KytheBeam) CorpusRoots added in v0.0.28

func (k *KytheBeam) CorpusRoots() beam.PCollection

CorpusRoots returns the single *srvpb.CorpusRoots key-value for the Kythe FileTree service. The beam.PCollection has elements of type KV<string, *srvpb.CorpusRoots>.

func (*KytheBeam) CrossReferences added in v0.0.28

func (k *KytheBeam) CrossReferences() (sets, pages beam.PCollection)

CrossReferences returns a Kythe file decorations table derived from the Kythe input graph. The beam.PCollections have elements of type KV<string, *srvpb.PagedCrossReferences> and KV<string, *srvpb.PagedCrossReferences_Page>, respectively.

func (*KytheBeam) Decorations added in v0.0.28

func (k *KytheBeam) Decorations() beam.PCollection

Decorations returns a Kythe file decorations table derived from the Kythe input graph. The beam.PCollection has elements of type KV<string, *srvpb.FileDecorations>.

func (*KytheBeam) Directories added in v0.0.28

func (k *KytheBeam) Directories() beam.PCollection

Directories returns a Kythe *srvpb.FileDirectory table for the Kythe FileTree service. The beam.PCollection has elements of type KV<string, *srvpb.FileDirectory>.

func (*KytheBeam) Documents added in v0.0.28

func (k *KytheBeam) Documents() beam.PCollection

Documents returns a Kythe documentation table derived from the Kythe input graph. The beam.PCollection has elements of type KV<string, *srvpb.Document>.

func (*KytheBeam) Edges added in v0.0.28

func (k *KytheBeam) Edges() (beam.PCollection, beam.PCollection)

Edges returns a Kythe edges table derived from the Kythe input graph. The beam.PCollections have elements of type KV<string, *srvpb.PagedEdgeSet> and KV<string, *srvpb.EdgePage>, respectively.

func (*KytheBeam) Nodes added in v0.0.27

func (k *KytheBeam) Nodes() beam.PCollection

Nodes returns all *scpb.Nodes from the Kythe input graph.

func (*KytheBeam) References added in v0.0.28

func (k *KytheBeam) References() beam.PCollection

References returns all derived *ppb.References from the Kythe input graph.

func (*KytheBeam) SplitCrossReferences added in v0.0.29

func (k *KytheBeam) SplitCrossReferences() beam.PCollection

SplitCrossReferences returns a columnar Kythe cross-references table derived from the Kythe input graph. The beam.PCollection has elements of type KV<[]byte, []byte>.

func (*KytheBeam) SplitDecorations added in v0.0.29

func (k *KytheBeam) SplitDecorations() beam.PCollection

SplitDecorations returns a columnar Kythe file decorations table derived from the Kythe input graph. The beam.PCollection has elements of type KV<[]byte, []byte>.

func (*KytheBeam) SplitEdges added in v0.0.29

func (k *KytheBeam) SplitEdges() beam.PCollection

SplitEdges returns a columnar Kythe edges table derived from the Kythe input graph. The beam.PCollection have elements of type KV<[]byte, []byte>.

type Options added in v0.0.10

type Options struct {
	// Verbose determines whether to emit extra, and possibly excessive, log messages.
	Verbose bool

	// MaxPageSize is maximum number of edges/cross-references that are allowed in
	// PagedEdgeSets, CrossReferences, EdgePages, and CrossReferences_Pages.  If
	// MaxPageSize <= 0, no paging is attempted.
	MaxPageSize int

	// CompressShards determines whether intermediate data written to disk should
	// be compressed.
	CompressShards bool

	// MaxShardSize is the maximum number of elements to keep in-memory before
	// flushing an intermediary data shard to disk.
	MaxShardSize int
}

Options controls the behavior of pipeline.Run.

Directories

Path Synopsis
Package beamio provides Beam transformations for common IO patterns.
Package beamio provides Beam transformations for common IO patterns.
Package beamtest contains utilities to test Apache Beam pipelines.
Package beamtest contains utilities to test Apache Beam pipelines.
Package nodes provides Beam transformations over *scpb.Nodes.
Package nodes provides Beam transformations over *scpb.Nodes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL