pipeline

Documentation

Overview

Package pipeline contains Beam pipeline library functions for the SumDB verifiable map.

Index

• func CreateEntries(s beam.Scope, treeID int64, records beam.PCollection) beam.PCollection
• func MakeVersionLogs(s beam.Scope, treeID int64, metadata beam.PCollection) (beam.PCollection, beam.PCollection)
• func ParseLogInputs(s beam.Scope, logInputs beam.PCollection) beam.PCollection
• type InputLog
• type InputLogLeaf
• type InputLogMetadata
• type MapBuilder
  • func NewMapBuilder(source InputLog, treeID int64, prefixStrata int, versionLogs bool) MapBuilder
  • func (b *MapBuilder) Create(s beam.Scope, size int64) (beam.PCollection, beam.PCollection, InputLogMetadata, error)
  • func (b *MapBuilder) Update(s beam.Scope, lastTiles beam.PCollection, provenance InputLogMetadata, ...) (beam.PCollection, InputLogMetadata, error)
• type Metadata
  • func ParseStatementFn(l InputLogLeaf) Metadata
• type ModuleVersionLog

Functions

func CreateEntries

func CreateEntries(s beam.Scope, treeID int64, records beam.PCollection) beam.PCollection

CreateEntries converts the PCollection<Metadata> into a PCollection<Entry> that will be committed to by the map.

func MakeVersionLogs

func MakeVersionLogs(s beam.Scope, treeID int64, metadata beam.PCollection) (beam.PCollection, beam.PCollection)

MakeVersionLogs takes the Metadata for all modules and processes this by module in order to create logs of versions. The versions for each module are sorted (by ID in the original log), and a log is constructed for each module. This method returns two PCollections: the first is of type Entry and is the key/value data to include in the map, the second is of type ModuleVersionLog.

func ParseLogInputs

func ParseLogInputs(s beam.Scope, logInputs beam.PCollection) beam.PCollection

ParseLogInputs converts the PCollection<InputLogLeaf> into a PCollection<Metadata>.

Types

type InputLog

type InputLog interface {
	// Head returns the metadata of available entries.
	Head() (checkpoint []byte, count int64, err error)
	// Entries returns a PCollection of InputLogLeaf, containing entries in range [start, end).
	Entries(s beam.Scope, start, end int64) beam.PCollection
}

InputLog allows access to entries from the SumDB.

type InputLogLeaf

type InputLogLeaf struct {
	ID   int64
	Data []byte
}

InputLogLeaf is a leaf in an input log, with its sequence index and data.

type InputLogMetadata

type InputLogMetadata struct {
	Checkpoint []byte
	Entries    int64
}

InputLogMetadata describes the provenance information of the input log to be passed around atomically.

type MapBuilder

type MapBuilder struct {
	// contains filtered or unexported fields
}

MapBuilder contains the static configuration for a map, and allows maps at different log sizes to be built using its methods.

func NewMapBuilder

func NewMapBuilder(source InputLog, treeID int64, prefixStrata int, versionLogs bool) MapBuilder

NewMapBuilder returns a MapBuilder for a map with the given configuration.

func (*MapBuilder) Create

func (b *MapBuilder) Create(s beam.Scope, size int64) (beam.PCollection, beam.PCollection, InputLogMetadata, error)

Create builds a map from scratch, using the first `size` entries in the input log. If there aren't enough entries then it will fail. It returns a PCollection of *Tile as the first output, and any logs built will be output in the second PCollection (of type ModuleVersionLog).

func (*MapBuilder) Update

func (b *MapBuilder) Update(s beam.Scope, lastTiles beam.PCollection, provenance InputLogMetadata, size int64) (beam.PCollection, InputLogMetadata, error)

Update builds a map using the last version built, and updating it to include all the first `size` entries from the input log. If there aren't enough entries then it will fail. It returns a PCollection of *Tile as the first output.

type Metadata

type Metadata struct {
	ID       int64
	Module   string
	Version  string
	RepoHash string
	ModHash  string
}

Metadata is not really metadata, and is in fact the parsed RawCloneLeaf. TODO(mhutchinson): rename this.

func ParseStatementFn

func ParseStatementFn(l InputLogLeaf) Metadata

type ModuleVersionLog

type ModuleVersionLog struct {
	Module   string
	Versions []string
}

ModuleVersionLog represents the versions found for a single Go Module within the SumDB log. The versions are sorted by the order they are logged in SumDB.