pipeline

Documentation

Overview

Package pipeline contains Beam pipeline library functions for the CT verifiable map.

Index

• func MakeDomainLogs(s beam.Scope, treeID int64, logEntries beam.PCollection) (beam.PCollection, beam.PCollection)
• func NodeHash(left, right []byte) []byte
• func RecordHash(data []byte) []byte
• type DomainCertIndexLog
• type InputLog
• type InputLogLeaf
• type InputLogMetadata
• type MapBuilder
  • func NewMapBuilder(source InputLog, treeID int64, prefixStrata int) MapBuilder
  • func (b *MapBuilder) Create(ctx context.Context, s beam.Scope, size int64) (Result, error)
• type Result

Functions

func MakeDomainLogs

func MakeDomainLogs(s beam.Scope, treeID int64, logEntries beam.PCollection) (beam.PCollection, beam.PCollection)

MakeDomainLogs takes all domainEntry records, groups them by domain, and then creates a log containing all of the indices (in order) for which a cert relates to that domain. This method returns two PCollections: 1. the first is of type Entry; the key/value data to include in the map 2. the second is of type DomainCertIndexLog.

func NodeHash

func NodeHash(left, right []byte) []byte

NodeHash returns the hash for an interior tree node with the given left and right hashes.

func RecordHash

func RecordHash(data []byte) []byte

RecordHash returns the content hash for the given record data.

Types

type DomainCertIndexLog

type DomainCertIndexLog struct {
	Domain  string
	Indices []uint64
}

DomainCertIndexLog represents a key/value inside the map, and this value is a log. The key is the domain, and the list of indices is an ordered sequence of leaf index that contains certificates for this domain within the input log that was processed.

type InputLog

type InputLog interface {
	// Head returns the metadata of available entries.
	Head(ctx context.Context) (checkpoint []byte, count int64, err error)

	// Entries returns a PCollection of InputLogLeaf, containing entries in range [start, end).
	Entries(s beam.Scope, start, end int64) beam.PCollection
}

InputLog allows access to entries from the log.

type InputLogLeaf

type InputLogLeaf struct {
	Seq  int64
	Data []byte
}

InputLogLeaf is a leaf in an input log, with its sequence index and data.

type InputLogMetadata

type InputLogMetadata struct {
	Checkpoint []byte
	Entries    int64
}

InputLogMetadata describes the provenance information of the input log to be passed around atomically.

type MapBuilder

type MapBuilder struct {
	// contains filtered or unexported fields
}

MapBuilder contains the static configuration for a map, and allows maps at different log sizes to be built using its methods.

func NewMapBuilder

func NewMapBuilder(source InputLog, treeID int64, prefixStrata int) MapBuilder

NewMapBuilder returns a MapBuilder for a map with the given configuration.

func (*MapBuilder) Create

func (b *MapBuilder) Create(ctx context.Context, s beam.Scope, size int64) (Result, error)

Create builds a map from scratch, using the first `size` entries in the input log. If there aren't enough entries then it will fail.

type Result

type Result struct {
	// MapTiles is a PCollection of *batchmap.Tile.
	MapTiles beam.PCollection
	// DomainCounts is a PCollection of *DomainCertIndexLog.
	DomainCertIndexLogs beam.PCollection
	Metadata            InputLogMetadata
}

Result is returned on successful run of the pipeline. It primarily exists to name the output and aid readability, as PCollections are untyped in code, so having them as named fields at least aids a little.