core

Documentation

Overview

Package core provides transport-agnostic implementation of Migrillian tool.

Index

• func LoadConfigFromFile(filename string) (*configpb.MigrillianConfig, error)
• func RunMigration(ctx context.Context, ctrls []*Controller)
• func ValidateConfig(cfg *configpb.MigrillianConfig) (ctfe.LogBackendMap, error)
• func ValidateMigrationConfig(cfg *configpb.MigrationConfig) error
• type Controller
  • func NewController(opts Options, ctClient *client.LogClient, plClient *PreorderedLogClient, ...) *Controller
  • func (c *Controller) Run(ctx context.Context) error
  • func (c *Controller) RunWhenMaster(ctx context.Context) error
• type Options
• type PreorderedLogClient
  • func NewPreorderedLogClient(cli trillian.TrillianLogClient, tree *trillian.Tree, prefix string) (*PreorderedLogClient, error)

Functions

func LoadConfigFromFile

func LoadConfigFromFile(filename string) (*configpb.MigrillianConfig, error)

LoadConfigFromFile reads MigrillianConfig from the given filename, which should contain text-protobuf encoded configuration data.

func RunMigration

func RunMigration(ctx context.Context, ctrls []*Controller)

RunMigration migrates data from a number of CT logs to Trillian. Each log's migration is coordinated by the corresponding Controller. This function terminates when all Controllers are done (possibly with an erorr, or as a result of canceling the passed in context).

TODO(pavelkalinnikov):

• Surface status of each goroutine in monitoring.
• Handle Controller failures, e.g. restart them (depends on error type).
• Introduce a MultiController type.
• Start Controllers with random delays to prevent one instance (e.g. the one that has started first) capturing mastership over all logs at once. This is particularly important for mirroring.
• Add voluntary mastership resignations.

func ValidateConfig

func ValidateConfig(cfg *configpb.MigrillianConfig) (ctfe.LogBackendMap, error)

ValidateConfig verifies that MigrillianConfig is correct. In particular: - The log backends have distinct non-empty names and backend specs. - Migration configs are valid (as per ValidateMigrationConfig). - Migration configs specify backend names present in the set of backends. - Each migration config has a unique (backend, tree ID) pair. Returns a map from log backend names to the corresponding LogBackend.

func ValidateMigrationConfig

func ValidateMigrationConfig(cfg *configpb.MigrationConfig) error

ValidateMigrationConfig verifies that the migration config is sane.

Types

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

Controller coordinates migration from a CT log to a Trillian tree.

TODO(pavelkalinnikov): - Schedule a distributed fetch to increase throughput. - Store CT STHs in Trillian or make this tool stateful on its own. - Make fetching stateful to reduce master resigning aftermath.

func NewController

func NewController(
	opts Options,
	ctClient *client.LogClient,
	plClient *PreorderedLogClient,
	ef election.Factory,
	mf monitoring.MetricFactory,
) *Controller

NewController creates a Controller configured by the passed in options, CT and Trillian clients, and a master election factory.

The passed in MetricFactory is used to create per-tree metrics, and it should be the same for all instances. However, it is used only once.

func (*Controller) Run

func (c *Controller) Run(ctx context.Context) error

Run transfers CT log entries obtained via the CT log client to a Trillian pre-ordered log via Trillian client. If Options.Continuous is true then the migration process runs continuously trying to keep up with the target CT log. Returns if an error occurs, the context is canceled, or all the entries have been transferred (in non-Continuous mode).

func (*Controller) RunWhenMaster

func (c *Controller) RunWhenMaster(ctx context.Context) error

RunWhenMaster is a master-elected version of Run method. It executes Run whenever this instance captures mastership of the tree ID. As soon as the instance stops being the master, Run is canceled. The method returns if a severe error occurs, the passed in context is canceled, or fetching is completed (in non-Continuous mode). Releases mastership when terminates.

type Options

type Options struct {
	scanner.FetcherOptions
	Submitters          int
	BatchesPerSubmitter int
}

Options holds configuration for a Controller.

type PreorderedLogClient

type PreorderedLogClient struct {
	// contains filtered or unexported fields
}

PreorderedLogClient is a means of communicating with a single Trillian pre-ordered log tree.

func NewPreorderedLogClient

func NewPreorderedLogClient(
	cli trillian.TrillianLogClient, tree *trillian.Tree, prefix string,
) (*PreorderedLogClient, error)

NewPreorderedLogClient creates and initializes a pre-ordered log client.