Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RunPipeline ¶
func RunPipeline(j *jobservices.Job)
RunPipeline starts the main thread fo executing this job. It's analoguous to the manager side process for a distributed pipeline. It will begin "workers"
Types ¶
type CombineCharacteristic ¶
type CombineCharacteristic struct {
EnableLifting bool // Sets whether a combine composite does combiner lifting or not.
}
CombineCharacteristic holds the configuration for Combines.
type ParDoCharacteristic ¶
type ParDoCharacteristic struct {
DisableSDF bool // Sets whether a pardo supports SDFs or not.
}
ParDoCharacteristic holds the configuration for ParDos.
type RunnerCharacteristic ¶
type RunnerCharacteristic struct { SDKFlatten bool // Sets whether we should force an SDK side flatten. SDKGBK bool // Sets whether the GBK should be handled by the SDK, if possible by the SDK. }
RunnerCharacteristic holds the configuration for Runner based transforms, such as GBKs, Flattens.
type SourceConfig ¶
type SourceConfig struct { NumElements int64 `json:"num_records" beam:"num_records"` InitialSplits int64 `json:"initial_splits" beam:"initial_splits"` }
SourceConfig is a struct containing all the configuration options for a synthetic source. It should be created via a SourceConfigBuilder, not by directly initializing it (the fields are public to allow encoding).
Source Files ¶
Directories ¶
Path | Synopsis |
---|---|
Package config defines and handles the parsing and provision of configurations for the runner.
|
Package config defines and handles the parsing and provision of configurations for the runner. |
Package engine handles the operational components of a runner, to track elements, watermarks, timers, triggers etc
|
Package engine handles the operational components of a runner, to track elements, watermarks, timers, triggers etc |
Package jobservices handles services necessary WRT handling jobs from SDKs.
|
Package jobservices handles services necessary WRT handling jobs from SDKs. |
Package urns handles extracting urns from all the protos.
|
Package urns handles extracting urns from all the protos. |
Package worker handles interactions with SDK side workers, representing the worker services, communicating with those services, and SDK environments.
|
Package worker handles interactions with SDK side workers, representing the worker services, communicating with those services, and SDK environments. |
Click to show internal directories.
Click to hide internal directories.