xlangx

Documentation

Overview

Package xlangx contains various low-level utilities needed for adding cross-language transforms to the pipeline.

Index

• func EncodeStructPayload(pl interface{}) ([]byte, error)
• func Expand(edge *graph.MultiEdge, ext *graph.ExternalTransform) error
• func ResolveArtifacts(ctx context.Context, edges []*graph.MultiEdge, p *pipepb.Pipeline)
• func ResolveArtifactsWithConfig(ctx context.Context, edges []*graph.MultiEdge, cfg ResolveConfig) (paths map[string]string, err error)
• type ResolveConfig

Functions

func EncodeStructPayload

func EncodeStructPayload(pl interface{}) ([]byte, error)

EncodeStructPayload takes a native Go struct and returns a marshaled ExternalConfigurationPayload proto, containing a Schema representation of the original type and the original value encoded as a Row. This is intended to be used as the expansion payload for an External transform.

func Expand

func Expand(edge *graph.MultiEdge, ext *graph.ExternalTransform) error

Expand expands an unexpanded graph.ExternalTransform as a graph.ExpandedTransform and assigns it to the ExternalTransform's Expanded field. This requires querying an expansion service based on the configuration details within the ExternalTransform.

func ResolveArtifacts

func ResolveArtifacts(ctx context.Context, edges []*graph.MultiEdge, p *pipepb.Pipeline)

ResolveArtifacts acquires all dependencies for a cross-language transform

func ResolveArtifactsWithConfig

func ResolveArtifactsWithConfig(ctx context.Context, edges []*graph.MultiEdge, cfg ResolveConfig) (paths map[string]string, err error)

ResolveArtifactsWithConfig acquires all dependencies for cross-language transforms, but with some additional configuration to behavior. By default, this function performs the following steps for each cross-language transform in the list of edges:

1. Retrieves a list of dependencies needed from the expansion service.
2. Retrieves each dependency as an artifact and stages it to a default local filepath.
3. Adds the dependencies to the transform's stored environment proto.

The changes that can be configured are documented in ResolveConfig.

This returns a map of "local path" to "sdk path". By default these are identical, unless ResolveConfig.SdkPath has been set.

Types

type ResolveConfig

type ResolveConfig struct {
	// SdkPath replaces the default filepath for dependencies, but only in the
	// external environment proto to be used by the SDK Harness during pipeline
	// execution. This is used to specify alternate staging directories, such
	// as for staging artifacts remotely.
	//
	// Setting an SdkPath does not change staging behavior otherwise. All
	// artifacts still get staged to the default local filepath, and it is the
	// user's responsibility to stage those local artifacts to the SdkPath.
	SdkPath string

	// JoinFn is a function for combining SdkPath and individual artifact names.
	// If not specified, it defaults to using filepath.Join.
	JoinFn func(path, name string) string
}

ResolveConfig contains fields for configuring the behavior for resolving artifacts.