artifact

Documentation

Overview

Package artifact contains utilities for staging and retrieving artifacts.

Index

• Constants
• func Commit(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, ...) (string, error)
• func LegacyMultiRetrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, ...) error
• func Materialize(ctx context.Context, endpoint string, ...) ([]*pipepb.ArtifactInformation, error)
• func MultiRetrieve(ctx context.Context, cpus int, list []retrievable, dest string) error
• func MultiStage(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, cpus int, ...) ([]*jobpb.ArtifactMetadata, error)
• func MustExtractFilePayload(artifact *pipepb.ArtifactInformation) (string, string)
• func Retrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, ...) error
• func Stage(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, ...) (*jobpb.ArtifactMetadata, error)
• func StageDir(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, ...) ([]*jobpb.ArtifactMetadata, error)
• type KeyedFile

Constants

const (
	URNFileArtifact        = "beam:artifact:type:file:v1"
	URNPipRequirementsFile = "beam:artifact:role:pip_requirements_file:v1"
	URNStagingTo           = "beam:artifact:role:staging_to:v1"
	NoArtifactsStaged      = "__no_artifacts_staged__"
)

TODO(lostluck): 2018/05/28 Extract these from their enum descriptors in the pipeline_v1 proto

Functions

func Commit

func Commit(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, artifacts []*jobpb.ArtifactMetadata, st string) (string, error)

Commit commits a manifest with the given staged artifacts. It returns the staging token, if successful.

func LegacyMultiRetrieve

func LegacyMultiRetrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, cpus int, list []*jobpb.ArtifactMetadata, rt string, dest string) error

LegacyMultiRetrieve is exported for testing.

func Materialize

func Materialize(ctx context.Context, endpoint string, dependencies []*pipepb.ArtifactInformation, rt string, dest string) ([]*pipepb.ArtifactInformation, error)

Materialize is a convenience helper for ensuring that all artifacts are present and uncorrupted. It interprets each artifact name as a relative path under the dest directory. It does not retrieve valid artifacts already present. TODO(BEAM-9577): Return a mapping of filename to dependency, rather than []*jobpb.ArtifactMetadata. TODO(BEAM-9577): Leverage richness of roles rather than magic names to understand artifacts.

func MultiRetrieve

func MultiRetrieve(ctx context.Context, cpus int, list []retrievable, dest string) error

MultiRetrieve retrieves multiple artifacts concurrently, using at most 'cpus' goroutines. It retries each artifact a few times. Convenience wrapper.

func MultiStage

func MultiStage(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, cpus int, list []KeyedFile, st string) ([]*jobpb.ArtifactMetadata, error)

MultiStage stages a set of local files with the given keys. It returns the full artifact metadate. It retries each artifact a few times. Convenience wrapper.

func MustExtractFilePayload

func MustExtractFilePayload(artifact *pipepb.ArtifactInformation) (string, string)

func Retrieve

func Retrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, a *jobpb.ArtifactMetadata, rt string, dest string) error

Retrieve checks whether the given artifact is already successfully retrieved. If not, it retrieves into the dest directory. It overwrites any previous retrieval attempt and may leave a corrupt/partial local file on failure.

func Stage

func Stage(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, key, filename, st string) (*jobpb.ArtifactMetadata, error)

Stage stages a local file as an artifact with the given key. It computes the SHA256 and returns the full artifact metadata.

func StageDir

func StageDir(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, src string, st string) ([]*jobpb.ArtifactMetadata, error)

StageDir stages a local directory with relative path keys. Convenience wrapper.

Types

type KeyedFile

type KeyedFile struct {
	Key, Filename string
}

KeyedFile is a key and filename pair.