gcs

Documentation

Overview

Package gcs handles uploading files and raw data to blob storage and determines where in the GCS bucket data should go given a specific job specification

Index

• Constants
• func AliasForSpec(spec *downwardapi.JobSpec) string
• func LatestBuildForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) []string
• func LocalExport(ctx context.Context, exportDir string, uploadTargets map[string]UploadFunc) error
• func PathForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) string
• func RootForSpec(spec *downwardapi.JobSpec) string
• func Upload(ctx context.Context, bucket, gcsCredentialsFile, s3CredentialsFile string, ...) error
• func WriterOptionsFromFileName(filename string) (string, io.WriterOptions)
• type ReaderFunc
• type RepoPathBuilder
  • func NewExplicitRepoPathBuilder() RepoPathBuilder
  • func NewLegacyRepoPathBuilder(defaultOrg, defaultRepo string) RepoPathBuilder
  • func NewSingleDefaultRepoPathBuilder(defaultOrg, defaultRepo string) RepoPathBuilder
• type UploadFunc
  • func DataUpload(newReader ReaderFunc) UploadFunc
  • func DataUploadWithMetadata(newReader ReaderFunc, metadata map[string]string) UploadFunc
  • func DataUploadWithOptions(newReader ReaderFunc, attrs pkgio.WriterOptions) UploadFunc
  • func FileUpload(file string) UploadFunc
  • func FileUploadWithOptions(file string, opts pkgio.WriterOptions) UploadFunc

Constants

const (
	// PRLogs is the name of the directory we put PR logs in.
	PRLogs = "pr-logs"
	// NonPRLogs is the name of directory we put logs that are not PR logs in.
	NonPRLogs = "logs"
)

Functions

func AliasForSpec

func AliasForSpec(spec *downwardapi.JobSpec) string

AliasForSpec determines the GCS path aliases for a job spec

func LatestBuildForSpec

func LatestBuildForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) []string

LatestBuildForSpec determines the GCS path for storing the latest build id for a job. pathSegment can be nil so callers of this helper are not required to choose a path strategy but can still get back a result.

func LocalExport

func LocalExport(ctx context.Context, exportDir string, uploadTargets map[string]UploadFunc) error

LocalExport copies all of the data in the uploadTargets map to local files in parallel. The map is keyed on file path under the exportDir.

func PathForSpec

func PathForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) string

PathForSpec determines the GCS path prefix for files uploaded for a specific job spec

func RootForSpec

func RootForSpec(spec *downwardapi.JobSpec) string

RootForSpec determines the root GCS path for storing artifacts about the provided job.

func Upload

func Upload(ctx context.Context, bucket, gcsCredentialsFile, s3CredentialsFile string, compressFileTypes []string, uploadTargets map[string]UploadFunc) error

Upload uploads all the data in the uploadTargets map to blob storage in parallel. The map is keyed on blob storage path under the bucket. Files with an extension in the compressFileTypes list will be compressed prior to uploading

func WriterOptionsFromFileName

func WriterOptionsFromFileName(filename string) (string, io.WriterOptions)

WriterOptionsFromFileName guesses file attributes from the filename and returns the writerOptions and a simplified filename. For example, build-log.txt.gz would be:

Content-Type: text/plain; charset=utf-8
Content-Encoding: gzip

and the simplified filename would be build-log.txt (excluding the content encoding extension).

Types

type ReaderFunc

type ReaderFunc func() (io.ReadCloser, error)

type RepoPathBuilder

type RepoPathBuilder func(org, repo string) string

RepoPathBuilder builds GCS path segments and embeds defaulting behavior

func NewExplicitRepoPathBuilder

func NewExplicitRepoPathBuilder() RepoPathBuilder

NewExplicitRepoPathBuilder returns a builder that handles the path encoding where a path will always have an explicit "org_repo" path segment

func NewLegacyRepoPathBuilder

func NewLegacyRepoPathBuilder(defaultOrg, defaultRepo string) RepoPathBuilder

NewLegacyRepoPathBuilder returns a builder that handles the legacy path encoding where a path will only contain an org or repo if they are non-default

func NewSingleDefaultRepoPathBuilder

func NewSingleDefaultRepoPathBuilder(defaultOrg, defaultRepo string) RepoPathBuilder

NewSingleDefaultRepoPathBuilder returns a builder that handles the legacy path encoding where a path will contain org and repo for all but one default repo

type UploadFunc

type UploadFunc func(writer dataWriter) error

UploadFunc knows how to upload into an object

func DataUpload

func DataUpload(newReader ReaderFunc) UploadFunc

DataUpload returns an UploadFunc which copies all data from src reader into GCS.

func DataUploadWithMetadata

func DataUploadWithMetadata(newReader ReaderFunc, metadata map[string]string) UploadFunc

DataUploadWithMetadata returns an UploadFunc which copies all data from src reader into GCS and also sets the provided metadata fields onto the object.

func DataUploadWithOptions

func DataUploadWithOptions(newReader ReaderFunc, attrs pkgio.WriterOptions) UploadFunc

DataUploadWithOptions returns an UploadFunc which copies all data from src reader into GCS and also sets the provided attributes on the object.

func FileUpload

func FileUpload(file string) UploadFunc

FileUpload returns an UploadFunc which copies all data from the file on disk to the GCS object

func FileUploadWithOptions

func FileUploadWithOptions(file string, opts pkgio.WriterOptions) UploadFunc

FileUploadWithOptions returns an UploadFunc which copies all data from the file on disk into GCS object and also sets the provided attributes on the object.