processing

Documentation

Overview

Package processing contains code related to post-registration instance processing.

Index

• Constants
• func GetClientBinaryName(pkg string) string
• func GetClientPackage(platform string) (string, error)
• func IsClientPackage(pkg string) bool
• type BootstrapExtractorResult
  • func GetBootstrapExtractorResult(ctx context.Context, inst *model.Instance) (*BootstrapExtractorResult, error)
• type BootstrapPackageExtractor
  • func (bs *BootstrapPackageExtractor) Applicable(ctx context.Context, inst *model.Instance) (bool, error)
  • func (bs *BootstrapPackageExtractor) ID() string
  • func (bs *BootstrapPackageExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)
• type ClientExtractor
  • func (e *ClientExtractor) Applicable(ctx context.Context, inst *model.Instance) (bool, error)
  • func (e *ClientExtractor) ID() string
  • func (e *ClientExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)
• type ClientExtractorResult
  • func GetClientExtractorResult(ctx context.Context, inst *api.Instance) (*ClientExtractorResult, error)
  • func (r *ClientExtractorResult) ObjectRefAliases() []*api.ObjectRef
  • func (r *ClientExtractorResult) ToObjectRef() (*api.ObjectRef, error)
• type ExtractionResult
• type Extractor
  • func (ex *Extractor) Run(ctx context.Context, path string) (*ExtractionResult, error)
• type PackageReader
  • func NewPackageReader(r io.ReaderAt, size int64) (*PackageReader, error)
  • func (p *PackageReader) Files() []string
  • func (p *PackageReader) Open(path string) (io.ReadCloser, int64, error)
• type Processor
• type Result

Constants

const BootstrapPackageExtractorProcID = "bootstrap_extractor:v1"

BootstrapPackageExtractorProcID is identifier of BootstrapPackageExtractor.

const ClientExtractorProcID = "cipd_client_binary:v1"

ClientExtractorProcID is identifier of ClientExtractor processor.

Functions

func GetClientBinaryName

func GetClientBinaryName(pkg string) string

GetClientBinaryName returns name of CIPD binary inside the package.

Either 'cipd' or 'cipd.exe'.

func GetClientPackage

func GetClientPackage(platform string) (string, error)

GetClientPackage returns the name of the client package for CIPD client for the given platform.

Returns an error if the platform name is invalid.

func IsClientPackage

func IsClientPackage(pkg string) bool

IsClientPackage returns true if the given package stores a CIPD client.

Types

type BootstrapExtractorResult

type BootstrapExtractorResult struct {
	File       string // the name of the extracted file
	HashAlgo   string // the hash algorithm used to calculate its hash for CAS
	HashDigest string // its hex digest in the CAS
	Size       int64  // the size of the extracted file
}

BootstrapExtractorResult is stored as JSON in model.ProcessingResult.

func GetBootstrapExtractorResult

func GetBootstrapExtractorResult(ctx context.Context, inst *model.Instance) (*BootstrapExtractorResult, error)

GetBootstrapExtractorResult returns results of BootstrapPackageExtractor.

Returns:

(result, nil) on success.
(nil, datastore.ErrNoSuchEntity) if results are not available.
(nil, transient-tagged error) on retrieval errors.
(nil, non-transient-tagged error) if the extractor failed.

type BootstrapPackageExtractor

type BootstrapPackageExtractor struct {
	CAS cas.StorageServer
	// contains filtered or unexported fields
}

BootstrapPackageExtractor is a processor that extracts files from bootstrap packages (per bootstrap.cfg service config).

func (*BootstrapPackageExtractor) Applicable

func (bs *BootstrapPackageExtractor) Applicable(ctx context.Context, inst *model.Instance) (bool, error)

Applicable is a part of the Processor interface.

func (*BootstrapPackageExtractor) ID

func (bs *BootstrapPackageExtractor) ID() string

ID is a part of the Processor interface.

func (*BootstrapPackageExtractor) Run

func (bs *BootstrapPackageExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)

Run is a part of the Processor interface.

type ClientExtractor

type ClientExtractor struct {
	CAS cas.StorageServer
	// contains filtered or unexported fields
}

ClientExtractor is a processor that extracts CIPD client binary from CIPD client packages (infra/tools/cipd/...) and stores it in the CAS, so it can be fetched directly.

This is needed to support CIPD client bootstrap using e.g. 'curl'.

func (*ClientExtractor) Applicable

func (e *ClientExtractor) Applicable(ctx context.Context, inst *model.Instance) (bool, error)

Applicable is part of Processor interface.

func (*ClientExtractor) ID

func (e *ClientExtractor) ID() string

ID is part of Processor interface.

func (*ClientExtractor) Run

func (e *ClientExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)

Run is part of Processor interface.

type ClientExtractorResult

type ClientExtractorResult struct {
	ClientBinary struct {
		Size int64 `json:"size"`

		// Algo used to name the extracted file, matches the client package algo.
		HashAlgo   string `json:"hash_algo"`   // cas.HashAlgo enum serialized to string
		HashDigest string `json:"hash_digest"` // as hex string

		// AllHashDigests are hex digests of the extracted file calculated using all
		// algos known to the server at the time the file was uploaded.
		//
		// Keys are cas.HashAlgo enum values as strings ('SHA1', 'SHA256', ...).
		//
		// If empty (for old records), only supported algo is HashAlgo from above
		// (which for old records is always SHA1).
		AllHashDigests map[string]string `json:"all_hash_digests"`
	} `json:"client_binary"`
}

ClientExtractorResult is stored in JSON form as a result of ClientExtractor execution.

Compatible with Python version of the backend.

If format of this struct changes in a non backward compatible way, the version number in ClientExtractorProcID should change too.

func GetClientExtractorResult

func GetClientExtractorResult(ctx context.Context, inst *api.Instance) (*ClientExtractorResult, error)

GetClientExtractorResult returns results of client extractor processor.

They contain a reference to the unpacked CIPD binary object in the Google Storage.

Returns:

(result, nil) on success.
(nil, datastore.ErrNoSuchEntity) if results are not available.
(nil, transient-tagged error) on retrieval errors.
(nil, non-transient-tagged error) if the client extractor failed.

func (*ClientExtractorResult) ObjectRefAliases

func (r *ClientExtractorResult) ObjectRefAliases() []*api.ObjectRef

ObjectRefAliases is list of ObjectRefs calculated using all hash algos known to the server when the client binary was extracted.

Additionally all algos not understood by the server right NOW are skipped too. This may arise if the server was rolled back, but some files have already been uploaded with a newer algo.

func (*ClientExtractorResult) ToObjectRef

func (r *ClientExtractorResult) ToObjectRef() (*api.ObjectRef, error)

ToObjectRef returns a reference to the extracted client binary in CAS.

The returned ObjectRef is validated to be syntactically correct already.

type ExtractionResult

type ExtractionResult struct {
	Path   string                     // the file path passed to Run
	Ref    *api.ObjectRef             // reference to the extracted file in the CAS
	Size   int64                      // the size of the file in bytes
	Hashes map[api.HashAlgo]hash.Hash // all calculated hashes
}

ExtractionResult is a result of a successful file extraction.

type Extractor

type Extractor struct {
	// Reader is an already open package file.
	Reader *PackageReader

	// CAS is the destination CAS implementation.
	CAS cas.StorageServer

	// PrimaryHash is the hash algorithm to use to name the file in the CAS.
	PrimaryHash api.HashAlgo

	// AlternativeHashes is a list of hashes to calculate in addition to
	// the PrimaryHash.
	AlternativeHashes []api.HashAlgo

	// Uploader returns io.Writer that uploads to the given destination URL.
	//
	// If nil, will use a Google Storage uploader. Useful in tests.
	Uploader func(ctx context.Context, size int64, uploadURL string) io.Writer

	// BufferSize is size of the buffer for GS uploads (default is 2 Mb).
	BufferSize int
}

Extractor can extract files from the package, writing them to the CAS.

func (*Extractor) Run

func (ex *Extractor) Run(ctx context.Context, path string) (*ExtractionResult, error)

Run extracts a single file from the package.

type PackageReader

type PackageReader struct {
	// contains filtered or unexported fields
}

PackageReader knows how to extract files from CIPD packages.

CIPD packages are actually zip archives, but we don't want to expose it everywhere.

func NewPackageReader

func NewPackageReader(r io.ReaderAt, size int64) (*PackageReader, error)

NewPackageReader opens the package by reading its directory.

func (*PackageReader) Files

func (p *PackageReader) Files() []string

Files returns names of files inside the package.

func (*PackageReader) Open

func (p *PackageReader) Open(path string) (io.ReadCloser, int64, error)

Open opens some file inside the package for reading.

Returns the ReadCloser and the uncompressed file size.

type Processor

type Processor interface {
	// ID is unique identifier of this processor used store processing results in
	// the datastore.
	ID() string

	// Applicable returns true if this processor should be applied to an instance.
	//
	// Called as a part of the instance registration datastore transaction.
	// Returns an error if the decision can't be made. It will result in
	// a transient instance registration error.
	Applicable(ctx context.Context, inst *model.Instance) (bool, error)

	// Run executes the processing on the package instance.
	//
	// Returns either a result, or a transient error. All fatal errors should be
	// communicated through the result.
	//
	// Must be idempotent. The processor may be called multiple times when
	// retrying task queue tasks.
	Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (Result, error)
}

Processor runs some post-processing step on a package instance after it has been uploaded.

type Result

type Result struct {
	Result any   // JSON-serializable summary extracted by the processor
	Err    error // if non-nil, contains the fatal error message
}

Result is returned by processors.

It is either some JSON-serializable data or a fatal error.