processing

Documentation

Overview

Package processing contains code related to post-registration instance processing.

Index

• Constants
• func GetClientBinaryName(pkg string) string
• func GetClientPackage(platform string) (string, error)
• func IsClientPackage(pkg string) bool
• type ClientExtractor
  • func (e *ClientExtractor) Applicable(inst *model.Instance) bool
  • func (e *ClientExtractor) ID() string
  • func (e *ClientExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)
• type ClientExtractorResult
  • func GetClientExtractorResult(c context.Context, inst *api.Instance) (*ClientExtractorResult, error)
  • func (r *ClientExtractorResult) ObjectRefAliases() []*api.ObjectRef
  • func (r *ClientExtractorResult) ToObjectRef() (*api.ObjectRef, error)
• type PackageReader
  • func NewPackageReader(r io.ReaderAt, size int64) (*PackageReader, error)
  • func (p *PackageReader) Open(path string) (io.ReadCloser, int64, error)
• type Processor
• type Result

Constants

const ClientExtractorProcID = "cipd_client_binary:v1"

ClientExtractorProcID is identifier of ClientExtractor processor.

Functions

func GetClientBinaryName

func GetClientBinaryName(pkg string) string

GetClientBinaryName returns name of CIPD binary inside the package.

Either 'cipd' or 'cipd.exe'.

func GetClientPackage

func GetClientPackage(platform string) (string, error)

GetClientPackage returns the name of the client package for CIPD client for the given platform.

Returns an error if the platform name is invalid.

func IsClientPackage

func IsClientPackage(pkg string) bool

IsClientPackage returns true if the given package stores a CIPD client.

Types

type ClientExtractor

type ClientExtractor struct {
	CAS cas.StorageServer
	// contains filtered or unexported fields
}

ClientExtractor is a processor that extracts CIPD client binary from CIPD client packages (infra/tools/cipd/...) and stores it in the CAS, so it can be fetched directly.

This is needed to support CIPD client bootstrap using e.g. 'curl'.

func (*ClientExtractor) Applicable

func (e *ClientExtractor) Applicable(inst *model.Instance) bool

Applicable is part of Processor interface.

func (*ClientExtractor) ID

func (e *ClientExtractor) ID() string

ID is part of Processor interface.

func (*ClientExtractor) Run

func (e *ClientExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)

Run is part of Processor interface.

type ClientExtractorResult

type ClientExtractorResult struct {
	ClientBinary struct {
		Size int64 `json:"size"`

		// Algo used to name the extracted file, matches the client package algo.
		HashAlgo   string `json:"hash_algo"`   // cas.HashAlgo enum serialized to string
		HashDigest string `json:"hash_digest"` // as hex string

		// AllHashDigests are hex digests of the extracted file calculated using all
		// algos known to the server at the time the file was uploaded.
		//
		// Keys are cas.HashAlgo enum values as strings ('SHA1', 'SHA256', ...).
		//
		// If empty (for old records), only supported algo is HashAlgo from above
		// (which for old records is always SHA1).
		AllHashDigests map[string]string `json:"all_hash_digests"`
	} `json:"client_binary"`
}

ClientExtractorResult is stored in JSON form as a result of ClientExtractor execution.

Compatible with Python version of the backend.

If format of this struct changes in a non backward compatible way, the version number in ClientExtractorProcID should change too.

func GetClientExtractorResult

func GetClientExtractorResult(c context.Context, inst *api.Instance) (*ClientExtractorResult, error)

GetClientExtractorResult returns results of client extractor processor.

They contain a reference to the unpacked CIPD binary object in the Google Storage.

Returns:

(result, nil) on success.
(nil, datastore.ErrNoSuchEntity) if results are not available.
(nil, transient-tagged error) on retrieval errors.
(nil, non-transient-tagged error) if the client extractor failed.

func (*ClientExtractorResult) ObjectRefAliases

func (r *ClientExtractorResult) ObjectRefAliases() []*api.ObjectRef

ObjectRefAliases is list of ObjectRefs calculated using all hash algos known to the server when the client binary was extracted.

Additionally all algos not understood by the server right NOW are skipped too. This may arise if the server was rolled back, but some files have already been uploaded with a newer algo.

func (*ClientExtractorResult) ToObjectRef

func (r *ClientExtractorResult) ToObjectRef() (*api.ObjectRef, error)

ToObjectRef returns a reference to the extracted client binary in CAS.

The returned ObjectRef is validated to be syntactically correct already.

type PackageReader

type PackageReader struct {
	// contains filtered or unexported fields
}

PackageReader knows how to extract files from CIPD packages.

CIPD packages are actually zip archives, but we don't want to expose it everywhere.

func NewPackageReader

func NewPackageReader(r io.ReaderAt, size int64) (*PackageReader, error)

NewPackageReader opens the package by reading its directory.

func (*PackageReader) Open

func (p *PackageReader) Open(path string) (io.ReadCloser, int64, error)

Open opens some file inside the package for reading.

Returns the ReadCloser and the uncompressed file size.

type Processor

type Processor interface {
	// ID is unique identifier of this processor used store processing results in
	// the datastore.
	ID() string

	// Applicable returns true if this processor should be applied to an instance.
	Applicable(inst *model.Instance) bool

	// Run executes the processing on the package instance.
	//
	// Returns either a result, or a transient error. All fatal errors should be
	// communicated through the result.
	//
	// Must be idempotent. The processor may be called multiple times when
	// retrying task queue tasks.
	Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (Result, error)
}

Processor runs some post-processing step on a package instance after it has been uploaded.

type Result

type Result struct {
	Result interface{} // JSON-serializable summary extracted by the processor
	Err    error       // if non-nil, contains the fatal error message
}

Result is returned by processors.

It is either some JSON-serializable data or a fatal error.