Documentation ¶
Overview ¶
Package processing contains code related to post-registration instance processing.
Index ¶
- Constants
- func GetClientBinaryName(pkg string) string
- func GetClientPackage(platform string) (string, error)
- func IsClientPackage(pkg string) bool
- type BootstrapExtractorResult
- type BootstrapPackageExtractor
- type ClientExtractor
- type ClientExtractorResult
- type ExtractionResult
- type Extractor
- type PackageReader
- type Processor
- type Result
Constants ¶
const BootstrapPackageExtractorProcID = "bootstrap_extractor:v1"
BootstrapPackageExtractorProcID is identifier of BootstrapPackageExtractor.
const ClientExtractorProcID = "cipd_client_binary:v1"
ClientExtractorProcID is identifier of ClientExtractor processor.
Variables ¶
This section is empty.
Functions ¶
func GetClientBinaryName ¶
GetClientBinaryName returns name of CIPD binary inside the package.
Either 'cipd' or 'cipd.exe'.
func GetClientPackage ¶
GetClientPackage returns the name of the client package for CIPD client for the given platform.
Returns an error if the platform name is invalid.
func IsClientPackage ¶
IsClientPackage returns true if the given package stores a CIPD client.
Types ¶
type BootstrapExtractorResult ¶
type BootstrapExtractorResult struct { File string // the name of the extracted file HashAlgo string // the hash algorithm used to calculate its hash for CAS HashDigest string // its hex digest in the CAS Size int64 // the size of the extracted file }
BootstrapExtractorResult is stored as JSON in model.ProcessingResult.
func GetBootstrapExtractorResult ¶
func GetBootstrapExtractorResult(ctx context.Context, inst *model.Instance) (*BootstrapExtractorResult, error)
GetBootstrapExtractorResult returns results of BootstrapPackageExtractor.
Returns:
(result, nil) on success. (nil, datastore.ErrNoSuchEntity) if results are not available. (nil, transient-tagged error) on retrieval errors. (nil, non-transient-tagged error) if the extractor failed.
type BootstrapPackageExtractor ¶
type BootstrapPackageExtractor struct { CAS cas.StorageServer // contains filtered or unexported fields }
BootstrapPackageExtractor is a processor that extracts files from bootstrap packages (per bootstrap.cfg service config).
func (*BootstrapPackageExtractor) Applicable ¶
func (bs *BootstrapPackageExtractor) Applicable(ctx context.Context, inst *model.Instance) (bool, error)
Applicable is a part of the Processor interface.
func (*BootstrapPackageExtractor) ID ¶
func (bs *BootstrapPackageExtractor) ID() string
ID is a part of the Processor interface.
func (*BootstrapPackageExtractor) Run ¶
func (bs *BootstrapPackageExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)
Run is a part of the Processor interface.
type ClientExtractor ¶
type ClientExtractor struct { CAS cas.StorageServer // contains filtered or unexported fields }
ClientExtractor is a processor that extracts CIPD client binary from CIPD client packages (infra/tools/cipd/...) and stores it in the CAS, so it can be fetched directly.
This is needed to support CIPD client bootstrap using e.g. 'curl'.
func (*ClientExtractor) Applicable ¶
Applicable is part of Processor interface.
func (*ClientExtractor) ID ¶
func (e *ClientExtractor) ID() string
ID is part of Processor interface.
func (*ClientExtractor) Run ¶
func (e *ClientExtractor) Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (res Result, err error)
Run is part of Processor interface.
type ClientExtractorResult ¶
type ClientExtractorResult struct { ClientBinary struct { Size int64 `json:"size"` // Algo used to name the extracted file, matches the client package algo. HashAlgo string `json:"hash_algo"` // cas.HashAlgo enum serialized to string HashDigest string `json:"hash_digest"` // as hex string // AllHashDigests are hex digests of the extracted file calculated using all // algos known to the server at the time the file was uploaded. // // Keys are cas.HashAlgo enum values as strings ('SHA1', 'SHA256', ...). // // If empty (for old records), only supported algo is HashAlgo from above // (which for old records is always SHA1). AllHashDigests map[string]string `json:"all_hash_digests"` } `json:"client_binary"` }
ClientExtractorResult is stored in JSON form as a result of ClientExtractor execution.
Compatible with Python version of the backend.
If format of this struct changes in a non backward compatible way, the version number in ClientExtractorProcID should change too.
func GetClientExtractorResult ¶
func GetClientExtractorResult(ctx context.Context, inst *api.Instance) (*ClientExtractorResult, error)
GetClientExtractorResult returns results of client extractor processor.
They contain a reference to the unpacked CIPD binary object in the Google Storage.
Returns:
(result, nil) on success. (nil, datastore.ErrNoSuchEntity) if results are not available. (nil, transient-tagged error) on retrieval errors. (nil, non-transient-tagged error) if the client extractor failed.
func (*ClientExtractorResult) ObjectRefAliases ¶
func (r *ClientExtractorResult) ObjectRefAliases() []*api.ObjectRef
ObjectRefAliases is list of ObjectRefs calculated using all hash algos known to the server when the client binary was extracted.
Additionally all algos not understood by the server right NOW are skipped too. This may arise if the server was rolled back, but some files have already been uploaded with a newer algo.
func (*ClientExtractorResult) ToObjectRef ¶
func (r *ClientExtractorResult) ToObjectRef() (*api.ObjectRef, error)
ToObjectRef returns a reference to the extracted client binary in CAS.
The returned ObjectRef is validated to be syntactically correct already.
type ExtractionResult ¶
type ExtractionResult struct { Path string // the file path passed to Run Ref *api.ObjectRef // reference to the extracted file in the CAS Size int64 // the size of the file in bytes Hashes map[api.HashAlgo]hash.Hash // all calculated hashes }
ExtractionResult is a result of a successful file extraction.
type Extractor ¶
type Extractor struct { // Reader is an already open package file. Reader *PackageReader // CAS is the destination CAS implementation. CAS cas.StorageServer // PrimaryHash is the hash algorithm to use to name the file in the CAS. PrimaryHash api.HashAlgo // AlternativeHashes is a list of hashes to calculate in addition to // the PrimaryHash. AlternativeHashes []api.HashAlgo // Uploader returns io.Writer that uploads to the given destination URL. // // If nil, will use a Google Storage uploader. Useful in tests. Uploader func(ctx context.Context, size int64, uploadURL string) io.Writer // BufferSize is size of the buffer for GS uploads (default is 2 Mb). BufferSize int }
Extractor can extract files from the package, writing them to the CAS.
type PackageReader ¶
type PackageReader struct {
// contains filtered or unexported fields
}
PackageReader knows how to extract files from CIPD packages.
CIPD packages are actually zip archives, but we don't want to expose it everywhere.
func NewPackageReader ¶
func NewPackageReader(r io.ReaderAt, size int64) (*PackageReader, error)
NewPackageReader opens the package by reading its directory.
func (*PackageReader) Files ¶
func (p *PackageReader) Files() []string
Files returns names of files inside the package.
func (*PackageReader) Open ¶
func (p *PackageReader) Open(path string) (io.ReadCloser, int64, error)
Open opens some file inside the package for reading.
Returns the ReadCloser and the uncompressed file size.
type Processor ¶
type Processor interface { // ID is unique identifier of this processor used store processing results in // the datastore. ID() string // Applicable returns true if this processor should be applied to an instance. // // Called as a part of the instance registration datastore transaction. // Returns an error if the decision can't be made. It will result in // a transient instance registration error. Applicable(ctx context.Context, inst *model.Instance) (bool, error) // Run executes the processing on the package instance. // // Returns either a result, or a transient error. All fatal errors should be // communicated through the result. // // Must be idempotent. The processor may be called multiple times when // retrying task queue tasks. Run(ctx context.Context, inst *model.Instance, pkg *PackageReader) (Result, error) }
Processor runs some post-processing step on a package instance after it has been uploaded.