inventory

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 21, 2020 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// Add represents those tags that are freely promotable, without fear of an
	// overwrite (we are only adding tags).
	Add TagOp = iota
	// Move represents those tags that conflict with existing digests, and so
	// should be moved to re-point to the digest that we want to promote as
	// defined in the manifest. It can be thought of a Delete followed by an
	// Add.
	Move = iota
	// Delete represents those tags that are not in the manifest and should thus
	// be removed and deleted. This is a kind of "demotion".
	Delete = iota
)
View Source 
const (
	// ThinManifestDepth specifies the number of items in a path if we split the
	// path into its parts, starting from the "topmost" folder given as an
	// argument to -thin-manifest-dir. E.g., a well-formed path is something
	// like:
	//
	//  ["", "manifests", "foo", "promoter-manifests.yaml"]
	//
	// . This is a result of some path handling/parsing logic in
	// ValidateThinManifestDirectoryStructure().
	ThinManifestDepth = 4
)

Variables

This section is empty.

Functions

func BytesToMB 

func BytesToMB(value int) int

BytesToMB converts a value from Bytes to MiB.

func CheckOverlappingEdges 

func CheckOverlappingEdges(
	edges map[PromotionEdge]interface{}) (map[PromotionEdge]interface{}, error)

CheckOverlappingEdges checks to ensure that all the edges taken together as a whole are consistent. It checks that there are no duplicate promotions desired to the same destination vertex (same destination PQIN). If the digests are the same for those edges, ignore because by definition the digests are cryptographically guaranteed to be the same thing (it doesn't matter if 2 different parties want the same exact image to become promoted to the same destination --- in many ways this is actually a good thing because it's a form of redundancy). However, return an error if the digests are different, because most likely this is at best just human error and at worst a malicious attack (someone trying to push an image to an endpoint they shouldn't own).

nolint[gocyclo]

func GetDeleteCmd 

func GetDeleteCmd(
	rc RegistryContext,
	useServiceAccount bool,
	img ImageName,
	digest Digest,
	force bool) []string

GetDeleteCmd generates the cloud command used to delete images (used for garbage collection).

func GetTokenKeyDomainRepoPath 

func GetTokenKeyDomainRepoPath(
	registryName RegistryName) (string, string, string)

GetTokenKeyDomainRepoPath splits a string by '/'. It's OK to do this because the RegistryName is already parsed against a Regex. (Maybe we should store the repo path separately when we do the initial parse...).

func GetWriteCmd 

func GetWriteCmd(
	dest RegistryContext,
	useServiceAccount bool,
	srcRegistry RegistryName,
	srcImageName ImageName,
	destImageName ImageName,
	digest Digest,
	tag Tag,
	tp TagOp) []string

GetWriteCmd generates a gcloud command that is used to make modifications to a Docker Registry.

func GrowManifest 

func GrowManifest(
	ctx context.Context,
	o GrowManifestOptions,
) error

GrowManifest modifies a manifest by adding images into it.

func IsSevereOccurrence 

func IsSevereOccurrence(
	vuln *grafeaspb.VulnerabilityOccurrence,
	severityThreshold int,
) bool

IsSevereOccurrence checks if a vulnerability is a high enough severity to fail the ImageVulnCheck.

func MBToBytes 

func MBToBytes(value int) int

MBToBytes converts a value from MiB to Bytes.

func MkReadManifestListCmdReal 

func MkReadManifestListCmdReal(
	sc *SyncContext,
	gmlc GCRManifestListContext) stream.Producer

MkReadManifestListCmdReal creates a stream.Producer which makes a real call over the network to read ManifestList information.

TODO: Consider replacing stream.Producer return type with a simple ([]byte, error) tuple instead.

func MkReadRepositoryCmdReal 

func MkReadRepositoryCmdReal(
	sc *SyncContext,
	rc RegistryContext) stream.Producer

MkReadRepositoryCmdReal creates a stream.Producer which makes a real call over the network.

func ParseContainerParts 

func ParseContainerParts(s string) (string, string, error)

ParseContainerParts splits up a registry name into its component pieces. Unfortunately it has some specialized logic around particular inputs; this could be removed in a future promoter manifest version which could force the user to provide these delineations for us.

nolint[gocyclo]

func SplitByKnownRegistries 

func SplitByKnownRegistries(
	r RegistryName,
	rcs []RegistryContext) (RegistryName, ImageName, error)

SplitByKnownRegistries splits a registry name into a RegistryName and ImageName. The purpose of this function is to split a long image path into 2 pieces --- the repository and the image name. We can't just split by the last "/" all the time, because some manifests have an image with a "/" in it.

func SplitRegistryImagePath 

func SplitRegistryImagePath(
	registryImagePath RegistryImagePath,
	knownRegistries []RegistryName) (RegistryName, ImageName, error)

SplitRegistryImagePath takes an arbitrary image path, and splits it into its component parts, according to the knownRegistries field. E.g., consider "gcr.io/foo/a/b/c" as the registryImagePath. If "gcr.io/foo" is in knownRegistries, then we split it into "gcr.io/foo" and "a/b/c". But if we were given "gcr.io/foo/a", we would split it into "gcr.io/foo/a" and "b/c".

func ToFQIN 

func ToFQIN(registryName RegistryName,
	imageName ImageName,
	digest Digest) string

ToFQIN combines a RegistryName, ImageName, and Digest to form a fully-qualified image name (FQIN).

func ToLQIN 

func ToLQIN(registryName RegistryName, imageName ImageName) string

ToLQIN converts a RegistryName and ImangeName to form a loosely-qualified image name (LQIN). Notice that it is missing tag information --- hence "loosely-qualified".

func ToPQIN 

func ToPQIN(registryName RegistryName, imageName ImageName, tag Tag) string

ToPQIN converts a RegistryName, ImageName, and Tag to form a partially-qualified image name (PQIN). It's less exact than a FQIN because the digest information is not used.

func ToPromotionEdges 

func ToPromotionEdges(
	mfests []Manifest) (map[PromotionEdge]interface{}, error)

ToPromotionEdges converts a list of manifests to a set of edges we want to try promoting.

func ValidateDigest 

func ValidateDigest(digest Digest) error

ValidateDigest validates the digest.

func ValidateRegistryImagePath 

func ValidateRegistryImagePath(rip RegistryImagePath) error

ValidateRegistryImagePath validates the RegistryImagePath.

func ValidateTag 

func ValidateTag(tag Tag) error

ValidateTag validates the tag.

func ValidateThinManifestDirectoryStructure 

func ValidateThinManifestDirectoryStructure(
	dir string,
) error

ValidateThinManifestDirectoryStructure enforces a particular directory structure for thin manifests. Most importantly, it requires that if a file named "foo/manifests/bar/promoter-manifest.yaml" exists, that a corresponding file named "foo/images/bar/promoter-manifest.yaml" must also exist.

nolint[gocyclo]

func WriteImages 

func WriteImages(manifest Manifest, rii RegInvImage) error

WriteImages writes images as YAML out to the expected path of the given (thin) manifest.

Types

type CapturedRequests 

type CapturedRequests map[PromotionRequest]int

CapturedRequests holds a map of all PromotionRequests that were generated. It is used for both -dry-run and testing.

type CollectedLogs 

type CollectedLogs struct {
	Errors Errors
}

CollectedLogs holds all the Errors that are generated as the promoter runs.

type Digest 

type Digest string

Digest is a string that contains the SHA256 hash of a Docker container image.

type DigestImageSize 

type DigestImageSize map[Digest]int

DigestImageSize holds information about the size of an image in bytes.

type DigestMediaType 

type DigestMediaType map[Digest]cr.MediaType

DigestMediaType holds media information about a Digest.

type DigestTags 

type DigestTags map[Digest]TagSlice

DigestTags is a map where each digest is associated with a TagSlice. It is associated with a TagSlice because an image digest can have more than 1 tag pointing to it, even within the same image name's namespace (tags are namespaced by the image name).

func (DigestTags) Overwrite 

func (a DigestTags) Overwrite(b DigestTags)

Overwrite insert's b's values into a.

type DigestTagsContext 

type DigestTagsContext struct {
	ImageName    ImageName
	RegistryName RegistryName
}

DigestTagsContext holds information about the request that was used to fetch the list of digests and associated tags for a particular image. It is used in ReadDigestsAndTags().

type Error 

type Error struct {
	Context string
	Error   error
}

Error contains slightly more verbosity than a standard "error".

type Errors 

type Errors []Error

Errors is a slice of Errors.

type GCRManifestListContext 

type GCRManifestListContext struct {
	RegistryContext RegistryContext
	ImageName       ImageName
	Tag             Tag
	Digest          Digest
}

GCRManifestListContext is used only for reading GCRManifestList information from GCR, in the function ReadGCRManifestLists.

type GCRPubSubPayload 

type GCRPubSubPayload struct {
	Action string `json:"action"`
	// The payload field is "digest", but really it is a FQIN like
	//  "gcr.io/linusa/small@sha256:35f442d8d56cc7a2d4000f3417d71f44a730b900f3df440c09a9c40c42c40f86".
	FQIN string `json:"digest,omitempty"`
	// Similarly, the field "tag is always a PQIN. E.g.,
	// "gcr.io/linusa/small:a".
	PQIN string `json:"tag,omitempty"`

	// Everything leading up to either the tag or digest. E.g., given
	// "us.gcr.io/k8s-artifacts-prod/foo/bar:1.0", this would be
	// "us.gcr.io/k8s-artifacts-prod/foo/bar".
	Path string
	// Image digest, if any.
	Digest Digest
	// Tag, if any.
	Tag Tag
}

GCRPubSubPayload is the message payload sent to a Pub/Sub topic by a GCR. nolint[lll]

func (GCRPubSubPayload) Match 

func (payload GCRPubSubPayload) Match(manifest Manifest) GcrPayloadMatch

Match checks whether a GCRPubSubPayload is mentioned in a Manifest. The degree of the match is reflected in the GcrPayloadMatch result.

func (*GCRPubSubPayload) PopulateExtraFields 

func (payload *GCRPubSubPayload) PopulateExtraFields() error

PopulateExtraFields takes the existing fields in GCRPubSubPayload and uses them to populate the extra convenience fields (these fields are derived from the FQIN and PQIN fields, and do not add any new information of their own). This is because the payload bundles up digests, tags, etc into a single string. Instead of dealing with them later on, we just break them up into the pieces we would like to use.

func (GCRPubSubPayload) String 

func (payload GCRPubSubPayload) String() string

Prettified prints the payload in a way that is stable and which hides extra fields which are redundant.

type GcrPayloadMatch 

type GcrPayloadMatch struct {
	// Path is true if the registry + image path (everything leading
	// up to either the digest or a tag) matches.
	PathMatch bool
	// Digest is set if the digest in the payload matches a digest in
	// the promoter manifest. This is ONLY matched if the path also matches.
	DigestMatch bool
	// Tag is ONLY matched if the digest also matches.
	TagMatch bool
	// Tag is only true if the digest matches, but the tag found in
	// the payload does NOT match what is found in the promoter manifest for the
	// digest. This can happen if somone manually tweaks a tag in GCR (assume
	// bad actor) to something other than what is specified in the promoter
	// manifest.
	TagMismatch bool
}

GcrPayloadMatch holds booleans for matching a GCRPubSubPayload against a promoter manifest.

type GrowManifestOptions 

type GrowManifestOptions struct {
	// BaseDir is the directory containing the thin promoter manifests.
	BaseDir string
	// StagingRepo is the staging subproject repo to read from. If no filters
	// are provided, all images are attempted to be promoted as-is without any
	// modifications.
	StagingRepo RegistryName
	// FilterImage is the image (name) to filter by. Optional.
	FilterImage ImageName
	// FilterDigest is the image digest to filter by. Optional.
	FilterDigest Digest
	// FilterTag is the image tag to filter by. Optional.
	FilterTag Tag
}

GrowManifestOptions holds the parameters for modifying manifests.

func (*GrowManifestOptions) Populate 

func (o *GrowManifestOptions) Populate(
	baseDir,
	stagingRepo,
	filterImage,
	filterDigest,
	filterTag string,
) error

Populate sets the values for GrowManifestOptions.

func (*GrowManifestOptions) Validate 

func (o *GrowManifestOptions) Validate() error

Validate validates the options.

type Image 

type Image struct {
	ImageName ImageName  `yaml:"name"`
	Dmap      DigestTags `yaml:"dmap,omitempty"`
}

Image holds information about an image. It's like an "Object" in the OOP sense, and holds all the information relating to a particular image that we care about.

type ImageDigest 

type ImageDigest struct {
	ImageName ImageName
	Digest    Digest
}

ImageDigest is a combination of the ImageName and Digest.

type ImageDigestTag 

type ImageDigestTag struct {
	ImageName ImageName
	Digest    Digest
	Tag       Tag
}

ImageDigestTag is a flattened key used by RegInvFlat.

type ImageName 

type ImageName string

The ImageName can be just the bare name itself (e.g., "addon-builder" in "gcr.io/k8s-image-staging/addon-builder") or the prefix + name ("foo/bar/baz/quux" in "gcr.io/hello/foo/bar/baz/quux").

type ImageRemovalCheck 

type ImageRemovalCheck struct {
	GitRepoPath    string
	MasterSHA      plumbing.Hash
	PullRequestSHA plumbing.Hash
	PullEdges      map[PromotionEdge]interface{}
}

ImageRemovalCheck implements the PreCheck interface and checks against pull requests that attempt to remove any images from the promoter manifests.

func MKRealImageRemovalCheck 

func MKRealImageRemovalCheck(
	gitRepoPath string,
	edges map[PromotionEdge]interface{},
) (*ImageRemovalCheck, error)

MKRealImageRemovalCheck returns an instance of ImageRemovalCheck.

func (*ImageRemovalCheck) Compare 

func (check *ImageRemovalCheck) Compare(
	edgesMaster map[PromotionEdge]interface{},
	edgesPullRequest map[PromotionEdge]interface{},
) error

Compare is a function of the ImageRemovalCheck that handles the comparison of the pull requests's set of promotion edges and the master branch's set of promotion edges.

func (*ImageRemovalCheck) Run 

func (check *ImageRemovalCheck) Run() error

Run executes ImageRemovalCheck on a set of promotion edges. Returns an error if the pull request removes images from the promoter manifests.

type ImageSizeCheck 

type ImageSizeCheck struct {
	MaxImageSize    int
	DigestImageSize DigestImageSize
	PullEdges       map[PromotionEdge]interface{}
}

ImageSizeCheck implements the PreCheck interface and checks against images that are larger than a size threshold (controlled by the max-image-size flag).

func MKRealImageSizeCheck 

func MKRealImageSizeCheck(
	maxImageSize int,
	edges map[PromotionEdge]interface{},
	digestImageSize DigestImageSize,
) *ImageSizeCheck

MKRealImageSizeCheck returns an instance of ImageSizeCheck which checks that all images to be promoted are under a max size.

func (*ImageSizeCheck) Run 

func (check *ImageSizeCheck) Run() error

Run is a function of ImageSizeCheck and checks that all images to be promoted are under the max file size.

type ImageSizeError 

type ImageSizeError struct {
	MaxImageSize    int
	OversizedImages map[string]int
	InvalidImages   map[string]int
}

ImageSizeError contains ImageSizeCheck information on images that are either over the promoter's max image size or have an invalid size of 0 or less.

func (ImageSizeError) Error 

func (err ImageSizeError) Error() string

Error is a function of ImageSizeError and implements the error interface.

type ImageTag 

type ImageTag struct {
	ImageName ImageName
	Tag       Tag
}

ImageTag is a combination of the ImageName and Tag.

type ImageVulnCheck 

type ImageVulnCheck struct {
	SyncContext       SyncContext
	PullEdges         map[PromotionEdge]interface{}
	SeverityThreshold int
	FakeVulnProducer  ImageVulnProducer
}

ImageVulnCheck implements the PreCheck interface and checks against images that have known vulnerabilities.

func MKImageVulnCheck 

func MKImageVulnCheck(
	syncContext SyncContext,
	newPullEdges map[PromotionEdge]interface{},
	severityThreshold int,
	fakeVulnProducer ImageVulnProducer,
) *ImageVulnCheck

MKImageVulnCheck returns an instance of ImageVulnCheck which checks against images that have known vulnerabilities. nolint[funlen]

func (*ImageVulnCheck) Run 

func (check *ImageVulnCheck) Run() error

Run is a function of ImageVulnCheck and checks that none of the images to be promoted have any severe vulnerabilities. nolint[errcheck]

type ImageVulnError 

type ImageVulnError struct {
	ImageName      ImageName
	Digest         Digest
	OccurrenceName string
	Vulnerability  *grafeaspb.VulnerabilityOccurrence
}

ImageVulnError contains ImageVulnCheck information on images that contain a vulnerability with a severity level at or above the defined threshold.

func (ImageVulnError) Error 

func (err ImageVulnError) Error() string

Error is a function of ImageSizeError and implements the error interface.

type ImageVulnProducer 

type ImageVulnProducer func(
	edge PromotionEdge,
) ([]*grafeaspb.Occurrence, error)

ImageVulnProducer is used by ImageVulnCheck to get the vulnerabilities for an image and allows for custom vulnerability producers for testing.

type ImageWithDigestSlice 

type ImageWithDigestSlice struct {
	// contains filtered or unexported fields
}

ImageWithDigestSlice uses a slice of digests instead of a map, allowing its contents to be sorted.

type Images 

type Images []Image

Images is a slice of Image types.

func ParseImagesFromFile 

func ParseImagesFromFile(filePath string) (Images, error)

ParseImagesFromFile parses an Images type from a file.

func ParseImagesYAML 

func ParseImagesYAML(b []byte) (Images, error)

ParseImagesYAML parses Images from a byteslice.

type Manifest 

type Manifest struct {
	// Registries contains the source and destination (Src/Dest) registry names.
	// There must be at least 2 registries: 1 source registry and 1 or more
	// destination registries.
	Registries []RegistryContext `yaml:"registries,omitempty"`
	Images     []Image           `yaml:"images,omitempty"`

	// Hidden fields; these are data structure optimizations that are populated
	// from the fields above. As they are redundant, there is no point in
	// storing this information in YAML.
	SrcRegistry *RegistryContext
	Filepath    string
}

Manifest stores the information in a manifest file (describing the desired state of a Docker Registry).

func FindManifest 

func FindManifest(o GrowManifestOptions) (Manifest, error)

FindManifest finds the manifest to modify.

func ParseManifestFromFile 

func ParseManifestFromFile(filePath string) (Manifest, error)

ParseManifestFromFile parses a Manifest from a filepath.

func ParseManifestYAML 

func ParseManifestYAML(b []byte) (Manifest, error)

ParseManifestYAML parses a Manifest from a byteslice. This function is separate from ParseManifestFromFile() so that it can be tested independently.

func ParseThinManifestFromFile 

func ParseThinManifestFromFile(filePath string) (Manifest, error)

ParseThinManifestFromFile parses a ThinManifest from a filepath and generates a Manifest.

func ParseThinManifestsFromDir 

func ParseThinManifestsFromDir(
	dir string,
) ([]Manifest, error)

ParseThinManifestsFromDir parses all thin Manifest files within a directory. We effectively have to create a map of manifests, keyed by the source registry (there can only be 1 source registry).

nolint[funlen]

func (*Manifest) Finalize 

func (m *Manifest) Finalize() error

Finalize finalizes a Manifest by populating extra fields.

func (*Manifest) ToRegInvImage 

func (manifest *Manifest) ToRegInvImage() RegInvImage

ToRegInvImage converts a Manifest into a RegInvImage.

func (Manifest) ToRegInvImageDigest 

func (m Manifest) ToRegInvImageDigest() RegInvImageDigest

ToRegInvImageDigest converts a Manifest to a RegInvImageDigest.

func (Manifest) ToRegInvImageTag 

func (m Manifest) ToRegInvImageTag() RegInvImageTag

ToRegInvImageTag converts a Manifest to a RegInvImageTag.

func (Manifest) Validate 

func (m Manifest) Validate() error

Validate checks for semantic errors in the yaml fields (the structure of the yaml is checked during unmarshaling).

type MasterInventory 

type MasterInventory map[RegistryName]RegInvImage

MasterInventory stores multiple RegInvImage elements, keyed by RegistryName.

func (*MasterInventory) PrettyValue 

func (mi *MasterInventory) PrettyValue() string

PrettyValue creates a prettified string representation of MasterInventory.

nolint[gocyclo]

type ParentDigest 

type ParentDigest map[Digest]Digest

ParentDigest holds a map of the digests of children to parent digests. It is a reverse mapping of ManifestLists, which point to all the child manifests.

type PopulateRequests 

type PopulateRequests func(
	*SyncContext,
	chan<- stream.ExternalRequest,
	*sync.WaitGroup)

PopulateRequests is a function that can generate requests used to fetch information about a Docker Registry, or to promote images. It basically generates the set of "gcloud ..." commands used to manipulate Docker Registries.

func MKPopulateRequestsForPromotionEdges 

func MKPopulateRequestsForPromotionEdges(
	toPromote map[PromotionEdge]interface{},
	mkProducer PromotionContext) PopulateRequests

MKPopulateRequestsForPromotionEdges takes in a map of PromotionEdges to promote and a PromotionContext and returns a PopulateRequests which can generate requests to be processed nolint[lll]

type PreCheck 

type PreCheck interface {
	Run() error
}

PreCheck represents a check function to run against a pull request that modifies the promoter manifests before oking promotion of the changes.

Run runs the defined check and returns an error if the check fails, returns nil otherwise.

type ProcessRequest 

type ProcessRequest func(
	*SyncContext,
	chan stream.ExternalRequest,
	chan<- RequestResult,
	*sync.WaitGroup,
	*sync.Mutex)

ProcessRequest is the counterpart to PopulateRequests. It is a function that can take a request (generated by PopulateRequests) and process it. In the ictual implementation (e.g. in ReadDigestsAndTags()) it closes over some other local variables to record the change of state in the Docker Registry that was touched by processing the request.

func MkRequestCapturer 

func MkRequestCapturer(captured *CapturedRequests) ProcessRequest

MkRequestCapturer returns a function that simply records requests as they are captured (slurped out from the reqs channel).

type PromotionContext 

type PromotionContext func(
	RegistryName,
	ImageName,
	RegistryContext,
	ImageName,
	Digest,
	Tag,
	TagOp) stream.Producer

PromotionContext holds all info required to create a stream that would produce a stream.Producer, as it relates to an intent to promote an image.

type PromotionEdge 

type PromotionEdge struct {
	SrcRegistry RegistryContext
	SrcImageTag ImageTag

	Digest Digest

	DstRegistry RegistryContext
	DstImageTag ImageTag
}

PromotionEdge represents a promotion "link" of an image repository between 2 registries.

func (PromotionEdge) VertexProps 

func (edge PromotionEdge) VertexProps(
	mi MasterInventory) (VertexProperty, VertexProperty)

VertexProps determines the properties of each vertex (src and dst) in the edge, depending on the state of the world in the MasterInventory.

func (PromotionEdge) VertexPropsFor 

func (edge PromotionEdge) VertexPropsFor(
	rc RegistryContext,
	imageTag ImageTag,
	mi MasterInventory) VertexProperty

VertexPropsFor examines one of the two vertices (src or dst) of a PromotionEdge.

type PromotionRequest 

type PromotionRequest struct {
	TagOp          TagOp
	RegistrySrc    RegistryName
	RegistryDest   RegistryName
	ServiceAccount string
	ImageNameSrc   ImageName
	ImageNameDest  ImageName
	Digest         Digest
	DigestOld      Digest // Only for tag moves.
	Tag            Tag
}

PromotionRequest contains all the information required for any type of promotion (or demotion!) (involving any TagOp).

func (*PromotionRequest) PrettyValue 

func (pr *PromotionRequest) PrettyValue() string

PrettyValue is a prettified string representation of a PromotionRequest.

type RegInvFlat 

type RegInvFlat map[ImageDigestTag]interface{}

RegInvFlat is a flattened view of a Docker Registry, where the keys contain all 3 attributes --- the image name, digest, and tag.

func (RegInvFlat) ToSet 

func (a RegInvFlat) ToSet() container.Set

ToSet converts a RegInvFlat to a Set.

type RegInvImage 

type RegInvImage map[ImageName]DigestTags

A RegInvImage is a map containing all of the image names, and their associated digest-to-tags mappings. It is the simplest view of a Docker Registry, because the keys are just the ImageNames (where each ImageName does *not* include the registry name, because we already key this by the RegistryName in MasterInventory).

The ImageName is actually a path name, because it can be "foo/bar/baz", where the image name is the string after the last slash (in this case, "baz").

func ApplyFilters 

func ApplyFilters(o GrowManifestOptions, rii RegInvImage) (RegInvImage, error)

ApplyFilters applies the filters in the options to whittle down the given rii.

func EdgesToRegInvImage 

func EdgesToRegInvImage(
	edges map[PromotionEdge]interface{},
	destRegistry string) RegInvImage

EdgesToRegInvImage takes the destination endpoints of all edges and converts their information to a RegInvImage type. It uses only those edges that are trying to promote to the given destination registry.

func ExcludeTags 

func ExcludeTags(rii RegInvImage, excludedTags map[Tag]bool) RegInvImage

ExcludeTags removes tags in rii that match excludedTags.

func FilterByDigest 

func FilterByDigest(rii RegInvImage, filterDigest Digest) RegInvImage

FilterByDigest removes all images in RegInvImage that do not match the filterDigest.

func FilterByImage 

func FilterByImage(rii RegInvImage, filterImage ImageName) RegInvImage

FilterByImage removes all images in RegInvImage that do not match the filterImage.

func FilterByTag 

func FilterByTag(rii RegInvImage, filterTag string) RegInvImage

FilterByTag removes all images in RegInvImage that do not match the filterTag.

func ReadStagingRepo 

func ReadStagingRepo(
	o GrowManifestOptions,
) (RegInvImage, error)

ReadStagingRepo reads the StagingRepo, and applies whatever filters are available to the resulting RegInvImage. This RegInvImage is what we want to inject into the "images.yaml" of a thin manifest.

func Union 

func Union(a, b RegInvImage) RegInvImage

Union inject b's contents into a. However, it does so in a special way.

func (RegInvImage) Minus 

func (a RegInvImage) Minus(b RegInvImage) RegInvImage

Minus is a set operation.

func (*RegInvImage) ToCSV 

func (rii *RegInvImage) ToCSV() string

ToCSV is like ToYAML, but instead of printing things in an indented format, it prints one image on each line as a CSV. If there is a tag pointing to the image, then it is printed next to the image on the same line.

E.g.

nolint[lll] a@sha256:0000000000000000000000000000000000000000000000000000000000000000,a:1.0 a@sha256:0000000000000000000000000000000000000000000000000000000000000000,a:latest b@sha256:1111111111111111111111111111111111111111111111111111111111111111,-

func (RegInvImage) ToRegInvImageDigest 

func (ri RegInvImage) ToRegInvImageDigest() RegInvImageDigest

ToRegInvImageDigest takes a RegInvImage and converts it to a RegInvImageDigest.

func (RegInvImage) ToRegInvImageTag 

func (ri RegInvImage) ToRegInvImageTag() RegInvImageTag

ToRegInvImageTag converts a RegInvImage to a RegInvImageTag.

func (RegInvImage) ToSet 

func (a RegInvImage) ToSet() container.Set

ToSet converts a RegInvImage to a Set.

func (*RegInvImage) ToSorted 

func (rii *RegInvImage) ToSorted() []ImageWithDigestSlice

ToSorted converts a RegInvImage type to a sorted structure.

func (*RegInvImage) ToYAML 

func (rii *RegInvImage) ToYAML(o YamlMarshalingOpts) string

ToYAML displays a RegInvImage as YAML, but with the map items sorted alphabetically.

nolint[gocognit]

func (RegInvImage) Union 

func (a RegInvImage) Union(b RegInvImage) RegInvImage

Union is a set operation.

type RegInvImageDigest 

type RegInvImageDigest map[ImageDigest]TagSlice

RegInvImageDigest is a view of a Docker Reqistry, keyed by ImageDigest.

func (RegInvImageDigest) Intersection 

func (a RegInvImageDigest) Intersection(b RegInvImageDigest) RegInvImageDigest

Intersection is a set operation.

func (RegInvImageDigest) Minus 

func (a RegInvImageDigest) Minus(b RegInvImageDigest) RegInvImageDigest

Minus is a set operation.

func (*RegInvImageDigest) PrettyValue 

func (riid *RegInvImageDigest) PrettyValue() string

PrettyValue converts a RegInvImageDigest to a prettified string representation.

func (RegInvImageDigest) ToRegInvImageTag 

func (riid RegInvImageDigest) ToRegInvImageTag() RegInvImageTag

ToRegInvImageTag converts a RegInvImageDigest to a RegInvImageTag.

func (RegInvImageDigest) ToSet 

func (a RegInvImageDigest) ToSet() container.Set

ToSet converts a RegInvFlat to a Set.

type RegInvImageTag 

type RegInvImageTag map[ImageTag]Digest

RegInvImageTag is keyed by a ImageTag.

func (RegInvImageTag) Intersection 

func (a RegInvImageTag) Intersection(b RegInvImageTag) RegInvImageTag

Intersection is a set operation.

func (RegInvImageTag) Minus 

func (a RegInvImageTag) Minus(b RegInvImageTag) RegInvImageTag

Minus is a set operation.

func (RegInvImageTag) ToRegInvImageDigest 

func (riit RegInvImageTag) ToRegInvImageDigest() RegInvImageDigest

ToRegInvImageDigest converts a RegInvImageTag to a RegInvImageDigest.

func (RegInvImageTag) ToSet 

func (a RegInvImageTag) ToSet() container.Set

ToSet converts a RegInvImageTag to a Set.

type Registry 

type Registry struct {
	RegistryName      string
	RegistryNameLong  RegistryName
	RegInvImageDigest RegInvImageDigest
}

Registry is another way to look at a Docker Registry; it is used during Promotion.

func (*Registry) PrettyValue 

func (r *Registry) PrettyValue() string

PrettyValue converts a Registry to a prettified string representation.

type RegistryContext 

type RegistryContext struct {
	Name           RegistryName `yaml:"name,omitempty"`
	ServiceAccount string       `yaml:"service-account,omitempty"`
	Token          gcloud.Token `yaml:"-"`
	Src            bool         `yaml:"src,omitempty"`
}

RegistryContext holds information about a registry, to be written in a manifest file.

func GetSrcRegistry 

func GetSrcRegistry(rcs []RegistryContext) (*RegistryContext, error)

GetSrcRegistry gets the source registry.

type RegistryImagePath 

type RegistryImagePath string

RegistryImagePath is the registry name and image name, without the tag. E.g. "gcr.io/foo/bar/baz/image".

type RegistryName 

type RegistryName string

RegistryName is the leading part of an image name that includes the domain; it is everything that is not the actual image name itself. E.g., "gcr.io/google-containers".

type RequestResult 

type RequestResult struct {
	Context stream.ExternalRequest
	Errors  Errors
}

RequestResult contains information about the result of running a request (e.g., a "gcloud" command, or perhaps in the future, a REST call).

type RootRepo 

type RootRepo string

RootRepo is the toplevel Docker repository (e.g., gcr.io/foo (GCR domain name + GCP project name).

type SyncContext 

type SyncContext struct {
	Threads           int
	DryRun            bool
	UseServiceAccount bool
	Inv               MasterInventory
	InvIgnore         []ImageName
	RegistryContexts  []RegistryContext
	SrcRegistry       *RegistryContext
	Tokens            map[RootRepo]gcloud.Token
	DigestMediaType   DigestMediaType
	DigestImageSize   DigestImageSize
	ParentDigest      ParentDigest
	Logs              CollectedLogs
}

SyncContext is the main data structure for performing the promotion.

func MakeSyncContext 

func MakeSyncContext(
	mfests []Manifest,
	threads int,
	dryRun, useSvcAcc bool) (SyncContext, error)

MakeSyncContext creates a SyncContext.

func (*SyncContext) ClearRepository 

func (sc *SyncContext) ClearRepository(
	regName RegistryName,
	mkProducer func(RegistryContext, ImageName, Digest) stream.Producer,
	customProcessRequest *ProcessRequest)

ClearRepository wipes out all Docker images from a registry! Use with caution. nolint[gocyclo]

TODO: Maybe split this into 2 parts, so that each part can be unit-tested separately (deletion of manifest lists vs deletion of other media types).

func (*SyncContext) ExecRequests 

func (sc *SyncContext) ExecRequests(
	populateRequests PopulateRequests,
	processRequest ProcessRequest) error

ExecRequests uses the Worker Pool pattern, where MaxConcurrentRequests determines the number of workers to spawn.

nolint[funlen]

func (*SyncContext) FilterPromotionEdges 

func (sc *SyncContext) FilterPromotionEdges(
	edges map[PromotionEdge]interface{},
	readRepos bool,
) (map[PromotionEdge]interface{}, bool)

FilterPromotionEdges generates all "edges" that we want to promote.

func (*SyncContext) GarbageCollect 

func (sc *SyncContext) GarbageCollect(
	mfest Manifest,
	mkProducer func(RegistryContext, ImageName, Digest) stream.Producer,
	customProcessRequest *ProcessRequest)

GarbageCollect deletes all images that are not referenced by Docker tags. nolint[gocyclo]

func (*SyncContext) GetPromotionCandidates 

func (sc *SyncContext) GetPromotionCandidates(
	edges map[PromotionEdge]interface{}) (map[PromotionEdge]interface{}, bool)

This filters out those edges from ToPromotionEdges (found in []Manifest), to only those PromotionEdges that makes sense to keep around. For example, we want to remove all edges that have already been promoted.

nolint[funlen] nolint[gocyclo]

func (*SyncContext) IgnoreFromPromotion 

func (sc *SyncContext) IgnoreFromPromotion(regName RegistryName)

IgnoreFromPromotion works by building up a new Inv type of those images that should NOT be bothered to be Promoted; these get ignored in the Promote() step later down the pipeline.

func (*SyncContext) LogJSONSummary 

func (sc *SyncContext) LogJSONSummary()

LogJSONSummary logs the SyncContext's Logs as a prettified JSON.

func (*SyncContext) PopulateTokens 

func (sc *SyncContext) PopulateTokens() error

PopulateTokens populates the SyncContext's Tokens map with actual usable access tokens.

func (*SyncContext) PrintCapturedRequests 

func (sc *SyncContext) PrintCapturedRequests(capReqs *CapturedRequests)

PrintCapturedRequests pretty-prints all given PromotionRequests.

func (*SyncContext) Promote 

func (sc *SyncContext) Promote(
	edges map[PromotionEdge]interface{},
	mkProducer func(
		RegistryName,
		ImageName,
		RegistryContext,
		ImageName,
		Digest,
		Tag,
		TagOp) stream.Producer,
	customProcessRequest *ProcessRequest) error

Promote performs container image promotion by realizing the intent in the Manifest.

nolint[gocyclo]

func (*SyncContext) ReadGCRManifestLists 

func (sc *SyncContext) ReadGCRManifestLists(
	mkProducer func(*SyncContext, GCRManifestListContext) stream.Producer)

ReadGCRManifestLists reads all manifest lists and populates the ParentDigest field of the SyncContext. ParentDigest is a map of values of the form map[ChildDigest]ParentDigest; and so, if a digest has an entry in this map, it is referenced by a parent DockerManifestList.

TODO: Combine this function with ReadRegistries().

nolint[gocyclo]

func (*SyncContext) ReadRegistries 

func (sc *SyncContext) ReadRegistries(
	toRead []RegistryContext,
	recurse bool,
	mkProducer func(*SyncContext, RegistryContext) stream.Producer)

ReadRegistries reads all images in all registries in the SyncContext Each registry is composed of a image repositories, which can be recursive.

To summarize: a docker *registry* is a set of *repositories*. It just so happens that to end-users, repositores resemble a tree structure because they are delineated by familiar filesystem-like "directory" paths.

We use the term "registry" to mean the "root repository" in this program, but to be technically correct, for gcr.io/google-containers/foo/bar/baz:

  • gcr.io is the registry
  • gcr.io/google-containers is the toplevel repository (or "root" repo)
  • gcr.io/google-containers/foo is a child repository
  • gcr.io/google-containers/foo/bar is a child repository
  • gcr.io/google-containers/foo/bar/baz is a child repository

It may or may not be the case that the child repository is empty. E.g., if only one image gcr.io/google-containers/foo/bar/baz:1.0 exists in the entire registry, the foo/ and bar/ subdirs are empty repositories.

The root repo, or "registry" in the loose sense, is what we care about. This is because in GCR, each root repo is given its own service account and credentials that extend to all child repos. And also in GCR, the name of the root repo is the same as the name of the GCP project that hosts it.

NOTE: Repository names may overlap with image names. E.g., it may be in the example above that there are images named gcr.io/google-containers/foo:2.0 and gcr.io/google-containers/foo/baz:2.0.

nolint[gocyclo]

func (*SyncContext) RemoveChildDigestEntries 

func (sc *SyncContext) RemoveChildDigestEntries(rii RegInvImage) RegInvImage

RemoveChildDigestEntries removes all tagless images in RegInvImage that are referenced by ManifestLists in the Registries.

func (*SyncContext) RunChecks 

func (sc *SyncContext) RunChecks(
	preChecks []PreCheck,
) error

RunChecks runs defined PreChecks in order to check the promotion.

type Tag 

type Tag string

Tag is a Docker tag.

type TagOp 

type TagOp int

TagOp is an enum that describes the various types of tag-modifying operations. These actions are a bit more low-level, and currently support 3 operations: adding, moving, and deleting.

func (*TagOp) PrettyValue 

func (op *TagOp) PrettyValue() string

PrettyValue is a prettified string representation of a TagOp.

type TagSet 

type TagSet map[Tag]interface{}

TagSet is a set of Tags.

func (TagSet) Intersection 

func (a TagSet) Intersection(b TagSet) TagSet

Intersection is a set operation.

func (TagSet) Minus 

func (a TagSet) Minus(b TagSet) TagSet

Minus is a set operation.

func (TagSet) ToSet 

func (a TagSet) ToSet() container.Set

ToSet converts a TagSet to a Set.

func (TagSet) Union 

func (a TagSet) Union(b TagSet) TagSet

Union is a set operation.

type TagSlice 

type TagSlice []Tag

TagSlice is a slice of Tags.

func (TagSlice) Intersection 

func (a TagSlice) Intersection(b TagSlice) TagSet

Intersection is a set operation.

func (TagSlice) Minus 

func (a TagSlice) Minus(b TagSlice) TagSet

Minus is a set operation.

func (TagSlice) ToTagSet 

func (a TagSlice) ToTagSet() TagSet

ToTagSet converts a TagSlice to a TagSet.

func (TagSlice) Union 

func (a TagSlice) Union(b TagSlice) TagSet

Union is a set operation.

type ThinManifest 

type ThinManifest struct {
	Registries []RegistryContext `yaml:"registries,omitempty"`

	ImagesPath string `yaml:"imagesPath,omitempty"`
}

ThinManifest is a more secure Manifest because it does not define the Images[] directly, but moves it to a separate location. The idea is to define a ThinManifest type as a YAML in one folder, and to define the []Image in another folder, and to have far stricter ACLs for the ThinManifest type. Then, PRs modifying just the []Image YAML won't be able to modify the src/destination repos or the credentials tied to them.

func ParseThinManifestYAML 

func ParseThinManifestYAML(b []byte) (ThinManifest, error)

ParseThinManifestYAML parses a ThinManifest from a byteslice.

type VertexProperty 

type VertexProperty struct {
	// Pqin means that the entire path, including the registry name, image
	// name, and tag, in that combination, exists.
	PqinExists      bool
	DigestExists    bool
	PqinDigestMatch bool
	BadDigest       Digest
	OtherTags       TagSlice
}

VertexProperty describes the properties of an Edge, with respect to the state of the world.

type YamlMarshalingOpts 

type YamlMarshalingOpts struct {
	// Render multiple tags on separate lines. I.e.,
	// prefer
	//
	//    sha256:abc...:
	//    - one
	//    - two
	//
	// over
	//
	//    sha256:abc...: ["one", "two"]
	//
	// If there is only 1 tag, it will be on one line in brackets (e.g.,
	// '["one"]').
	SplitTagsOverMultipleLines bool

	// Do not quote the digest. I.e., prefer
	//
	//    sha256:...:
	//
	// over
	//
	//    "sha256:...":
	//
	BareDigest bool
}

YamlMarshalingOpts holds options for tweaking the YAML output.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.