storage

Documentation

Overview

Package storage provides common functionality supporting Vervet Underground storage.

Index

• Constants
• func AssertCollateVersion(c *qt.C, s Storage)
• func DigestHeader(value string) string
• func GetSantizedHost(name string) string
• type CollatedVersionMappedSpecs
• type Collator
  • func NewCollator(options ...CollatorOption) (*Collator, error)
  • func (c *Collator) Add(service string, revision ContentRevision)
  • func (c *Collator) Collate() (map[vervet.Version]openapi3.T, error)
• type CollatorOption
  • func CollatorExcludePattern(excludePatterns vervet.ExcludePatterns) CollatorOption
  • func CollatorOverlay(overlay string) CollatorOption
• type ContentRevision
• type ContentRevisions
  • func (r ContentRevisions) Len() int
  • func (r ContentRevisions) Less(i, j int) bool
  • func (r ContentRevisions) Swap(i, j int)
• type Digest
  • func NewDigest(contents []byte) Digest
• type ReadOnlyStorage
• type ServiceRevisions
  • func NewServiceRevisions() *ServiceRevisions
  • func (s *ServiceRevisions) Add(revision ContentRevision)
  • func (s ServiceRevisions) ResolveLatestRevision(version vervet.Version) (ContentRevision, error)
• type Storage

Constants

const (
	CollatedVersionsFolder = "collated-versions/"
	ServiceVersionsFolder  = "service-versions/"
)

Functions

func AssertCollateVersion

func AssertCollateVersion(c *qt.C, s Storage)

func DigestHeader

func DigestHeader(value string) string

DigestHeader returns a content digest parsed from a Digest HTTP response header as defined in https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-digest-headers-05#section-3. The returned digest is algorithm-prefixed so that other digest schemes may be supported later if needed.

Returns "" if no digest is available.

func GetSantizedHost

func GetSantizedHost(name string) string

Types

type CollatedVersionMappedSpecs

type CollatedVersionMappedSpecs map[vervet.Version]openapi3.T

CollatedVersionMappedSpecs Compiled aggregated spec for all services at that given version.

type Collator

type Collator struct {
	// contains filtered or unexported fields
}

Collator is an aggregate of service specs and uniqueVersions scraped by VU. It is responsible for collating uniqueVersions and specs from all services VU manages. This is the top level resource all storage classes should use for producing collated data.

func NewCollator

func NewCollator(options ...CollatorOption) (*Collator, error)

NewCollator returns a new Collator instance.

func (*Collator) Add

func (c *Collator) Add(service string, revision ContentRevision)

Add a new service and revision to the Collator.

func (*Collator) Collate

func (c *Collator) Collate() (map[vervet.Version]openapi3.T, error)

Collate processes added service revisions to collate unified versions and OpenAPI specs for each version.

type CollatorOption

type CollatorOption func(*Collator) error

CollatorOption defines an optional setting when creating a new Collator.

func CollatorExcludePattern

func CollatorExcludePattern(excludePatterns vervet.ExcludePatterns) CollatorOption

CollatorExcludePattern is a CollatorOption which specifies an exclude pattern to apply when collating OpenAPI document objects.

func CollatorOverlay

func CollatorOverlay(overlay string) CollatorOption

CollatorOverlay is a CollatorOption which specifies an OpenAPI document overlay to apply to the collated result. Top-level fields in the overlay replaces top-level fields in the collated result. Paths are merged.

type ContentRevision

type ContentRevision struct {
	// Service is the name of the service.
	Service string
	// Version is the API version number of this content.
	Version vervet.Version
	// Timestamp represents when this revision was scraped.
	Timestamp time.Time
	// Digest is the sha of the revision derived from its content.
	Digest Digest
	// Blob is the actual content of this revision, the unmarshalled API spec.
	Blob []byte
}

ContentRevision is the exact contents and metadata of a service's version at scraping timestamp.

type ContentRevisions

type ContentRevisions []ContentRevision

ContentRevisions provides a deterministically ordered slice of content revisions. Revisions are ordered by vervet version then timestamp, newest to oldest. In the unlikely event of two revisions having the same version and timestamp, the digest is used as a tie-breaker.

func (ContentRevisions) Len

func (r ContentRevisions) Len() int

Len implements sort.Interface.

func (ContentRevisions) Less

func (r ContentRevisions) Less(i, j int) bool

Less implements sort.Interface.

func (ContentRevisions) Swap

func (r ContentRevisions) Swap(i, j int)

Swap implements sort.Interface.

type Digest

type Digest string

Digest is a sha256 calculation generated into a specific string format prefixed by "sha256:" followed by the sha256 value generated.

func NewDigest

func NewDigest(contents []byte) Digest

NewDigest returns the digest of the given contents.

type ReadOnlyStorage

type ReadOnlyStorage interface {
	// HasVersion returns whether the storage has already stored the service
	// API spec version at the given content digest.
	HasVersion(ctx context.Context, name string, version string, digest string) (bool, error)

	// VersionIndex fetches the Storage Versions index compiled by VU
	VersionIndex(ctx context.Context) (vervet.VersionIndex, error)

	// Version fetches the Storage Version spec compiled by VU
	Version(ctx context.Context, version string) ([]byte, error)
}

ReadOnlyStorage defines functionality needed to fetch spec versions. Implmentations can assume that the storage has already been populated through the Storage interface.

type ServiceRevisions

type ServiceRevisions struct {
	// contains filtered or unexported fields
}

ServiceRevisions tracks a collection of ContentRevisions and API uniqueVersions for a single service.

func NewServiceRevisions

func NewServiceRevisions() *ServiceRevisions

NewServiceRevisions returns a new instance of ServiceRevisions.

func (*ServiceRevisions) Add

func (s *ServiceRevisions) Add(revision ContentRevision)

Add registers a new ContentRevision for the service.

func (ServiceRevisions) ResolveLatestRevision

func (s ServiceRevisions) ResolveLatestRevision(version vervet.Version) (ContentRevision, error)

ResolveLatestRevision returns the latest revision that matches the given version date. If no exact version is found, it uses vervet to resolve the most recent version date at the same stability. When multiple revisions are found for a given version, the content revision with the latest scrape timestamp is returned.

type Storage

type Storage interface {
	// NotifyVersions tells the storage which versions are currently available.
	// This is the primary mechanism by which the storage layer discovers and
	// processes versions which are removed post-sunset.
	NotifyVersions(ctx context.Context, name string, versions []string, scrapeTime time.Time) error

	// CollateVersions tells the storage to execute the compilation and
	// update all VU-formatted specs from all services and their
	// respective versions gathered.
	CollateVersions(ctx context.Context, serviceFilter map[string]bool) error

	// NotifyVersion tells the storage to store the given version contents at
	// the scrapeTime. The storage implementation must detect and ignore
	// duplicate version contents, as some services may not provide content
	// digest headers in their responses.
	NotifyVersion(ctx context.Context, name string, version string, contents []byte, scrapeTime time.Time) error

	ReadOnlyStorage
}

Storage defines the storage functionality needed in order to store service API version spec snapshots.