vervet

package module
v4.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 3, 2022 License: Apache-2.0 Imports: 20 Imported by: 0

README

vervet

Vervet is an HTTP API version lifecycle management tool, allowing APIs to be designed, developed, versioned and released from resources independently and concurrently.

In a large organization, there might be many teams involved in delivering a large API -- such as at Snyk where Vervet was developed.

Within a single small team, there is still often a need to simultaneously try new things in parts of an API while maintaining stability.

While Vervet was developed in the context of a RESTful API, Vervet can be used with any HTTP API expressed in OpenAPI 3 -- even if it does not adhere to strict REST principles.

API Versioning

To summarize the API versioning supported by Vervet:

What is versioned?

Resource versions are defined in OpenAPI 3, as if the resource were a standalone service.

How are resource version specs organized?

Resources are organized in a standard directory structure by release date, using OpenAPI extensions to define lifecycle concepts like stability.

How does versioning work?
  • Resources are versioned independently by date and stability, with a well-defined deprecation and sunsetting policy.
  • Additive, non-breaking changes can be made to released versions. Breaking changes trigger a new version.
  • New versions deprecate and sunset prior versions, on a timeline determined by the stability level.

Read more about API versioning.

Features

A brief tour of Vervet's features.

Compilation

Vervet compiles the OpenAPI spec of each resource version into a series of OpenAPI specifications that describe the entire application, at each distinct release in its underlying parts.

Given a directory structure of resource versions, each defined by an OpenAPI spec as if it were an independent service:

$ tree resources
resources
├── _examples
│   └── hello-world
│       ├── 2021-06-01
│       │   └── spec.yaml
│       ├── 2021-06-07
│       │   └── spec.yaml
│       └── 2021-06-13
│           └── spec.yaml
└── projects
    └── 2021-06-04
        └── spec.yaml

and a Vervet project configuration that instructs how to put them together:

$ cat .vervet.yaml
apis:
  my-api:
    resources:
      - path: 'resources'
    output:
      path: 'versions'

vervet compile aggregates these resources' individual OpenAPI specifications to describe the entire service API at each distinct version date and stability level from its component parts.

$ tree versions
versions/
├── 2021-06-01
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-01~beta
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-01~experimental
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-04
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-04~beta
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-04~experimental
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-07
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-07~beta
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-07~experimental
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-13
│   ├── spec.json
│   └── spec.yaml
├── 2021-06-13~beta
│   ├── spec.json
│   └── spec.yaml
└── 2021-06-13~experimental
    ├── spec.json
    └── spec.yaml
Linting

Vervet is not an OpenAPI linter. It coordinates and frontends OpenAPI linting, allowing different rules to be applied to different parts of an API, or different stages of the compilation process (source component specs, output compiled specs). It also allows exceptions to be made to certain resource versions, so that new rules do not break already-released parts of the API.

Vervet currently supports linting OpenAPI specifications with:

  • Spectral
  • Sweater Comb, as a self-contained Docker image which combines a linter and custom opinionated rulesets.

Direct Spectral linting may be soon deprecated in favor of container-based linting.

Generation

Since Vervet models the composition and construction of an API, it is well positioned to coordinate code and artifact generation through templates.

Generators are defined in .vervet.yaml:

generators:
  version-readme:
    scope: version
    filename: "resources/{{ .Resource }}/{{ .Version }}/README"
    template: ".vervet/templates/README.tmpl"
  version-spec:
    scope: version
    filename: "resources/{{ .Resource }}/{{ .Version }}/spec.yaml"
    template: ".vervet/templates/spec.yaml.tmpl"

In this case, generators produce a boilerplate OpenAPI specification containing HTTP methods to create, list, get, update, and delete a resource, and a nice README when a new resource version is created. OpenAPI specifications can be tedious to write from scratch; generators help developers focus on adding the content that matters most.

Generators are defined using Go templates. Template syntax is also used to express filename interpolation per resource, per version.

apis:
  my-api:
    resources:
      - path: 'resources'
    generators:
      - version-readme
      - version-spec
    output:
      path: 'versions'

Generators are applied during lifecycle commands, such as creating a new resource version:

$ vervet version new my-api thing
$ tree resources
resources
└── thing
    └── 2021-10-21
        ├── README
        └── spec.yaml

Generators support multiple stages. For example, once a boilerplate spec.yaml is generated, it can be fed into subsequent generators that produce code, API gateway configuration, Grafana dashboards, and HTTP load tests.

A more advanced example, ExpressJS controllers generated from each operation in a resource version OpenAPI spec:

generators:
  version-spec:
    scope: version
    filename: "resources/{{ .Resource }}/{{ .Version }}/spec.yaml"
    template: ".vervet/templates/spec.yaml.tmpl"
  version-controller:
    scope: version
    # `files:` generates a collection of files -- which itself is expressed as a
    # YAML template.  Keys in this YAML are the paths of the files to generate,
    # whose values are the file contents.
    files: |-
      {{- $resource := .Resource -}}
      {{- $version := .Version -}}
      {{- range $path, $pathItem := .Data.Spec.paths -}}
      {{- range $method, $operation := $pathItem -}}
      {{- $operationId := $operation.operationId -}}
      {{/* Construct a context object using the 'map' function */}}
      {{- $ctx := map "Context" . "OperationId" $operationId }}
      resources/{{ $resource }}/{{ $version }}/{{ $operationId }}.ts: |-
        {{/*
             Evaluate the template by including it with the necessary context.
             The generator's template is included as "contents" from within the
             `files:` template.
           */}}
        {{ include "contents" $ctx | indent 2 }}
      {{ end }}
      {{- end -}}
    template: ".vervet/resource/version/controller.ts.tmpl"
    data:
      Spec:
        # generated above in version-spec, accessible from within the `files:`
        # template as `.Data.Spec`.
        include: "resources/{{ .Resource }}/{{ .Version }}/spec.yaml"
apis:
  my-api:
    resources:
      - path: 'resources'
        generators:
          # order is important
          - version-spec
          - version-controller
    output:
      path: 'versions'

In this case, a template is being applied per operationId in the spec.yaml generated in the prior step. version-controller produces a collection of files, a controller module per resource, per version, per operation. This is possible because generators are applied in the order they are declared on each set of resources.

Scaffolding

Just as generators automate the generation of artifacts as part of the versioning lifecycle, scaffolds are used to bootstrap a new greenfield Vervet API project with useful defaults:

  • Vervet project configuration (.vervet.yaml)
  • Directory structure and layout for API specifications
  • Generator templates
  • Linter rulesets

Scaffolds are great in a microservice/SOA self-service ecosystem, where new services may be created often, and need a set of sensible defaults to quickly get started.

$ mkdir my-new-service
$ cd my-new-service
$ vervet scaffold init ../vervet-api-scaffold/
$ tree -a
.
├── .vervet
│   ├── components
│   │   ├── common.yaml
│   │   ├── errors.yaml
│   │   ├── headers
│   │   │   └── headers.yaml
│   │   ├── parameters
│   │   │   ├── pagination.yaml
│   │   │   └── version.yaml
│   │   ├── responses
│   │   │   ├── 204.yaml
│   │   │   ├── 400.yaml
│   │   │   ├── 401.yaml
│   │   │   ├── 403.yaml
│   │   │   ├── 404.yaml
│   │   │   ├── 409.yaml
│   │   │   ├── 429.yaml
│   │   │   └── 500.yaml
│   │   ├── tag.yaml
│   │   ├── types.yaml
│   │   └── version.yaml
│   ├── openapi
│   │   └── spec.yaml
│   └── templates
│       ├── README.tmpl
│       └── spec.yaml.tmpl
├── .vervet.yaml
└── api
    ├── resources
    └── versions

This scaffold sets up a new project with standard OpenAPI components that are referenced by resource OpenAPI boilerplate templates. New resources are generated already conforming to our JSON API standards and paginated list operations.

Installation

NPM
npm install -g @snyk/vervet

NPM packaging adapted from https://github.com/manifoldco/torus-cli.

Source

Go >= 1.16 required.

go build ./cmd/vervet

or

make build

Development

Vervet uses a reference set of OpenAPI documents in testdata/resources in tests. CLI tests compare runtime compiled output with pre-compiled, expected output in testdata/output to detect regressions.

When introducing changes that intentionally change the content of compiled output:

  • Run go generate ./testdata to update the contents of testdata/output
  • Verify that the compiled output is correct
  • Commit the changes to testdata/output in your proposed branch

Documentation

Overview

Package vervet supports opinionated API versioning tools.

Index

Constants

View Source 
const (
	// ExtSnykApiStability is used to annotate a top-level resource version
	// spec with its API release stability level.
	ExtSnykApiStability = "x-snyk-api-stability"

	// ExtSnykApiResource is used to annotate a path in a compiled OpenAPI spec
	// with its source resource name.
	ExtSnykApiResource = "x-snyk-api-resource"

	// ExtSnykApiVersion is used to annotate a path in a compiled OpenAPI spec
	// with its resolved release version.
	ExtSnykApiVersion = "x-snyk-api-version"

	// ExtSnykApiReleases is used to annotate a path in a compiled OpenAPI spec
	// with all the release versions containing a change in the path info. This
	// is useful for navigating changes in a particular path across versions.
	ExtSnykApiReleases = "x-snyk-api-releases"

	// ExtSnykDeprecatedBy is used to annotate a path in a resource version
	// spec with the subsequent version that deprecates it. This may be used
	// by linters, service middleware and API documentation to indicate which
	// version deprecates a given version.
	ExtSnykDeprecatedBy = "x-snyk-deprecated-by"

	// ExtSnykSunsetEligible is used to annotate a path in a resource version
	// spec which is deprecated, with the sunset eligible date: the date after
	// which the resource version may be removed and no longer available.
	ExtSnykSunsetEligible = "x-snyk-sunset-eligible"
)
View Source 
const (
	// SunsetWIP is the duration past deprecation after which a work-in-progress version may be sunset.
	SunsetWIP = 0

	// SunsetExperimental is the duration past deprecation after which an experimental version may be sunset.
	SunsetExperimental = 31 * 24 * time.Hour

	// SunsetBeta is the duration past deprecation after which a beta version may be sunset.
	SunsetBeta = 91 * 24 * time.Hour

	// SunsetGA is the duration past deprecation after which a GA version may be sunset.
	SunsetGA = 181 * 24 * time.Hour
)
View Source 
const (
	// ExtSnykIncludeHeaders is used to annotate a response with a list of
	// headers. While OpenAPI supports header references, it does not yet
	// support including a collection of common headers. This extension is used
	// by vervet to include headers from a referenced document when compiling
	// OpenAPI specs.
	ExtSnykIncludeHeaders = "x-snyk-include-headers"
)
View Source 
const SpecGlobPattern = "**/[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]/spec.yaml"

SpecGlobPattern defines the expected directory structure for the versioned OpenAPI specs of a single resource: subdirectories by date, of the form YYYY-mm-dd, each containing a spec.yaml file.

Variables

View Source 
var ErrNoMatchingVersion = fmt.Errorf("no matching version")

ErrNoMatchingVersion indicates the requested version cannot be satisfied by the declared versions that are available.

Functions

func ExtensionString 

func ExtensionString(extProps openapi3.ExtensionProps, key string) (string, error)

ExtensionString returns the string value of an OpenAPI extension.

func IncludeHeaders 

func IncludeHeaders(doc *Document) error

IncludeHeaders adds response headers included with the ExtSnykIncludeHeaders extension property.

func IsExtensionNotFound 

func IsExtensionNotFound(err error) bool

IsExtensionNotFound returns bool whether error from ExtensionString is not found versus unexpected.

func LoadVersions 

func LoadVersions(root fs.FS) ([]*openapi3.T, error)

LoadVersions loads all Vervet-compiled and versioned API specs from a filesystem root and returns them.

func Localize 

func Localize(doc *Document) error

Localize rewrites all references in an OpenAPI document to local references.

func Merge 

func Merge(dst, src *openapi3.T, replace bool)

Merge adds the paths and components from a source OpenAPI document root, to a destination document root.

TODO: This is a naive implementation that should be improved to detect and resolve conflicts better. For example, distinct resources might have localized references with the same URIs but different content. Content-addressible resource versions may further facilitate governance; this also would facilitate detecting and relocating such conflicts.

func ToSpecJSON 

func ToSpecJSON(v interface{}) ([]byte, error)

ToSpecJSON renders an OpenAPI document object as JSON.

func ToSpecYAML 

func ToSpecYAML(v interface{}) ([]byte, error)

ToSpecYAML renders an OpenAPI document object as YAML.

func VersionDateStrings 

func VersionDateStrings(vs []Version) []string

VersionDateStrings returns a slice of distinct version date strings for a slice of Versions. Consecutive duplicate dates are removed.

func WithGeneratedComment 

func WithGeneratedComment(yamlBuf []byte) ([]byte, error)

WithGeneratedComment prepends a comment to YAML output indicating the file was generated.

Types

type Document 

type Document struct {
	*openapi3.T
	// contains filtered or unexported fields
}

Document is an OpenAPI 3 document object model.

func NewDocumentFile 

func NewDocumentFile(specFile string) (_ *Document, returnErr error)

NewDocumentFile loads an OpenAPI spec file from the given file path, returning a document object.

func (*Document) LoadReference 

func (d *Document) LoadReference(relPath, refPath string, target interface{}) (_ string, returnErr error)

LoadReference loads a reference from refPath, relative to relPath, into target. The relative path of the reference is returned, so that references may be chain-loaded with successive calls.

func (*Document) Location 

func (d *Document) Location() *url.URL

Location returns the URL from where the document was loaded.

func (*Document) MarshalJSON 

func (d *Document) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (*Document) RelativePath 

func (d *Document) RelativePath() string

RelativePath returns the relative path for resolving references from the file path location of the top-level document: the directory which contains the file from which the top-level document was loaded.

func (*Document) ResolveRefs 

func (d *Document) ResolveRefs() error

ResolveRefs resolves all Ref types in the document, causing the Value field of each Ref to be loaded and populated from its referenced location.

type ResourceVersion 

type ResourceVersion struct {
	*Document
	Name    string
	Version Version
	// contains filtered or unexported fields
}

ResourceVersion defines a specific version of a resource, corresponding to a standalone OpenAPI specification document that defines its operations, schema, etc. While a resource spec may declare multiple paths, they should all describe operations on a single conceptual resource.

func (*ResourceVersion) Validate 

func (e *ResourceVersion) Validate(ctx context.Context) error

Validate returns whether the ResourceVersion is valid. The OpenAPI specification must be valid, and must declare at least one path.

type ResourceVersions 

type ResourceVersions struct {
	// contains filtered or unexported fields
}

ResourceVersions defines a collection of multiple versions of a resource.

func LoadResourceVersions 

func LoadResourceVersions(epPath string) (*ResourceVersions, error)

LoadResourceVersions returns a ResourceVersions slice parsed from a directory structure of resource specs. This directory will be of the form: 

resource/
+- 2021-01-01
   +- spec.yaml
+- 2021-06-21
   +- spec.yaml
+- 2021-07-14
   +- spec.yaml

The resource version stability level is defined by the ExtSnykApiStability extension value at the top-level of the OpenAPI document.

func LoadResourceVersionsFileset 

func LoadResourceVersionsFileset(specYamls []string) (*ResourceVersions, error)

LoadResourceVersionFileset returns a ResourceVersions slice parsed from the directory structure described above for LoadResourceVersions.

func (*ResourceVersions) At 

func (e *ResourceVersions) At(vs string) (*ResourceVersion, error)

At returns the ResourceVersion matching a version string. The version of the resource returned will be the latest available version with a stability equal to or greater than the requested version, or ErrNoMatchingVersion if no matching version is available.

func (*ResourceVersions) Name 

func (e *ResourceVersions) Name() string

Name returns the resource name for a collection of resource versions.

func (*ResourceVersions) Versions 

func (e *ResourceVersions) Versions() []Version

Versions returns a slice containing each Version defined for this resource.

type SpecVersions 

type SpecVersions struct {
	// contains filtered or unexported fields
}

SpecVersions stores a collection of versioned OpenAPI specs.

func LoadSpecVersions 

func LoadSpecVersions(root string) (*SpecVersions, error)

LoadSpecVersions returns SpecVersions loaded from a directory structure containing one or more Resource subdirectories.

func LoadSpecVersionsFileset 

func LoadSpecVersionsFileset(epPaths []string) (*SpecVersions, error)

LoadSpecVersionsFileset returns SpecVersions loaded from a set of spec files.

func (*SpecVersions) At 

func (sv *SpecVersions) At(v Version) (*openapi3.T, error)

At returns the OpenAPI document that matches the given version. If the version is not an exact match for an API release, the OpenAPI document effective on the given version date for the version stability level is returned. Returns ErrNoMatchingVersion if there is no release matching this version.

func (*SpecVersions) Versions 

func (sv *SpecVersions) Versions() VersionSlice

Versions returns the distinct API versions in this collection of OpenAPI documents.

type Stability 

type Stability int

Stability defines the stability level of the version. 

const (

	// StabilityWIP means the API is a work-in-progress and not yet ready.
	StabilityWIP Stability = iota

	// StabilityExperimental means the API is experimental and still subject to
	// drastic change.
	StabilityExperimental Stability = iota

	// StabilityBeta means the API is becoming more stable, but may undergo some
	// final changes before being released.
	StabilityBeta Stability = iota

	// StabilityGA means the API has been released and will not change.
	StabilityGA Stability = iota
)

func MustParseStability 

func MustParseStability(s string) Stability

MustParseStability parses a stability string into a Stability type, panicking if the string is invalid.

func ParseStability 

func ParseStability(s string) (Stability, error)

ParseStability parses a stability string into a Stability type, returning an error if the string is invalid.

func (Stability) Compare 

func (s Stability) Compare(sr Stability) int

Compare returns -1 if the given stability level is less than, 0 if equal to, and 1 if greater than the caller target stability level.

func (Stability) String 

func (s Stability) String() string

String returns a string representation of the stability level. This method will panic if the value is empty.

type Version 

type Version struct {
	Date      time.Time
	Stability Stability
}

Version defines an API version. API versions may be dates of the form "YYYY-mm-dd", or stability tags "beta", "experimental".

func MustParseVersion 

func MustParseVersion(s string) Version

MustParseVersion parses a version string into a Version type, panicking if the string is invalid.

func ParseVersion 

func ParseVersion(s string) (Version, error)

ParseVersion parses a version string into a Version type, returning an error if the string is invalid.

func (Version) AddDays 

func (v Version) AddDays(days int) Version

AddDays returns the version corresponding to adding the given number of days to the version date.

func (Version) Compare 

func (v Version) Compare(vr Version) int

Compare returns -1 if the given version is less than, 0 if equal to, and 1 if greater than the caller target version.

func (Version) DateString 

func (v Version) DateString() string

DateString returns the string representation of the version date in YYYY-mm-dd form.

func (Version) DeprecatedBy 

func (v Version) DeprecatedBy(vr Version) bool

DeprecatedBy returns true if the given version deprecates the caller target version.

func (Version) String 

func (v Version) String() string

String returns the string representation of the version in YYYY-mm-dd~Stability form. This method will panic if the value is empty.

func (Version) Sunset 

func (v Version) Sunset(vr Version) (time.Time, bool)

Sunset returns, given a potentially deprecating version, the eligible sunset date and whether the caller target version would actually be deprecated and sunset by the given version.

type VersionSlice 

type VersionSlice []Version

VersionSlice is a sortable, searchable slice of Versions.

func (VersionSlice) Deprecates 

func (vs VersionSlice) Deprecates(q Version) (Version, bool)

Deprecates returns the version that deprecates the given version in the slice.

func (VersionSlice) Len 

func (vs VersionSlice) Len() int

Len implements sort.Interface.

func (VersionSlice) Less 

func (vs VersionSlice) Less(i, j int) bool

Less implements sort.Interface.

func (VersionSlice) Resolve 

func (vs VersionSlice) Resolve(q Version) (Version, error)

Resolve returns the most recent Version in the slice with equal or greater stability.

This method requires that the VersionSlice has already been sorted with sort.Sort, otherwise behavior is undefined.

func (VersionSlice) ResolveIndex 

func (vs VersionSlice) ResolveIndex(q Version) (int, error)

ResolveIndex returns the slice index of the most recent Version in the slice with equal or greater stability.

This method requires that the VersionSlice has already been sorted with sort.Sort, otherwise behavior is undefined.

func (VersionSlice) Strings 

func (vs VersionSlice) Strings() []string

Strings returns a slice of string versions

func (VersionSlice) Swap 

func (vs VersionSlice) Swap(i, j int)

Swap implements sort.Interface.

Source Files

View all Source files

Directories

Path Synopsis
cmd
vervet
internal
cmd
Package cmd provides subcommands for the vervet CLI.
 Package cmd provides subcommands for the vervet CLI.
compiler
files
generator
linter
linter/optic
Package optic supports linting OpenAPI specs with Optic CI and Sweater Comb.
 Package optic supports linting OpenAPI specs with Optic CI and Sweater Comb.
linter/spectral
scaffold
Package versionware provides routing and middleware for building versioned HTTP services.
 Package versionware provides routing and middleware for building versioned HTTP services.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.