schemagen

README

Schemagen

A tool used to generate validation schemas for Kubernetes CRDs

Validation Schemas

Custom Resources in Kubernetes support defining a structural schema, which is validated on writes (ie create and update). V1 CRDs (required in Kube 1.22 and beyond) require validation schemas. These schemas provide syntax validation (ie does the configuration respect the API structure).

Options

crdDirectory - Path to the directory where CRDs will be read from and written to.

jsonSchemaTool - Name of the tool used to generate JsonSchemas, defaults to protoc. cue is also supported, but can run excessively long for deeply nested protos.

removeDescriptionsFromSchema - Whether to remove descriptions from validation schemas, defaults to false. Descriptions are a non-functional aspect of CRD validation schemas, and therefore it is recommended to not include descriptions since they can expand the size of the CRD.

enumAsIntOrString - Whether to assign Enum fields the x-kubernetes-int-or-string property which allows the value to either be an integer, or a string, defaults to false. Only strings are supported when this is false. Projects which consume this library may have custom enum marshalers. If the project may marshal an enum as an int or a string, the underlying schema needs to support that as well.

messagesWithEmptySchema - A list of message names (ie core.solo.io.Status) for which we should generate a validation schema that accepts all properties. This is useful when certain messages take too long to compute schemas, or are recursive.

Implementation

This tool executes the following steps to generate validation schemas:

1. Determine if a project is configured to generate validation schemas. If not, return immediately.
2. Read all the CRDs from the provided crdDirectory
3. Choose the JsonSchemaTool. We currently support cue and protoc though protoc is used in Gloo Edge
4. Generate all JsonSchemas for the project
5. For each CRD
   • Match the CRD with the JsonSchema, using the GVK.
   • Write the modified CRD back to the file, using the GVK to generate the file name

NOTE It would be nice to move complete CRD generation into this tool, but for now we only modify the schema.

Generation Tools

We currently support 2 separate implementations for schema generation.

Cue

Use cuelang as an intermediate language for transpiling protobuf schemas to openapi v3 with k8s structural schema constraints.

This is our preferred implementation. It is not used in production yet, due to some performance issues with the Gloo Edge API. However, our goal is to eventually migrate our code to rely on this implementation. This code was included in this iteration so that we could compare the generated schemas between the implementations.

cuelang/cue#944 tracks the issue we face when using cuelang with the Gloo Edge API. Specifically, as more oneof's are added to the API, the performance degrades dramatically, to the point where it takes longer than 15 minutes to generate schemas. The authors are aware of this issue and are actively working on improving it. Until it is resolved, we cannot move forward with this implementation.

skv2 leverages Cue to build the validation schemas for CRDs.

Protoc

protoc-gen-openapi is a plugin for the Google protocol buffer compiler to generate openAPI V3 spec for any given input protobuf. It runs as a protoc-gen- binary that the protobuf compiler infers from the openapi_out flag.

This is our current implementation that is used in production, but we hope only temporarily. As described above, the latest version of cue doesn't work with the Gloo Edge API for performance reasons.

This implementation relies on the protobuf compiler being run with an openapi_out flag. For each resource proto, we run protoc with the corresponding flags, write the schemas to a temp file, parse that file and convert it into a JsonSchema.

We based our work off an old istio tool. It has since been removed from the repo, but we cloned it, and made necessary modifications to support specific Gloo protos.

Documentation

Index

• Variables
• func GenerateOpenApiValidationSchemas(project *model.Project, options *ValidationSchemaOptions, ...) error
• func GetCRDFromFile(pathToFile string) (apiextv1.CustomResourceDefinition, error)
• func GetCRDsFromDirectory(crdDirectory string) ([]apiextv1.CustomResourceDefinition, error)
• func NewCueGenerator(importsCollector collector.Collector, absoluteRoot string) *cueGenerator
• func NewProtocGenerator(importsCollector collector.Collector, absoluteRoot string, ...) *protocGenerator
• type CrdWriter
  • func NewCrdWriter(crdDirectory string) *CrdWriter
  • func (c *CrdWriter) ApplyValidationSchemaToCRD(crd apiextv1.CustomResourceDefinition, ...) error
• type JsonSchemaGenerator
• type OpenApiSchemas
• type ValidationSchemaOptions

Variables

var (
	ApiVersionMismatch = func(expected, actual string) error {
		return eris.Errorf("Expected ApiVersion [%s] but found [%s]", expected, actual)
	}
)

Functions

func GenerateOpenApiValidationSchemas

func GenerateOpenApiValidationSchemas(project *model.Project, options *ValidationSchemaOptions, importsCollector collector.Collector, absoluteRoot string) error

func GetCRDFromFile

func GetCRDFromFile(pathToFile string) (apiextv1.CustomResourceDefinition, error)

func GetCRDsFromDirectory

func GetCRDsFromDirectory(crdDirectory string) ([]apiextv1.CustomResourceDefinition, error)

func NewCueGenerator

func NewCueGenerator(importsCollector collector.Collector, absoluteRoot string) *cueGenerator

func NewProtocGenerator

func NewProtocGenerator(importsCollector collector.Collector, absoluteRoot string, validationSchemaOptions *ValidationSchemaOptions) *protocGenerator

Types

type CrdWriter

type CrdWriter struct {
	// contains filtered or unexported fields
}

func NewCrdWriter

func NewCrdWriter(crdDirectory string) *CrdWriter

func (*CrdWriter) ApplyValidationSchemaToCRD

func (c *CrdWriter) ApplyValidationSchemaToCRD(crd apiextv1.CustomResourceDefinition, validationSchema *apiextv1.CustomResourceValidation) error

type JsonSchemaGenerator

type JsonSchemaGenerator interface {
	GetJsonSchemaForProject(project *model.Project) (map[schema.GroupVersionKind]*apiextv1.JSONSchemaProps, error)
}

type OpenApiSchemas

type OpenApiSchemas map[string]*openapi.OrderedMap

Mapping from protobuf message name to OpenApi schema

type ValidationSchemaOptions

type ValidationSchemaOptions struct {
	// Path to the directory where CRDs will be read from and written to
	CrdDirectory string

	// Tool used to generate JsonSchemas, defaults to protoc
	JsonSchemaTool string

	// Whether to remove descriptions from validation schemas
	// Default: false
	//
	// NOTE: I'd prefer a positive field name (ie includeDescriptions)
	//	but I wanted to avoid changing the default behavior
	RemoveDescriptionsFromSchema bool

	// Whether to assign Enum fields the `x-kubernetes-int-or-string` property
	// which allows the value to either be an integer or a string
	// If this is false, only string values are allowed
	// Default: false
	EnumAsIntOrString bool

	// A list of messages (core.solo.io.Status) whose validation schema should
	// not be generated
	MessagesWithEmptySchema []string

	// Whether to exclude kubebuilder markers and validations (such as PreserveUnknownFields, MinItems, default, and all CEL rules)
	// Type and Required markers will be included regardless
	// Default: false
	DisableKubeMarkers bool
}