xds

README

xDS

Background

xDS is the set of discovery services and APIs used by Envoy to discover its dynamic resources.

xDS Server

Gloo Edge is an xDS server. It maintains a snapshot-based, in-memory cache and responds to xDS requests with the resources that are requested.

Snapshot

A snapshot is a versioned group of resources. In Gloo Edge, we rely on an Envoy snapshot, a snapshot of the xDS resources that Gloo serves to Envoy.

Snapshot Cache

A snapshot cache maintains a single versioned snapshot per key. It also responds to open xDS requests.

xDS Callbacks

xDS callbacks are a set of callbacks that are invoked asynchronously during the lifecycle of an xDS request.

Gloo Edge open source does not define any xDS callbacks. However, these callbacks are a type of extension that can be injected at runtime. Gloo Edge Enterprise defines xDS callbacks, and injects them into the Control Plane at runtime.

Server

An xDS server defines a set of handlers for streaming discovery requests.

xDS Services

The xDS server is configured to expose the following discovery services in Gloo Edge:

ListenerDiscoveryService

The ListenerDiscoveryService allows Envoy to discover Listeners at runtime.

RouteDiscoveryService

The RouteDiscoveryService allows Envoy to discover routing configuration for an HttpConnectionManager filter at runtime.

ClusterDiscoveryService

The ClusterDiscoveryService allows Envoy to discover routable destinations at runtime.

EndpointDiscoveryService

The EndpointDiscoveryService allows Envoy to discover members in a cluster at runtime.

AggregatedDiscoveryService

The AggregatedDiscoveryService allows Envoy to discover all resource types over a single stream at runtime.

SoloDiscoveryService

The SoloDiscoveryService is a custom xDS service, used to serve resources of Any type, that is based on Envoy's Aggregated Discovery Service.

In addition to serving configuration for Envoy resources, the Gloo xDS server is also responsible for serving configuration to a number of enterprise extensions (ie ext-auth and rate-limit)

The SoloDiscoveryService is required to serve these extension resources. It is largely based on the Envoy v2 API, and since it is purely an internal API, we do not need to upgrade the API to match the Envoy xDS API. This issue contains additional context around the reason behind this custom discovery service.

xDS Requests

Gloo Edge supports managing configuration for multiple proxies through a single xDS server. To do so, it stores each snapshot in the cache at a key that is unique to that proxy.

To guarantee that proxies initiate requests for the snapshot they want, it is critical that we establish a naming pattern for cache keys. This pattern must be used both by the proxies requesting the resources from the cache, and the controllers that set the resources in the cache.

The naming convention that we follow is "NAMESPACE~NAME"

Proxies identify the cache key that they are interested in by specifying their node.metadata.role to the cache key using the above naming pattern. An example of this can be found in the bootstrap configuration for proxies

xDS Debugging

Debugging xDS behavior can be challenging, below are a few techniques to help:

Control Plane Logging

The Gloo translation loop is responsible for converting a Gloo API Snapshot into an xDS Snapshot. After completing a translation loop, there is a log line indicating what key in the snapshot cache was updated.

xDS Debug Endpoint

Gloo supports running in dev mode, and when that is enabled, xDS Snapshots are exposed via an endpoint.

Useful information

• Hoot YouTube series about xDS
• Envoy xDS
• Hoot Repository

Documentation

Overview

Package server provides an implementation of a streaming xDS server.

Package server provides an implementation of a streaming xDS server.

Index

• Constants
• func GetValidKeys(proxies v1.ProxyList, extensionKeys map[string]struct{}) []string
• func NewEndpointsSnapshotFromResources(endpoints cache.Resources, clusters cache.Resources) cache.Snapshot
• func NewSnapshotFromResources(endpoints cache.Resources, clusters cache.Resources, routes cache.Resources, ...) cache.Snapshot
• func SetEdsOnCluster(out *envoy_config_cluster_v3.Cluster, settings *v1.Settings)
• func SetupEnvoyXds(grpcServer *grpc.Server, xdsServer envoyserver.Server, ...)
• func SnapshotKey(proxy *v1.Proxy) string
• type EnvoyServerV3
  • func NewEnvoyServerV3(genericServer server.Server) EnvoyServerV3
• type EnvoySnapshot
  • func NewSnapshot(version string, endpoints []cache.Resource, clusters []cache.Resource, ...) *EnvoySnapshot
  • func (s *EnvoySnapshot) Clone() cache.Snapshot
  • func (s *EnvoySnapshot) Consistent() error
  • func (this *EnvoySnapshot) Equal(that *EnvoySnapshot) bool
  • func (s *EnvoySnapshot) GetResources(typ string) cache.Resources
  • func (s *EnvoySnapshot) MakeConsistent()
• type GlooXdsServer
  • func NewGlooXdsServer(genericServer server.Server) GlooXdsServer
• type ProxyKeyHasher
  • func NewNodeHasher() *ProxyKeyHasher
  • func (h *ProxyKeyHasher) ID(node *envoy_config_core_v3.Node) string

Constants

const FallbackNodeKey = "misconfigured-node"

used to let nodes know they have a bad config we assign a "fix me" snapshot for bad nodes

Functions

func GetValidKeys

func GetValidKeys(proxies v1.ProxyList, extensionKeys map[string]struct{}) []string

Called in Syncer when a new set of proxies arrive used to trim snapshots whose proxies have been deleted

func NewEndpointsSnapshotFromResources

func NewEndpointsSnapshotFromResources(
	endpoints cache.Resources,
	clusters cache.Resources,
) cache.Snapshot

func NewSnapshotFromResources

func NewSnapshotFromResources(
	endpoints cache.Resources,
	clusters cache.Resources,
	routes cache.Resources,
	listeners cache.Resources,
) cache.Snapshot

func SetEdsOnCluster

func SetEdsOnCluster(out *envoy_config_cluster_v3.Cluster, settings *v1.Settings)

func SetupEnvoyXds

func SetupEnvoyXds(grpcServer *grpc.Server, xdsServer envoyserver.Server, envoyCache envoycache.SnapshotCache)

register xDS methods with GRPC server

func SnapshotKey

func SnapshotKey(proxy *v1.Proxy) string

SnapshotKey of Proxy == Role in Envoy Configmap == "Node" in Envoy semantics

Types

type EnvoyServerV3

type EnvoyServerV3 interface {
	envoy_service_endpoint_v3.EndpointDiscoveryServiceServer
	envoy_service_cluster_v3.ClusterDiscoveryServiceServer
	envoy_service_route_v3.RouteDiscoveryServiceServer
	envoy_service_listener_v3.ListenerDiscoveryServiceServer
	envoy_service_discovery_v3.AggregatedDiscoveryServiceServer
}

Server is a collection of handlers for streaming discovery requests.

func NewEnvoyServerV3

func NewEnvoyServerV3(genericServer server.Server) EnvoyServerV3

NewServer creates handlers from a config watcher and an optional logger.

type EnvoySnapshot

type EnvoySnapshot struct {
	// Endpoints are items in the EDS V3 response payload.
	Endpoints cache.Resources

	// Clusters are items in the CDS response payload.
	Clusters cache.Resources

	// Routes are items in the RDS response payload.
	Routes cache.Resources

	// Listeners are items in the LDS response payload.
	Listeners cache.Resources
}

Snapshot is an internally consistent snapshot of xDS resources. Consistently is important for the convergence as different resource types from the snapshot may be delivered to the proxy in arbitrary order.

func NewSnapshot

func NewSnapshot(
	version string,
	endpoints []cache.Resource,
	clusters []cache.Resource,
	routes []cache.Resource,
	listeners []cache.Resource,
) *EnvoySnapshot

NewSnapshot creates a snapshot from response types and a version.

func (*EnvoySnapshot) Clone

func (s *EnvoySnapshot) Clone() cache.Snapshot

func (*EnvoySnapshot) Consistent

func (s *EnvoySnapshot) Consistent() error

Consistent check verifies that the dependent resources are exactly listed in the snapshot: - all EDS resources are listed by name in CDS resources - all RDS resources are listed by name in LDS resources

Note that clusters and listeners are requested without name references, so Envoy will accept the snapshot list of clusters as-is even if it does not match all references found in xDS.

func (*EnvoySnapshot) Equal

func (this *EnvoySnapshot) Equal(that *EnvoySnapshot) bool

Equal checks is 2 snapshots are equal, important since reflect.DeepEqual no longer works with proto4

func (*EnvoySnapshot) GetResources

func (s *EnvoySnapshot) GetResources(typ string) cache.Resources

GetResources selects snapshot resources by type.

func (*EnvoySnapshot) MakeConsistent

func (s *EnvoySnapshot) MakeConsistent()

MakeConsistent removes any items that fail to link to parent resources in the snapshot. It will also add placeholder routes for listeners referencing non-existent routes.

type GlooXdsServer

type GlooXdsServer interface {
	discovery_service.SoloDiscoveryServiceServer
}

Server includes handlers for streaming aggregate discovery requests via the SoloDiscoveryService.

func NewGlooXdsServer

func NewGlooXdsServer(genericServer server.Server) GlooXdsServer

type ProxyKeyHasher

type ProxyKeyHasher struct{}

Returns the node.metadata.role from the envoy bootstrap config if not found, it returns a key for the Fallback snapshot which alerts the user their Envoy is missing the required role key.

func NewNodeHasher

func NewNodeHasher() *ProxyKeyHasher

func (*ProxyKeyHasher) ID

func (h *ProxyKeyHasher) ID(node *envoy_config_core_v3.Node) string