istioagent

package
v1.0.0-rc2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2023 License: Apache-2.0 Imports: 64 Imported by: 0

README

Istio Agent

This document describes the internal architecture of Istio agent.

High Level overview

High Level overview

At a high level, the Istio agent acts as an intermediate proxy between Istiod and Envoy. This is done at two levels. For distributing workload certificates, Envoy will send SDS requests to the agent, causing the agent to submit a CSR to the configured CA (generally Istiod). For other configuration, Envoy will send ADS requests to the agent, which will be forwarded to the configured discovery server (general Istiod).

CA Flow

istio-agent checks the presence of a socket file on the defined socket path /var/run/secrets/workload-spiffe-uds/socket.

  • If a socket is found, istio-agent will not start its own SDS Server and Envoy will be configured to use that socket as its source of cryptographic material.
  • If a socket is not found, istio-agent then checks whether certificate files are present or not on the defined certificate path /var/run/secrets/workload-spiffe-credentials. If certificate files are found, istio-agent will start its own SDS Server, listening and serving these certificates on the defined socket path /var/run/secrets/workload-spiffe-uds/socket, while also keeping file watchers on them. Envoy proxy then connects to istio-agent's SDS Server through the defined socket path and gets the cryptographic materials of the certificate files served by the SDS API.
  • If istio-agent does not find either the socket, or the certificate files in their respective paths it will start its own SDS Server using a caClient to connect to istiod or an external CA, to fetch cryptographic materials (See Default CA Flow).

SDS decision flow

Default CA Flow through istio-agent

CA Flow

A single SDS request from Envoy goes through a few different layers in istio-agent.

  1. First, the request is handled by the SDS server. This is mostly just an intermediate translation layer exposing the SecretManager to Envoy, without much business logic. For each resource requested by Envoy, the SDS server will call SecretManager.GenerateSecret(resourceName)
  2. When GenerateSecret is called, the SecretManager is expected to return a new certificate. This can occur in a few ways.
    1. The most common method (pictured above) is to sign a new certificate by calling the configured CA. Typically, this is Istiod.
    2. If the certificate is not yet expired, the SecretManager also can return a cache response. In practice, this would only happen in cases where Envoy were to re-request a resource, which is fairly rare.
    3. SecretManager can also read certificates from files. When this is configured, no CA client is used.
  3. The caClient will be configured to use either JWT or mTLS authentication. For JWT authentication, gRPC's PerRPCCredentials is configured with a TokenProvider which handles the logic of adding the proper JWT to each request. mTLS is configured by a tls.Config that points to files on disk.

It should be noted there is a circular dependency with mTLS authentication; in order to fetch a certificate we need a certificate. This can be handled in various ways:

  • GenerateSecret may additionally write any signed certificates to disk, with OUTPUT_CERTS configured.
  • Users may have external CA setups that pre-configure certificates.
  • The CaClient can use JWT token for the initial setup, then switch to mTLS certificates.

Note that OUTPUT_CERTS can be used to refresh certificates using previously provisioned certificates, by configuring the ca client to use certificates written to the same directory we have configured them to be written to.

Authentication

The agent supports two forms of authentication with the CA/discovery servers: mTLS and JWT. Varying deployment topologies mix and match these two.

For a standard Kubernetes deployment, both CA and discovery will use JWT authentication, with a token automatically generated and rotated by Kubernetes.

For discovery, the JWT token will be read directly from a file and sent as is. For CA, this logic is a bit more complex, as the support for external CAs is more mature than external discovery servers. This supports some additional configuration, a CredentialFetcher which allows fetching a token from places other than a file (for example, a local metadata server), and a TokenExchanger which allows exchanging a token for another form to match the CA server requirements.

For VMs, the standard flow is for the user to provision a short-lived JWT token onto the VM. After the initial CSR, certificates are written to disk and mTLS is used for future requests. If the VM restarted, it would continue to use the certificates written to disk, assuming the downtime is less than certificate expiration. This is why the certificates are persisted to disk, rather than kept in memory like in the standard Kubernetes deployment.

Certificate Rotation

The agent also handles rotating certificates near expiration. It does so by triggering a callback from the SecretManager to the SDS server when a certificate is near expiration (configurable by SECRET_GRACE_PERIOD_RATIO, defaulting to half of the expiration). If the SDS server is still interested in this certificate (ie, Envoy is still connected and requesting the certificate), the SDS server will send another request to generate a new secret and push the updated certificate to Envoy. This ensures that we do not permanently watch certificates even after Envoy has stopped requested them; if there are no subscriptions they update will be ignored. If Envoy later watches these certificates again, a new one will be generated on demand.

Configuration

Variable Description
CA_ADDR Address of CA, defaults to discoveryAddress
CA_PROVIDER Type of CA; supported values are GoogleCA or Citadel (although anything but GoogleCA will use Citadel); defaults to Citadel
PROV_CERT certificates to be used for mTLS communication with control plane only; NOT for workload mTLS
OUTPUT_CERT write all fetched certificates to some directory. Used to support applications that need certificates (Prometheus) as well as rotating mTLS control plane authentication.
FILE_MOUNTED_CERTS completely disable CA path, exclusively use certs mounted into the pod with set certificate file locations
CREDENTIAL_FETCHER_TYPE allows using custom credential fetcher, for VMs with existing identity
CREDENTIAL_IDENTITY_PROVIDER just used to control the audience for VMs with existing identity
PROXY_XDS_VIA_AGENT use istio-agent to proxy XDS. True for all use cases now, likely can be always-on now or soon
PROXY_XDS_DEBUG_VIA_AGENT Offer XDS istio.io/debug API on agent's 15004 HTTP endpoint. (Requires PROXY_XDS_VIA_AGENT)
{XDS,CA}_ROOT_CA explicitly configure root certificate path
PILOT_CERT_PROVIDER just used to determine XDS/CA root certificate; redundant with {XDS,CA}_ROOT_CA.

Documentation

Index

Constants

View Source
const (
	MetadataClientCertKey   = "ISTIO_META_TLS_CLIENT_KEY"
	MetadataClientCertChain = "ISTIO_META_TLS_CLIENT_CERT_CHAIN"
	MetadataClientRootCert  = "ISTIO_META_TLS_CLIENT_ROOT_CERT"
)
View Source
const (

	// CitadelCACertPath is the directory for Citadel CA certificate.
	// This is mounted from config map 'istio-ca-root-cert'. Part of startup,
	// this may be replaced with ./etc/certs, if a root-cert.pem is found, to
	// handle secrets mounted from non-citadel CAs.
	CitadelCACertPath = "./var/run/secrets/istio"
)

To debug: curl -X POST localhost:15000/logging?config=trace - to see SendingDiscoveryRequest Breakpoints in secretcache.go GenerateSecret.. Note that istiod currently can't validate the JWT token unless it runs on k8s Main problem is the JWT validation check which hardcodes the k8s server address and token location.

To test on a local machine, for debugging:

kis exec $POD -- cat /run/secrets/istio-token/istio-token > var/run/secrets/tokens/istio-token kis port-forward $POD 15010:15010 &

You can also copy the K8S CA and a token to be used to connect to k8s - but will need removing the hardcoded addr kis exec $POD -- cat /run/secrets/kubernetes.io/serviceaccount/{ca.crt,token} > var/run/secrets/kubernetes.io/serviceaccount/

Or disable the jwt validation while debugging SDS problems.

Variables

This section is empty.

Functions

This section is empty.

Types

type Agent

type Agent struct {
	// contains filtered or unexported fields
}

Agent contains the configuration of the agent, based on the injected environment: - SDS hostPath if node-agent was used - /etc/certs/key if Citadel or other mounted Secrets are used - root cert to use for connecting to XDS server - CA address, with proper defaults and detection

func NewAgent

func NewAgent(proxyConfig *mesh.ProxyConfig, agentOpts *AgentOptions, sopts *security.Options, eopts envoy.ProxyConfig) *Agent

NewAgent hosts the functionality for local SDS and XDS. This consists of the local SDS server and associated clients to sign certificates (when not using files), and the local XDS proxy (including health checking for VMs and DNS proxying).

func (*Agent) Check

func (a *Agent) Check() (err error)

func (*Agent) Close

func (a *Agent) Close()

func (*Agent) EnvoyDisabled

func (a *Agent) EnvoyDisabled() bool

EnvoyDisabled if true indicates calling Run will not run and wait for Envoy.

func (*Agent) FindRootCAForCA

func (a *Agent) FindRootCAForCA() (string, error)

FindRootCAForCA Find the root CA to use when connecting to the CA (Istiod or external).

func (*Agent) FindRootCAForXDS

func (a *Agent) FindRootCAForXDS() (string, error)

FindRootCAForXDS determines the root CA to be configured in bootstrap file. It may be different from the CA for the cert server - which is based on CA_ADDR In addition it deals with the case the XDS server is on port 443, expected with a proper cert. /etc/ssl/certs/ca-certificates.crt

func (*Agent) GRPCBootstrapPath

func (a *Agent) GRPCBootstrapPath() string

GRPCBootstrapPath returns the most recently generated gRPC bootstrap or nil if there is none.

func (*Agent) GetDNSTable

func (a *Agent) GetDNSTable() *dnsProto.NameTable

func (*Agent) GetKeyCertsForXDS

func (a *Agent) GetKeyCertsForXDS() (string, string)

GetKeyCertsForXDS return the key cert files path for connecting with xds.

func (*Agent) Run

func (a *Agent) Run(ctx context.Context) (func(), error)

Run is a non-blocking call which returns either an error or a function to await for completion.

func (*Agent) WaitForSigterm

func (a *Agent) WaitForSigterm() bool

WaitForSigterm if true indicates calling Run will block until SIGTERM or SIGNT is received.

type AgentOptions

type AgentOptions struct {
	// ProxyXDSDebugViaAgent if true will listen on 15004 and forward queries
	// to XDS istio.io/debug. (Requires ProxyXDSViaAgent).
	ProxyXDSDebugViaAgent bool
	// Port value for the debugging endpoint.
	ProxyXDSDebugViaAgentPort int
	// DNSCapture indicates if the XDS proxy has dns capture enabled or not
	// This option will not be considered if proxyXDSViaAgent is false.
	DNSCapture bool
	// DNSAddr is the DNS capture address
	DNSAddr string
	// ProxyType is the type of proxy we are configured to handle
	ProxyType model.NodeType
	// ProxyNamespace to use for local dns resolution
	ProxyNamespace string
	// ProxyDomain is the DNS domain associated with the proxy (assumed
	// to include the namespace as well) (for local dns resolution)
	ProxyDomain string
	// Node identifier used by Envoy
	ServiceNode string

	// XDSRootCerts is the location of the root CA for the XDS connection. Used for setting platform certs or
	// using custom roots.
	XDSRootCerts string

	// CARootCerts of the location of the root CA for the CA connection. Used for setting platform certs or
	// using custom roots.
	CARootCerts string

	// Extra headers to add to the XDS connection.
	XDSHeaders map[string]string

	// Is the proxy an IPv6 proxy
	IsIPv6 bool

	// Path to local UDS to communicate with Envoy
	XdsUdsPath string

	// Ability to retrieve ProxyConfig dynamically through XDS
	EnableDynamicProxyConfig bool

	// All of the proxy's IP Addresses
	ProxyIPAddresses []string

	// Enables dynamic generation of bootstrap.
	EnableDynamicBootstrap bool

	// Envoy status port (that circles back to the agent status port). Really belongs to the proxy config.
	// Cannot be eradicated because mistakes have been made.
	EnvoyStatusPort int

	// Envoy prometheus port that circles back to its admin port for prom endpoint. Really belongs to the
	// proxy config.
	EnvoyPrometheusPort int

	MinimumDrainDuration time.Duration

	ExitOnZeroActiveConnections bool

	// Cloud platform
	Platform platform.Environment

	// GRPCBootstrapPath if set will generate a file compatible with GRPC_XDS_BOOTSTRAP
	GRPCBootstrapPath string

	// Disables all envoy agent features
	DisableEnvoy          bool
	DownstreamGrpcOptions []grpc.ServerOption

	IstiodSAN string

	WASMInsecureRegistries []string
}

AgentOptions contains additional config for the agent, not included in ProxyConfig. Most are from env variables ( still experimental ) or for testing only. Eventually most non-test settings should graduate to ProxyConfig Please don't add 100 parameters to the NewAgent function (or any other)!

type ProxyConnection

type ProxyConnection struct {
	// contains filtered or unexported fields
}

type ResponseHandler

type ResponseHandler func(resp *any.Any) error

ResponseHandler handles a XDS response in the agent. These will not be forwarded to Envoy. Currently, all handlers function on a single resource per type, so the API only exposes one resource.

type XdsProxy

type XdsProxy struct {
	// contains filtered or unexported fields
}

XDS Proxy proxies all XDS requests from envoy to istiod, in addition to allowing subsystems inside the agent to also communicate with either istiod/envoy (eg dns, sds, etc). The goal here is to consolidate all xds related connections to istiod/envoy into a single tcp connection with multiple gRPC streams. TODO: Right now, the workloadSDS server and gatewaySDS servers are still separate connections. These need to be consolidated. TODO: consolidate/use ADSC struct - a lot of duplication.

func (*XdsProxy) DeltaAggregatedResources

requests from envoy for aditya: downstream -> envoy (anything "behind" xds proxy) upstream -> istiod (in front of xds proxy)?

func (*XdsProxy) HandleDeltaUpstream

func (*XdsProxy) HandleUpstream

func (*XdsProxy) InitIstiodDialOptions

func (p *XdsProxy) InitIstiodDialOptions(agent *Agent) error

func (*XdsProxy) PersistDeltaRequest

func (p *XdsProxy) PersistDeltaRequest(req *discovery.DeltaDiscoveryRequest)

func (*XdsProxy) PersistRequest

func (p *XdsProxy) PersistRequest(req *discovery.DiscoveryRequest)

PersistRequest sends a request to the currently connected proxy. Additionally, on any reconnection to the upstream XDS request we will resend this request.

func (*XdsProxy) RegisterStream

func (p *XdsProxy) RegisterStream(c *ProxyConnection)

func (*XdsProxy) StreamAggregatedResources

Every time envoy makes a fresh connection to the agent, we reestablish a new connection to the upstream xds This ensures that a new connection between istiod and agent doesn't end up consuming pending messages from envoy as the new connection may not go to the same istiod. Vice versa case also applies.

func (*XdsProxy) UnregisterStream

func (p *XdsProxy) UnregisterStream(c *ProxyConnection)

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL