clustercache

Documentation

Overview

Package clustercache implements the ClusterCache.

The ClusterCache is used to create and cache clients, REST configs, etc. for workload clusters.

Setup

ClusterCache can be set up via the SetupWithManager function. It will add the ClusterCache as a reconciler to the Manager and then return a ClusterCache object.

Usage

Typically the ClusterCache is passed to controllers and then it can be used by e.g. simply calling GetClient to retrieve a client for the given Cluster (see the ClusterCache interface for more details). If GetClient does return an error, controllers should not requeue, requeue after or return an error and go in exponential backoff. Instead, they should register a source (via GetClusterSource) to receive reconnect events from the ClusterCache.

Implementation details

The ClusterCache internally runs a reconciler that: - tries to create a connection to a workload cluster

• if it fails, it will retry after roughly the ConnectionCreationRetryInterval

- if the connection is established it will run continuous health checking every HealthProbe.Interval.

• if the health checking fails more than HealthProbe.FailureThreshold times consecutively or if an unauthorized error occurs, the connection will be disconnected (a subsequent Reconcile will try to connect again)

- if other reconcilers (e.g. the Machine controller) got a source to watch for events, they will get notified if:

• a connect or disconnect happened
• the health probe didn't succeed for a certain amount of time (if the WatchForProbeFailure option was used)

Implementation considerations

Some considerations about the trade-offs that have been made:

• There is intentionally only one Reconciler that handles both the connect/disconnect and health checking. This is a lot easier from a locking perspective than trying to coordinate between multiple reconcilers.
• We are never holding the write lock on the clusterAccessor for an extended period of time (e.g. during connection creation, which can time out after a few seconds). This is important so that other reconcilers that are calling GetClient etc. are never blocked.

Index

• Variables
• type CacheOptions
• type CacheOptionsIndex
• type ClientCacheOptions
• type ClientOptions
• type ClusterCache
  • func NewFakeClusterCache(workloadClient client.Client, clusterKey client.ObjectKey, ...) ClusterCache
  • func SetupWithManager(ctx context.Context, mgr manager.Manager, options Options, ...) (ClusterCache, error)
• type GetClusterSourceOption
• type GetClusterSourceOptions
  • func (o *GetClusterSourceOptions) ApplyOptions(opts []GetClusterSourceOption) *GetClusterSourceOptions
• type Options
• type RESTClient
• type SourceWatcher
• type TypedWatcherOptions
• type WatchForProbeFailure
  • func (n WatchForProbeFailure) ApplyToGetClusterSourceOptions(opts *GetClusterSourceOptions)
• type Watcher
  • func NewWatcher[object client.Object, request comparable](options TypedWatcherOptions[object, request]) Watcher
• type WatcherOptions

Variables

var ErrClusterNotConnected = errors.New("connection to the workload cluster is down")

ErrClusterNotConnected is returned by the ClusterCache when e.g. a Client cannot be returned because there is no connection to the workload cluster.

var NodeProviderIDIndex = CacheOptionsIndex{
	Object:       &corev1.Node{},
	Field:        index.NodeProviderIDField,
	ExtractValue: index.NodeByProviderID,
}

NodeProviderIDIndex is used to index Nodes by ProviderID.

Types

type CacheOptions

type CacheOptions struct {
	// SyncPeriod is the sync period of the cache.
	SyncPeriod *time.Duration

	// ByObject restricts the cache's ListWatch to the desired fields per GVK at the specified object.
	ByObject map[client.Object]cache.ByObject

	// Indexes are the indexes added to the cache.
	Indexes []CacheOptionsIndex
}

CacheOptions are the cache options for the caches that are created per cluster.

type CacheOptionsIndex

type CacheOptionsIndex struct {
	// Object is the object for which the index is created.
	Object client.Object

	// Field is the field of the index that can later be used when selecting
	// objects with a field selector.
	Field string

	// ExtractValue is a func that extracts the index value from an object.
	ExtractValue client.IndexerFunc
}

CacheOptionsIndex is a index that is added to the cache.

type ClientCacheOptions

type ClientCacheOptions struct {
	// DisableFor is a list of objects that should never be read from the cache.
	// Get & List calls for objects configured here always result in a live lookup.
	DisableFor []client.Object
}

ClientCacheOptions are the cache options for the clients that are created per cluster.

type ClientOptions

type ClientOptions struct {
	// Timeout is the timeout used for the REST config, client and cache.
	// Defaults to 10s.
	Timeout time.Duration

	// QPS is the maximum queries per second from the controller client
	// to the Kubernetes API server of workload clusters.
	// It is used for the REST config, client and cache.
	// Defaults to 20.
	QPS float32

	// Burst is the maximum number of queries that should be allowed in
	// one burst from the controller client to the Kubernetes API server of workload clusters.
	// It is used for the REST config, client and cache.
	// Default 30.
	Burst int

	// UserAgent is the user agent used for the REST config, client and cache.
	UserAgent string

	// Cache are the cache options defining how clients should interact with the underlying cache.
	Cache ClientCacheOptions
}

ClientOptions are the client options for the clients that are created per cluster.

type ClusterCache

type ClusterCache interface {
	// GetClient returns a cached client for the given cluster.
	// If there is no connection to the workload cluster ErrClusterNotConnected will be returned.
	GetClient(ctx context.Context, cluster client.ObjectKey) (client.Client, error)

	// GetReader returns a cached read-only client for the given cluster.
	// If there is no connection to the workload cluster ErrClusterNotConnected will be returned.
	GetReader(ctx context.Context, cluster client.ObjectKey) (client.Reader, error)

	// GetRESTConfig returns a REST config for the given cluster.
	// If there is no connection to the workload cluster ErrClusterNotConnected will be returned.
	GetRESTConfig(ctx context.Context, cluster client.ObjectKey) (*rest.Config, error)

	// GetClientCertificatePrivateKey returns a private key that is generated once for a cluster
	// and can then be used to generate client certificates. This is e.g. used in KCP to generate a client
	// cert to communicate with etcd.
	// This private key is stored and cached in the ClusterCache because it's expensive to generate a new
	// private key in every single Reconcile.
	GetClientCertificatePrivateKey(ctx context.Context, cluster client.ObjectKey) (*rsa.PrivateKey, error)

	// Watch watches a workload cluster for events.
	// Each unique watch (by input.Name) is only added once after a Connect (otherwise we return early).
	// During a disconnect existing watches (i.e. informers) are shutdown when stopping the cache.
	// After a re-connect watches will be re-added (assuming the Watch method is called again).
	// If there is no connection to the workload cluster ErrClusterNotConnected will be returned.
	Watch(ctx context.Context, cluster client.ObjectKey, watcher Watcher) error

	// GetLastProbeSuccessTimestamp returns the time when the health probe was successfully executed last.
	GetLastProbeSuccessTimestamp(ctx context.Context, cluster client.ObjectKey) time.Time

	// GetClusterSource returns a Source of Cluster events.
	// The mapFunc will be used to map from Cluster to reconcile.Request.
	// reconcile.Requests will always be enqueued on connect and disconnect.
	// Additionally the WatchForProbeFailure(time.Duration) option can be used to enqueue reconcile.Requests
	// if the health probe didn't succeed for the configured duration.
	// Note: GetClusterSource would ideally take a mapFunc that has a *clusterv1.Cluster instead of a client.Object
	// as a parameter, but then the existing mapFuncs we already use in our Reconcilers wouldn't work and we would
	// have to implement new ones.
	GetClusterSource(controllerName string, mapFunc func(ctx context.Context, cluster client.Object) []ctrl.Request, opts ...GetClusterSourceOption) source.Source
}

ClusterCache is a component that caches clients, caches etc. for workload clusters.

func NewFakeClusterCache

func NewFakeClusterCache(workloadClient client.Client, clusterKey client.ObjectKey, watchObjects ...string) ClusterCache

NewFakeClusterCache creates a new fake ClusterCache that can be used by unit tests.

func SetupWithManager

func SetupWithManager(ctx context.Context, mgr manager.Manager, options Options, controllerOptions controller.Options) (ClusterCache, error)

SetupWithManager sets up a ClusterCache with the given Manager and Options. This will add a reconciler to the Manager and returns a ClusterCache which can be used to retrieve e.g. Clients for a given Cluster.

type GetClusterSourceOption

type GetClusterSourceOption interface {
	// ApplyToGetClusterSourceOptions applies this option to the given GetClusterSourceOptions.
	ApplyToGetClusterSourceOptions(option *GetClusterSourceOptions)
}

GetClusterSourceOption is an option that modifies GetClusterSourceOptions for a GetClusterSource call.

type GetClusterSourceOptions

type GetClusterSourceOptions struct {
	// contains filtered or unexported fields
}

GetClusterSourceOptions allows to set options for the GetClusterSource method.

func (*GetClusterSourceOptions) ApplyOptions

func (o *GetClusterSourceOptions) ApplyOptions(opts []GetClusterSourceOption) *GetClusterSourceOptions

ApplyOptions applies the passed list of options to GetClusterSourceOptions, and then returns itself for convenient chaining.

type Options

type Options struct {
	// SecretClient is the client used to access secrets of type secret.Kubeconfig, i.e. Secrets
	// with the following name format: "<cluster-name>-kubeconfig".
	// Ideally this is a client that caches only kubeconfig secrets, it is highly recommended to avoid caching all secrets.
	// An example on how to create an ideal secret caching client can be found in the core Cluster API controller main.go file.
	SecretClient client.Reader

	// WatchFilterValue is the label value used to filter events prior to reconciliation.
	// If a filter excludes a cluster from reconciliation, the accessors for this cluster
	// will never be created.
	WatchFilterValue string

	// Cache are the cache options for the caches that are created per cluster.
	Cache CacheOptions

	// Client are the client options for the clients that are created per cluster.
	Client ClientOptions
}

Options defines the options to configure a ClusterCache.

type RESTClient

type RESTClient interface {
	Get() *rest.Request
}

RESTClient is an interface only containing the methods of *rest.RESTClient that we use. Using this interface instead of *rest.RESTClient makes it possible to use the fake.RESTClient in unit tests.

type SourceWatcher

type SourceWatcher[request comparable] interface {
	Watch(src source.TypedSource[request]) error
}

SourceWatcher is a scoped-down interface from Controller that only has the Watch func.

type TypedWatcherOptions

type TypedWatcherOptions[object client.Object, request comparable] struct {
	// Name represents a unique Watch request for the specified Cluster.
	// The name is used to track that a specific watch is only added once to a cache.
	// After a connection (and thus also the cache) has been re-created, watches have to be added
	// again by calling the Watch method again.
	Name string

	// Watcher is the watcher (controller) whose Reconcile() function will be called for events.
	Watcher SourceWatcher[request]

	// Kind is the type of resource to watch.
	Kind object

	// EventHandler contains the event handlers to invoke for resource events.
	EventHandler handler.TypedEventHandler[object, request]

	// Predicates is used to filter resource events.
	Predicates []predicate.TypedPredicate[object]
}

TypedWatcherOptions specifies the parameters used to establish a new watch for a workload cluster. A source.TypedKind source (configured with Kind, TypedEventHandler and Predicates) will be added to the Watcher. To watch for events, the source.TypedKind will create an informer on the Cache that we have created and cached for the given Cluster.

type WatchForProbeFailure

type WatchForProbeFailure time.Duration

WatchForProbeFailure will configure the Cluster source to enqueue reconcile.Requests if the health probe didn't succeed for the configured duration. For example if WatchForProbeFailure is set to 5m, an event will be sent if LastProbeSuccessTimestamp is 5m in the past (i.e. health probes didn't succeed in the last 5m).

func (WatchForProbeFailure) ApplyToGetClusterSourceOptions

func (n WatchForProbeFailure) ApplyToGetClusterSourceOptions(opts *GetClusterSourceOptions)

ApplyToGetClusterSourceOptions applies WatchForProbeFailure to the given GetClusterSourceOptions.

type Watcher

type Watcher interface {
	Name() string
	Object() client.Object
	Watch(cache cache.Cache) error
}

Watcher is an interface that can start a Watch.

func NewWatcher

func NewWatcher[object client.Object, request comparable](options TypedWatcherOptions[object, request]) Watcher

NewWatcher creates a Watcher for the workload cluster. A source.TypedKind source (configured with Kind, TypedEventHandler and Predicates) will be added to the SourceWatcher. To watch for events, the source.TypedKind will create an informer on the Cache that we have created and cached for the given Cluster.

type WatcherOptions

type WatcherOptions = TypedWatcherOptions[client.Object, ctrl.Request]

WatcherOptions specifies the parameters used to establish a new watch for a workload cluster. A source.TypedKind source (configured with Kind, TypedEventHandler and Predicates) will be added to the Watcher. To watch for events, the source.TypedKind will create an informer on the Cache that we have created and cached for the given Cluster.