mk8s

README

Mesh / MultiCluster K8S helpers

This repo includes examples and helpers for using K8S in a (hopefully) simpler and more efficient way.

Using informers, operators, controllers and the officially recommended tools remains the recommended approach for most cases, and are widely documented and used.

This project is focused on a few specific use cases and optimizations:

1. Istio-like multicluster - where an app works with a set of clusters
2. Large sets of data where we don't want the entire set in memory, as is the case with informers.
3. Caching data on disk, for fast startup and recovery, and operating in pubsub-like mode.

Wrappers for K8S client loading

The intent is to support apps that operate on multiple clusters - like Istio multicluster - but with more flexibility.

For example the 'gcp' module will load all GKE clusters in the project or hub. More clusters can be loaded using Istio Secrets.

The default init logic is:

• load KUBECONFIG if set explicitly
• else, load ~/.kube/config - and load ALL contexts
• else, load in-cluster

In the end, Default cluster will be set if at least one cluster is available.

Using the default cluster we can get JWT tokens and use them to access GKE and Hub to load more clusters as needed.

// instead of:
restCfg, err := clientcmd.BuildConfigFromFlags(apiServerURL, kubeConfig)
client, err := kubernetes.NewForConfig(config)

// use:

k, err := k8s.New(ctx, nil)
client, err := kubernetes.NewForConfig(k.Default.RestConfig)

Terms

K8S defines or re-defines many words and concepts. To avoid confusion, I like to make clear what they mean (to me) and how they map to common concepts.

• Api-server - this is the main k8s server, containing many modules and functions.
  • 'Config database' is one of the core functions - actually implemented by etcd or a Kine-supported real database providing a basic 'no-SQL' storage. Storage is typical, but not required, in-memory or dynamic is also valid.
  • CRD is an OpenAPI description of a type, or a schema. Can also be viewed as a 'table' in SQL. Typically stored in the database - but it doesn't have to.
  • 'CR' or 'resource' is a row in the table - an individual object conforming to the CRD schema.
  • Names: each CR can be identified by a namespace and a name (except cluster-scoped resources - no namespace). The FQDN/URL also includes the cluster location.
  • 'revision' is one of the most important concepts, it allows tracking changes to a resource. Each resource has the last revision - but older revisions may also be around. Normally K8S deletes the old revision - a recovery system may keep track of them, like in git.
  • watch - pubsub like mechanism to get real-time updates.
• ServiceAccount and RBAC: K8S wraps a config database and adds access control to each object.
• Gateway and API server. K8S provide a proxy service to other HTTP/2 and websocket services.
• Running containers - what most people associate with K8S is actually handled by a CRI (runtime) like docker. This project is NOT concerned with containers, only with K8S functions as a config database and API gateway.

Using K8S rest client directly

Informers and regular K8S 'controllers' are based on a cached replica of type. The informers can filter the data they cache using namespace and label selectors - but the entire object is sent to the client as json (or protobuf for core objects).

After retrieving the full CR, the informers may build additional indexes and may drop parts of the response to reduce memory use.

The common pattern is at startup to retrieve all the CRs of all types that are needed, and after getting all the data start watching for changes.

Using generated 'clientset' directly

Code organization

K8S

This is the main package used to configure the K8S rest client.

GKE

The github.com/costinm/mk8s/gke package includes a small library to autoconfigure K8S clients based on the GKE and Fleet APIs and using a metadata server for tokens.

gke-gcloud-auth-plugin and gcloud remains the best option for using GKE with kubectl and for most other purposes.

Apis and generated client

The repo includes a number of tools and examples. All CRD are generated into manifests/crds, clientset in client/ and informers/listers in cachedclient.

Documentation

Index

• func Get[T interface{}]()
• func Is404(err error) bool
• func List[T interface{}](func(T) bool)
• func SetK8SLogging(flags string)
• type K8S
  • func Default() *K8S
  • func New(ctx context.Context, ns, name string) (*K8S, error)
  • func NewK8SSet(ctx context.Context, ns, name string) *K8S
  • func (kr *K8S) GetToken(ctx context.Context, aud string) (string, error)
  • func (kr *K8S) LoadKubeConfig(configFile string) error
• type K8SClient
• type K8SCluster
  • func NewK8SCluster(ctx context.Context, ns, name string) *K8SCluster
  • func (kr *K8SCluster) Client() *kubernetes.Clientset
  • func (k *K8SCluster) GcpInfo() (string, string, string)
  • func (kr *K8SCluster) GetCM(ctx context.Context, ns string, name string) (map[string]string, error)
  • func (kr *K8SCluster) GetSecret(ctx context.Context, ns string, name string) (map[string][]byte, error)
  • func (k *K8SCluster) GetToken(ctx context.Context, aud string) (string, error)
  • func (k *K8SCluster) GetTokenRaw(ctx context.Context, ns, ksa, aud string) (string, error)
  • func (k *K8SCluster) Label(ctx context.Context, name string) string
  • func (k *K8SCluster) Location() string
  • func (kr *K8SCluster) RestClient(group string, version string) (*rest.RESTClient, error)
  • func (kr *K8SCluster) RunAs(ns, ksa string) *K8SCluster
  • func (kr *K8SCluster) String() string
• type K8SRest

Functions

func Get

func Get[T interface{}]()

func Is404

func Is404(err error) bool

func List

func List[T interface{}](func(T) bool)

func SetK8SLogging

func SetK8SLogging(flags string)

Init klog.InitFlags from an env (to avoid messing with the CLI of the app). For example -v=9 lists full request content, -v=7 lists requests headers TODO: integrate with the dynamic logging.

Types

type K8S

type K8S struct {
	// Namespace to use by default
	Namespace string

	// KSA to use by default for getting tokens.
	KSA string

	// QPS overrides the default 5 QPS in the client.
	QPS float32

	// Burst overrides the default 10 burst in the client.
	Burst int

	// Primary config cluster - current context in config, in-cluster
	// picked by config
	Default *K8SCluster

	// LoadKubeConfig will populate this from a kubeconfig file,
	// followed optionally by GKE or other sources.
	ByName map[string]*K8SCluster
}

K8S implements the common interface for a set of K8S APIservers or servers implementing K8S patterns.

func Default

func Default() *K8S

func New

func New(ctx context.Context, ns, name string) (*K8S, error)

NewK8S will initialize a K8S cluster set.

If running in cluster, the 'local' cluster will be the default. Additional clusters can be loaded from istio kubeconfig files, kubeconfig, GKE, Fleet.

func NewK8SSet

func NewK8SSet(ctx context.Context, ns, name string) *K8S

func (*K8S) GetToken

func (kr *K8S) GetToken(ctx context.Context, aud string) (string, error)

GetToken returns a token with the given audience for the default KSA, using CreateToken request. Used by the STS token exchanger.

func (*K8S) LoadKubeConfig

func (kr *K8S) LoadKubeConfig(configFile string) error

LoadKubeConfig loads a set of clusters defined using kube config:

• explicitly set as param
• KUBECONFIG
• $HOME/.kube/config

error is set if KUBECONFIG is set or ~/.kube/config exists but fail to load. If the file doesn't exist, err is nil and nothing is loaded.

type K8SClient

type K8SClient[T interface{}] struct {
}

K8SClient is a typed wrapper for a REST client for a specific resource.

type K8SCluster

type K8SCluster struct {
	// RestConfig is the main config for creating rest clients using generated libraries.
	// The pattern is CLIENTPKG.NewForConfig(RestConfig) returning a Clientset with all generated types.
	//
	// dynamic is an exception - returns a dynamic client.
	//
	// The URL can be extracted with rest.DefaultServerURLFor(RestConfig)
	// Http client properly configured to talk with K8SAPIserver directly: rest.HTTPClientFor(RestConfig)
	RestConfig *rest.Config `json:-`

	// The name should be mangled - gke_PROJECT_LOCATION_NAME or connectgateway_PROJECT_NAME
	// or hostname.
	// Best practice: fleet name, also part of the domain suffix
	// Using the VENDOR_PROJECT_REGION_NAME for all would also be nice.
	Name string

	// The default and loaded clusters get namespace from config.
	// It is possible to clone the cluster and use a different namespace.
	// It is used for Token and other helper methods - using the client set
	// allows arbitrary namespaces.
	Namespace string

	// This is used to customize the K8S ServiceAccount used in GetToken requests.
	// K8SCluster implements interfaces to get JWTs signed by K8S, assuming the
	// principal defined in the config has the RBAC permissions to create tokens for
	// this KSA. If not set - default SA will be used.
	KSA string

	// RawConfig can be a GCP res.Config
	RawConfig interface{} `json:-`
	// contains filtered or unexported fields
}

K8SCluster represents a single configured K8S cluster. It wraps a rest.Config objects, as well as a Namespace and KSA it will act as.

func NewK8SCluster

func NewK8SCluster(ctx context.Context, ns, name string) *K8SCluster

func (*K8SCluster) Client

func (kr *K8SCluster) Client() *kubernetes.Clientset

Client returns a clientset for accessing the core objects.

func (*K8SCluster) GcpInfo

func (k *K8SCluster) GcpInfo() (string, string, string)

func (*K8SCluster) GetCM

func (kr *K8SCluster) GetCM(ctx context.Context, ns string, name string) (map[string]string, error)

func (*K8SCluster) GetSecret

func (kr *K8SCluster) GetSecret(ctx context.Context, ns string, name string) (map[string][]byte, error)

func (*K8SCluster) GetToken

func (k *K8SCluster) GetToken(ctx context.Context, aud string) (string, error)

func (*K8SCluster) GetTokenRaw

func (k *K8SCluster) GetTokenRaw(ctx context.Context, ns, ksa, aud string) (string, error)

func (*K8SCluster) Label

func (k *K8SCluster) Label(ctx context.Context, name string) string

func (*K8SCluster) Location

func (k *K8SCluster) Location() string

func (*K8SCluster) RestClient

func (kr *K8SCluster) RestClient(group string, version string) (*rest.RESTClient, error)

RestClient returns a K8S RESTClient for a specific resource - without generated interfaces.

group is "" for core resources. version is v1, etc

Serializer defaults to scheme.Codecs.WithoutConversion()

This results in a HttpClient configured with the client certs and server address, and a base URL like /apis/foo.com/v1

func (*K8SCluster) RunAs

func (kr *K8SCluster) RunAs(ns, ksa string) *K8SCluster

Return a new K8S cluster with same config and client, but different default namespace and KSA. The GetToken() requests will use the specified K8S namespace and KSA instead of the default.

func (*K8SCluster) String

func (kr *K8SCluster) String() string

type K8SRest

type K8SRest interface {
	// RestConfig returns the base config, which is used for all other clients.
	RestConfig() *rest.Config
}