aggregator

README

aggregator

This is a package that is intended to power the aggregation of microservices within Grafana. The concept as well as implementation is largely borrowed from kube-aggregator.

Why aggregate services?

Grafana's future architecture will entail the same API Server design as that of Kubernetes API Servers. API Servers provide a standard way of stitching together API Groups through discovery and shared routing patterns that allows them to aggregate to a parent API Server in a seamless manner. Since we desire to break Grafana monolith up into more functionally divided microservices, aggregation does the job of still being able to provide these services under a single address. Other benefits of aggregation include free health checks and being able to independently roll out features for each service without downtime.

To read more about the concept, see here.

Note that this aggregation will be a totally internal detail to Grafana. External fully functional API Servers that may themselves act as parent API Servers to Grafana will never be made aware of internal Grafana API Servers. Thus, any APIService objects corresponding to Grafana's API groups will take the address of Grafana's main API Server (the one that bundles grafana-aggregator).

Also, note that the single binary OSS offering of Grafana doesn't make use of the aggregator component at all, instead opting for local installation of all the Grafana API groups.

kube-aggregator versus grafana-aggregator

The grafana-aggregator component will work similarly to how kube-aggregator works for kube-apiserver, the major difference being that it doesn't require core V1 APIs such as Service. Early on, we decided to not have core V1 APIs in the root Grafana API Server. In order to still be able to implement aggregation, we do the following in this Go package:

1. We do not start the core shared informer factories as well as any default controllers that utilize them. This is achieved using DisabledPostStartHooks facility under the GenericAPIServer's RecommendedConfig.
2. We provide an externalname Kind API implementation under service.grafana.app group which works functionally equivalent to the idea with the same name under core/v1/Service.
3. Lastly, we swap the default available condition controller with the custom one written by us. This one is based on our externalname (service.grafana.app) implementation. We register separate PostStartHooks using AddPostStartHookOrDie on the GenericAPIServer to start the corresponding custom controller as well as requisite informer factories for our own externalname Kind.
4. For now, we bundle apiextensions-apiserver under our aggregator component. This is slightly different from K8s where kube-apiserver is called the top-level component and controlplane, aggregator and apiextensions-apiserver live under that instead.

Gotchas (Pay Attention)

1. grafana-aggregator uses file storage under data/grafana-apiserver (apiregistration.k8s.io, service.grafana.app). Thus, any restarts will still have any prior configured aggregation in effect.
2. During local development, ensure you start the aggregated service after launching the aggregator. This is so you have TLS and kubeconfig available for use with example aggregated api servers.
3. Ensure you have grafanaAPIServerWithExperimentalAPIs = false in your custom.ini. Otherwise, the example service the following guide uses for the aggregation test is bundled as a Local APIService and will cause configuration overwrites on startup.

Testing aggregation locally

1. Generate the PKI using openssl (for development purposes, we will use the CN of system:masters):

./hack/make-aggregator-pki.sh

1. Configure the aggregator:

[feature_toggles]
grafanaAPIServerEnsureKubectlAccess = true
; disable the experimental APIs flag to disable bundling of the example service locally
grafanaAPIServerWithExperimentalAPIs = false
kubernetesAggregator = true

[grafana-apiserver]
proxy_client_cert_file = ./data/grafana-aggregator/client.crt
proxy_client_key_file = ./data/grafana-aggregator/client.key

1. Start the server

make run

1. In another tab, apply the manifests:

export KUBECONFIG=$PWD/data/grafana-apiserver/grafana.kubeconfig
kubectl apply -f ./pkg/services/apiserver/aggregator/examples/manual-test/
# SAMPLE OUTPUT
# apiservice.apiregistration.k8s.io/v0alpha1.example.grafana.app created
# externalname.service.grafana.app/example-apiserver created

kubectl get apiservice
# SAMPLE OUTPUT
# NAME                           SERVICE                     AVAILABLE                      AGE
# v0alpha1.example.grafana.app   grafana/example-apiserver   False (FailedDiscoveryCheck)   29m

1. In another tab, start the example microservice that will be aggregated by the parent apiserver:

go run ./pkg/cmd/grafana apiserver \
  --runtime-config=example.grafana.app/v0alpha1=true \
  --secure-port 7443 \
  --tls-cert-file $PWD/data/grafana-aggregator/server.crt \
  --tls-private-key-file $PWD/data/grafana-aggregator/server.key \ 
  --requestheader-client-ca-file=$PWD/data/grafana-aggregator/ca.crt \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  -v 10

1. After 10 seconds, check APIService again. It should report as available.

export KUBECONFIG=$PWD/data/grafana-apiserver/grafana.kubeconfig
kubectl get apiservice
# SAMPLE OUTPUT
# NAME                           SERVICE                     AVAILABLE      AGE
# v0alpha1.example.grafana.app   grafana/example-apiserver   True           30m

1. For tear down of the above test:

kubectl delete -f ./pkg/services/apiserver/aggregator/examples/

Testing auto-registration of remote services locally

A sample aggregation config for remote services is provided under conf. Provided, you have the following setup in your custom.ini, the apiserver will register your remotely running services on startup.

; in custom.ini
; the bundle is only used when not in dev mode
apiservice_ca_bundle_file = ./data/grafana-aggregator/ca.crt

remote_services_file = ./pkg/services/apiserver/aggregator/examples/autoregister/apiservices.yaml

Documentation

Index

• Variables
• func CreateAggregatorServer(config *Config, delegateAPIServer genericapiserver.DelegationTarget) (*aggregatorapiserver.APIAggregator, error)
• func NewExternalNameResolver(externalNames servicelistersv0alpha1.ExternalNameLister) apiserver.ServiceResolver
• type AvailableConditionController
  • func NewAvailableConditionController(apiServiceInformer informers.APIServiceInformer, ...) (*AvailableConditionController, error)
  • func (c *AvailableConditionController) Run(workers int, stopCh <-chan struct{})
• type Config
  • func CreateAggregatorConfig(commandOptions *options.Options, ...) (*Config, error)
  • func NewConfig(aggregator *aggregatorapiserver.Config, ...) *Config
• type Priority
• type RemoteService
• type RemoteServicesConfig
• type ServiceResolver

Variables

var APIVersionPriorities = map[schema.GroupVersion]Priority{
	{Group: "", Version: "v1"}: {Group: 18000, Version: 1},

	{Group: "admissionregistration.k8s.io", Version: "v1"}:       {Group: 16700, Version: 15},
	{Group: "admissionregistration.k8s.io", Version: "v1beta1"}:  {Group: 16700, Version: 12},
	{Group: "admissionregistration.k8s.io", Version: "v1alpha1"}: {Group: 16700, Version: 9},
}

APIVersionPriorities are the proper way to resolve this letting the aggregator know the desired group and version-within-group order of the underlying servers is to refactor the genericapiserver.DelegationTarget to include a list of priorities based on which APIs were installed. This requires the APIGroupInfo struct to evolve and include the concept of priorities and to avoid mistakes, the core storage map there needs to be updated. That ripples out every bit as far as you'd expect, so for 1.7 we'll include the list here instead of being built up during storage.

Functions

func CreateAggregatorServer

func CreateAggregatorServer(config *Config, delegateAPIServer genericapiserver.DelegationTarget) (*aggregatorapiserver.APIAggregator, error)

func NewExternalNameResolver

func NewExternalNameResolver(externalNames servicelistersv0alpha1.ExternalNameLister) apiserver.ServiceResolver

Types

type AvailableConditionController

type AvailableConditionController struct {
	// contains filtered or unexported fields
}

AvailableConditionController handles checking the availability of registered API services.

func NewAvailableConditionController

func NewAvailableConditionController(
	apiServiceInformer informers.APIServiceInformer,
	externalNameInformer informersservicev0alpha1.ExternalNameInformer,
	apiServiceClient apiregistrationclient.APIServicesGetter,
	proxyTransportDial *transport.DialHolder,
	proxyCurrentCertKeyContent certKeyFunc,
	serviceResolver ServiceResolver,
) (*AvailableConditionController, error)

NewAvailableConditionController returns a new AvailableConditionController.

func (*AvailableConditionController) Run

func (c *AvailableConditionController) Run(workers int, stopCh <-chan struct{})

Run starts the AvailableConditionController loop which manages the availability condition of API services.

type Config

type Config struct {
	KubeAggregatorConfig *aggregatorapiserver.Config
	Informers            informersv0alpha1.SharedInformerFactory
	RemoteServicesConfig *RemoteServicesConfig
	// Builders contain prerequisite api groups for aggregator to function correctly e.g. ExternalName
	// Since the main APIServer delegate supports storage implementations that intend to be multi-tenant
	// Aggregator builders that we don't intend to use multi-tenant storage are kept in aggregator's
	// Delegate, one which is configured explicitly to use file storage only
	Builders []builder.APIGroupBuilder
}

func CreateAggregatorConfig

func CreateAggregatorConfig(commandOptions *options.Options, sharedConfig genericapiserver.RecommendedConfig, externalNamesNamespace string) (*Config, error)

func NewConfig

func NewConfig(aggregator *aggregatorapiserver.Config, informers informersv0alpha1.SharedInformerFactory, builders []builder.APIGroupBuilder, remoteServices *RemoteServicesConfig) *Config

remoteServices may be nil when not using aggregation

type Priority

type Priority struct {
	// Group indicates the order of the Group relative to other groups.
	Group int32
	// Version indicates the relative order of the Version inside of its group.
	Version int32
}

Priority defines group Priority that is used in discovery. This controls group position in the kubectl output.

type RemoteService

type RemoteService struct {
	Group   string `yaml:"group"`
	Version string `yaml:"version"`
	Host    string `yaml:"host"`
	Port    int32  `yaml:"port"`
}

type RemoteServicesConfig

type RemoteServicesConfig struct {
	ExternalNamesNamespace string
	InsecureSkipTLSVerify  bool
	CABundle               []byte
	Services               []RemoteService
	// contains filtered or unexported fields
}

type ServiceResolver

type ServiceResolver interface {
	ResolveEndpoint(namespace, name string, port int32) (*url.URL, error)
}

ServiceResolver knows how to convert a service reference into an actual location.