internal

package
v1.2.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 14, 2022 License: Apache-2.0 Imports: 51 Imported by: 0

Documentation

Overview

Package internal contains internal implementation details for the Kubeadm Control Plane.

Index

Constants

View Source 
const (
	// NodeBootstrapTokenAuthGroup specifies which group a Node Bootstrap Token should be authenticated in.
	NodeBootstrapTokenAuthGroup = "system:bootstrappers:kubeadm:default-node-token"

	// GetNodesClusterRoleName defines the name of the ClusterRole and ClusterRoleBinding to get nodes.
	GetNodesClusterRoleName = "kubeadm:get-nodes"

	// NodesGroup defines the well-known group for all nodes.
	NodesGroup = "system:nodes"

	// KubeletConfigMapRolePrefix defines base kubelet configuration ConfigMap role prefix.
	KubeletConfigMapRolePrefix = "kubeadm:"

	// KubeletConfigMapName defines base kubelet configuration ConfigMap name for kubeadm < 1.24.
	KubeletConfigMapName = "kubelet-config-%d.%d"

	// UnversionedKubeletConfigMapName defines base kubelet configuration ConfigMap for kubeadm >= 1.24.
	UnversionedKubeletConfigMapName = "kubelet-config"
)
View Source 
const (
	// KubeadmControlPlaneControllerName defines the controller used when creating clients.
	KubeadmControlPlaneControllerName = "kubeadm-controlplane-controller"
)

Variables

View Source 
var (

	// ErrControlPlaneMinNodes signals that a cluster doesn't meet the minimum required nodes
	// to remove an etcd member.
	ErrControlPlaneMinNodes = errors.New("cluster has fewer than 2 control plane nodes; removing an etcd member is not supported")
)
View Source 
var Log = klogr.New()

Log is the global logger for the internal package.

Functions

func ControlPlaneMachineLabelsForCluster added in v0.4.0 

func ControlPlaneMachineLabelsForCluster(kcp *controlplanev1.KubeadmControlPlane, clusterName string) map[string]string

ControlPlaneMachineLabelsForCluster returns a set of labels to add to a control plane machine for this specific cluster.

func ImageRepositoryFromClusterConfig added in v1.2.1 

func ImageRepositoryFromClusterConfig(clusterConfig *bootstrapv1.ClusterConfiguration, kubernetesVersion semver.Version) string

ImageRepositoryFromClusterConfig returns the image repository to use. It returns: * clusterConfig.ImageRepository if set. * "registry.k8s.io" if v1.22 <= version < v1.26 to migrate to the new registry * "" otherwise. Beginning with kubernetes v1.22, the default registry for kubernetes is registry.k8s.io instead of k8s.gcr.io which is why references should get migrated when upgrading to v1.22. The migration follows the behavior of `kubeadm upgrade`.

func MatchesKubeadmBootstrapConfig added in v0.4.0 

func MatchesKubeadmBootstrapConfig(machineConfigs map[string]*bootstrapv1.KubeadmConfig, kcp *controlplanev1.KubeadmControlPlane) collections.Func

MatchesKubeadmBootstrapConfig checks if machine's KubeadmConfigSpec is equivalent with KCP's KubeadmConfigSpec.

func MatchesMachineSpec added in v0.4.0 

func MatchesMachineSpec(infraConfigs map[string]*unstructured.Unstructured, machineConfigs map[string]*bootstrapv1.KubeadmConfig, kcp *controlplanev1.KubeadmControlPlane) func(machine *clusterv1.Machine) bool

MatchesMachineSpec returns a filter to find all machines that matches with KCP config and do not require any rollout. Kubernetes version, infrastructure template, and KubeadmConfig field need to be equivalent.

func MatchesTemplateClonedFrom added in v0.4.0 

func MatchesTemplateClonedFrom(infraConfigs map[string]*unstructured.Unstructured, kcp *controlplanev1.KubeadmControlPlane) collections.Func

MatchesTemplateClonedFrom returns a filter to find all machines that match a given KCP infra template.

Types

type ClusterStatus 

type ClusterStatus struct {
	// Nodes are a total count of nodes
	Nodes int32
	// ReadyNodes are the count of nodes that are reporting ready
	ReadyNodes int32
	// HasKubeadmConfig will be true if the kubeadm config map has been uploaded, false otherwise.
	HasKubeadmConfig bool
}

ClusterStatus holds stats information about the cluster.

type ControlPlane added in v0.3.2 

type ControlPlane struct {
	KCP      *controlplanev1.KubeadmControlPlane
	Cluster  *clusterv1.Cluster
	Machines collections.Machines
	// contains filtered or unexported fields
}

ControlPlane holds business logic around control planes. It should never need to connect to a service, that responsibility lies outside of this struct. Going forward we should be trying to add more logic to here and reduce the amount of logic in the reconciler.

func NewControlPlane added in v0.3.2 

func NewControlPlane(ctx context.Context, client client.Client, cluster *clusterv1.Cluster, kcp *controlplanev1.KubeadmControlPlane, ownedMachines collections.Machines) (*ControlPlane, error)

NewControlPlane returns an instantiated ControlPlane.

func (*ControlPlane) AsOwnerReference added in v0.3.2 

func (c *ControlPlane) AsOwnerReference() *metav1.OwnerReference

AsOwnerReference returns an owner reference to the KubeadmControlPlane.

func (*ControlPlane) EtcdImageData added in v0.3.2 

func (c *ControlPlane) EtcdImageData() (string, string)

EtcdImageData returns the etcd image data embedded in the ClusterConfiguration or empty strings if none are defined.

func (*ControlPlane) FailureDomainWithMostMachines added in v0.3.3 

func (c *ControlPlane) FailureDomainWithMostMachines(machines collections.Machines) *string

FailureDomainWithMostMachines returns a fd which exists both in machines and control-plane machines and has the most control-plane machines on it.

func (*ControlPlane) FailureDomains added in v0.3.3 

func (c *ControlPlane) FailureDomains() clusterv1.FailureDomains

FailureDomains returns a slice of failure domain objects synced from the infrastructure provider into Cluster.Status.

func (*ControlPlane) GenerateKubeadmConfig added in v0.3.2 

func (c *ControlPlane) GenerateKubeadmConfig(spec *bootstrapv1.KubeadmConfigSpec) *bootstrapv1.KubeadmConfig

GenerateKubeadmConfig generates a new kubeadm config for creating new control plane nodes.

func (*ControlPlane) HasDeletingMachine added in v0.3.2 

func (c *ControlPlane) HasDeletingMachine() bool

HasDeletingMachine returns true if any machine in the control plane is in the process of being deleted.

func (*ControlPlane) HasUnhealthyMachine added in v0.3.11 

func (c *ControlPlane) HasUnhealthyMachine() bool

HasUnhealthyMachine returns true if any machine in the control plane is marked as unhealthy by MHC.

func (*ControlPlane) HealthyMachines added in v0.3.11 

func (c *ControlPlane) HealthyMachines() collections.Machines

HealthyMachines returns the list of control plane machines not marked as unhealthy by MHC.

func (*ControlPlane) InitialControlPlaneConfig added in v0.3.2 

func (c *ControlPlane) InitialControlPlaneConfig() *bootstrapv1.KubeadmConfigSpec

InitialControlPlaneConfig returns a new KubeadmConfigSpec that is to be used for an initializing control plane.

func (*ControlPlane) IsEtcdManaged added in v0.3.9 

func (c *ControlPlane) IsEtcdManaged() bool

IsEtcdManaged returns true if the control plane relies on a managed etcd.

func (*ControlPlane) JoinControlPlaneConfig added in v0.3.2 

func (c *ControlPlane) JoinControlPlaneConfig() *bootstrapv1.KubeadmConfigSpec

JoinControlPlaneConfig returns a new KubeadmConfigSpec that is to be used for joining control planes.

func (*ControlPlane) Logger added in v0.3.2 

func (c *ControlPlane) Logger() logr.Logger

Logger returns a logger with useful context.

func (*ControlPlane) MachineInFailureDomainWithMostMachines added in v0.3.4 

func (c *ControlPlane) MachineInFailureDomainWithMostMachines(machines collections.Machines) (*clusterv1.Machine, error)

MachineInFailureDomainWithMostMachines returns the first matching failure domain with machines that has the most control-plane machines on it.

func (*ControlPlane) MachineInfrastructureTemplateRef added in v0.4.0 

func (c *ControlPlane) MachineInfrastructureTemplateRef() *corev1.ObjectReference

MachineInfrastructureTemplateRef returns the KubeadmControlPlane's infrastructure template for Machines.

func (*ControlPlane) MachineWithDeleteAnnotation added in v0.3.13 

func (c *ControlPlane) MachineWithDeleteAnnotation(machines collections.Machines) collections.Machines

MachineWithDeleteAnnotation returns a machine that has been annotated with DeleteMachineAnnotation key.

func (*ControlPlane) MachinesNeedingRollout added in v0.3.8 

func (c *ControlPlane) MachinesNeedingRollout() collections.Machines

MachinesNeedingRollout return a list of machines that need to be rolled out.

func (*ControlPlane) NeedsReplacementNode added in v0.3.2 

func (c *ControlPlane) NeedsReplacementNode() bool

NeedsReplacementNode determines if the control plane needs to create a replacement node during upgrade.

func (*ControlPlane) NewMachine added in v0.3.2 

func (c *ControlPlane) NewMachine(infraRef, bootstrapRef *corev1.ObjectReference, failureDomain *string) *clusterv1.Machine

NewMachine returns a machine configured to be a part of the control plane.

func (*ControlPlane) NextFailureDomainForScaleUp added in v0.3.8 

func (c *ControlPlane) NextFailureDomainForScaleUp() *string

NextFailureDomainForScaleUp returns the failure domain with the fewest number of up-to-date machines.

func (*ControlPlane) PatchMachines added in v0.3.11 

func (c *ControlPlane) PatchMachines(ctx context.Context) error

PatchMachines patches all the machines conditions.

func (*ControlPlane) UnhealthyMachines added in v0.3.11 

func (c *ControlPlane) UnhealthyMachines() collections.Machines

UnhealthyMachines returns the list of control plane machines marked as unhealthy by MHC.

func (*ControlPlane) UpToDateMachines added in v0.3.8 

func (c *ControlPlane) UpToDateMachines() collections.Machines

UpToDateMachines returns the machines that are up to date with the control plane's configuration and therefore do not require rollout.

func (*ControlPlane) Version added in v0.3.2 

func (c *ControlPlane) Version() *string

Version returns the KubeadmControlPlane's version.

type CoreDNSMigrator 

type CoreDNSMigrator struct{}

CoreDNSMigrator is a shim that can be used to migrate CoreDNS files from one version to another.

func (*CoreDNSMigrator) Migrate 

func (c *CoreDNSMigrator) Migrate(fromCoreDNSVersion, toCoreDNSVersion, corefile string, deprecations bool) (string, error)

Migrate calls the CoreDNS migration library to migrate a corefile.

type EtcdClientGenerator added in v0.4.0 

type EtcdClientGenerator struct {
	// contains filtered or unexported fields
}

EtcdClientGenerator generates etcd clients that connect to specific etcd members on particular control plane nodes.

func NewEtcdClientGenerator added in v0.4.0 

func NewEtcdClientGenerator(restConfig *rest.Config, tlsConfig *tls.Config, etcdDialTimeout time.Duration) *EtcdClientGenerator

NewEtcdClientGenerator returns a new etcdClientGenerator instance.

type EtcdMemberStatus added in v0.3.11 

type EtcdMemberStatus struct {
	Name       string
	Responsive bool
}

EtcdMemberStatus contains status information for a single etcd member.

type Management 

type Management struct {
	Client          client.Reader
	Tracker         *remote.ClusterCacheTracker
	EtcdDialTimeout time.Duration
}

Management holds operations on the management cluster.

func (*Management) Get added in v0.3.7 

func (m *Management) Get(ctx context.Context, key client.ObjectKey, obj client.Object) error

Get implements client.Reader.

func (*Management) GetMachinePoolsForCluster added in v0.4.0 

func (m *Management) GetMachinePoolsForCluster(ctx context.Context, cluster *clusterv1.Cluster) (*expv1.MachinePoolList, error)

GetMachinePoolsForCluster returns a list of machine pools owned by the cluster.

func (*Management) GetMachinesForCluster 

func (m *Management) GetMachinesForCluster(ctx context.Context, cluster *clusterv1.Cluster, filters ...collections.Func) (collections.Machines, error)

GetMachinesForCluster returns a list of machines that can be filtered or not. If no filter is supplied then all machines associated with the target cluster are returned.

func (*Management) GetWorkloadCluster 

func (m *Management) GetWorkloadCluster(ctx context.Context, clusterKey client.ObjectKey) (WorkloadCluster, error)

GetWorkloadCluster builds a cluster object. The cluster comes with an etcd client generator to connect to any etcd pod living on a managed machine.

func (*Management) List added in v0.3.7 

func (m *Management) List(ctx context.Context, list client.ObjectList, opts ...client.ListOption) error

List implements client.Reader.

type ManagementCluster 

type ManagementCluster interface {
	client.Reader

	GetMachinesForCluster(ctx context.Context, cluster *clusterv1.Cluster, filters ...collections.Func) (collections.Machines, error)
	GetMachinePoolsForCluster(ctx context.Context, cluster *clusterv1.Cluster) (*expv1.MachinePoolList, error)
	GetWorkloadCluster(ctx context.Context, clusterKey client.ObjectKey) (WorkloadCluster, error)
}

ManagementCluster defines all behaviors necessary for something to function as a management cluster.

type RemoteClusterConnectionError added in v0.3.7 

type RemoteClusterConnectionError struct {
	Name string
	Err  error
}

RemoteClusterConnectionError represents a failure to connect to a remote cluster.

func (*RemoteClusterConnectionError) Error added in v0.3.7 

func (e *RemoteClusterConnectionError) Error() string

Error satisfies the error interface.

func (*RemoteClusterConnectionError) Unwrap added in v0.3.7 

func (e *RemoteClusterConnectionError) Unwrap() error

Unwrap satisfies the unwrap error inteface.

type Workload 

type Workload struct {
	Client          ctrlclient.Client
	CoreDNSMigrator coreDNSMigrator
	// contains filtered or unexported fields
}

Workload defines operations on workload clusters.

func (*Workload) AllowBootstrapTokensToGetNodes added in v0.3.5 

func (w *Workload) AllowBootstrapTokensToGetNodes(ctx context.Context) error

AllowBootstrapTokensToGetNodes creates RBAC rules to allow Node Bootstrap Tokens to list nodes.

func (*Workload) ClusterStatus 

func (w *Workload) ClusterStatus(ctx context.Context) (ClusterStatus, error)

ClusterStatus returns the status of the cluster.

func (*Workload) EnsureResource added in v0.3.5 

func (w *Workload) EnsureResource(ctx context.Context, obj client.Object) error

EnsureResource creates a resoutce if the target resource doesn't exist. If the resource exists already, this function will ignore the resource instead.

func (*Workload) EtcdMembers added in v0.3.11 

func (w *Workload) EtcdMembers(ctx context.Context) ([]string, error)

EtcdMembers returns the current set of members in an etcd cluster.

NOTE: This methods uses control plane machines/nodes only to get in contact with etcd, but then it relies on etcd as ultimate source of truth for the list of members. This is intended to allow informed decisions on actions impacting etcd quorum.

func (*Workload) ForwardEtcdLeadership added in v0.3.2 

func (w *Workload) ForwardEtcdLeadership(ctx context.Context, machine *clusterv1.Machine, leaderCandidate *clusterv1.Machine) error

ForwardEtcdLeadership forwards etcd leadership to the first follower.

func (*Workload) ReconcileEtcdMembers added in v0.3.4 

func (w *Workload) ReconcileEtcdMembers(ctx context.Context, nodeNames []string, version semver.Version) ([]string, error)

ReconcileEtcdMembers iterates over all etcd members and finds members that do not have corresponding nodes. If there are any such members, it deletes them from etcd and removes their nodes from the kubeadm configmap so that kubeadm does not run etcd health checks on them.

func (*Workload) ReconcileKubeletRBACBinding 

func (w *Workload) ReconcileKubeletRBACBinding(ctx context.Context, version semver.Version) error

ReconcileKubeletRBACBinding will create a RoleBinding for the new kubelet version during upgrades. If the role binding already exists this function is a no-op.

func (*Workload) ReconcileKubeletRBACRole 

func (w *Workload) ReconcileKubeletRBACRole(ctx context.Context, version semver.Version) error

ReconcileKubeletRBACRole will create a Role for the new kubelet version during upgrades. If the role already exists this function is a no-op.

func (*Workload) RemoveEtcdMemberForMachine 

func (w *Workload) RemoveEtcdMemberForMachine(ctx context.Context, machine *clusterv1.Machine) error

RemoveEtcdMemberForMachine removes the etcd member from the target cluster's etcd cluster. Removing the last remaining member of the cluster is not supported.

func (*Workload) RemoveMachineFromKubeadmConfigMap 

func (w *Workload) RemoveMachineFromKubeadmConfigMap(ctx context.Context, machine *clusterv1.Machine, version semver.Version) error

RemoveMachineFromKubeadmConfigMap removes the entry for the machine from the kubeadm configmap.

func (*Workload) RemoveNodeFromKubeadmConfigMap added in v0.3.4 

func (w *Workload) RemoveNodeFromKubeadmConfigMap(ctx context.Context, name string, v semver.Version) error

RemoveNodeFromKubeadmConfigMap removes the entry for the node from the kubeadm configmap.

func (*Workload) UpdateAPIServerInKubeadmConfigMap added in v0.3.13 

func (w *Workload) UpdateAPIServerInKubeadmConfigMap(ctx context.Context, apiServer bootstrapv1.APIServer, version semver.Version) error

UpdateAPIServerInKubeadmConfigMap updates api server configuration in kubeadm config map.

func (*Workload) UpdateControllerManagerInKubeadmConfigMap added in v0.3.13 

func (w *Workload) UpdateControllerManagerInKubeadmConfigMap(ctx context.Context, controllerManager bootstrapv1.ControlPlaneComponent, version semver.Version) error

UpdateControllerManagerInKubeadmConfigMap updates controller manager configuration in kubeadm config map.

func (*Workload) UpdateCoreDNS 

func (w *Workload) UpdateCoreDNS(ctx context.Context, kcp *controlplanev1.KubeadmControlPlane, version semver.Version) error

UpdateCoreDNS updates the kubeadm configmap, coredns corefile and coredns deployment.

func (*Workload) UpdateEtcdConditions added in v0.3.11 

func (w *Workload) UpdateEtcdConditions(ctx context.Context, controlPlane *ControlPlane)

UpdateEtcdConditions is responsible for updating machine conditions reflecting the status of all the etcd members. This operation is best effort, in the sense that in case of problems in retrieving member status, it sets the condition to Unknown state without returning any error.

func (*Workload) UpdateEtcdExtraArgsInKubeadmConfigMap added in v0.4.1 

func (w *Workload) UpdateEtcdExtraArgsInKubeadmConfigMap(ctx context.Context, extraArgs map[string]string, version semver.Version) error

UpdateEtcdExtraArgsInKubeadmConfigMap sets extraArgs in the kubeadm config map.

func (*Workload) UpdateEtcdVersionInKubeadmConfigMap 

func (w *Workload) UpdateEtcdVersionInKubeadmConfigMap(ctx context.Context, imageRepository, imageTag string, version semver.Version) error

UpdateEtcdVersionInKubeadmConfigMap sets the imageRepository or the imageTag or both in the kubeadm config map.

func (*Workload) UpdateImageRepositoryInKubeadmConfigMap added in v0.3.3 

func (w *Workload) UpdateImageRepositoryInKubeadmConfigMap(ctx context.Context, imageRepository string, version semver.Version) error

UpdateImageRepositoryInKubeadmConfigMap updates the image repository in the kubeadm config map.

func (*Workload) UpdateKubeProxyImageInfo 

func (w *Workload) UpdateKubeProxyImageInfo(ctx context.Context, kcp *controlplanev1.KubeadmControlPlane, version semver.Version) error

UpdateKubeProxyImageInfo updates kube-proxy image in the kube-proxy DaemonSet.

func (*Workload) UpdateKubeletConfigMap 

func (w *Workload) UpdateKubeletConfigMap(ctx context.Context, version semver.Version) error

UpdateKubeletConfigMap will create a new kubelet-config-1.x config map for a new version of the kubelet. This is a necessary process for upgrades.

func (*Workload) UpdateKubernetesVersionInKubeadmConfigMap 

func (w *Workload) UpdateKubernetesVersionInKubeadmConfigMap(ctx context.Context, version semver.Version) error

UpdateKubernetesVersionInKubeadmConfigMap updates the kubernetes version in the kubeadm config map.

func (*Workload) UpdateSchedulerInKubeadmConfigMap added in v0.3.13 

func (w *Workload) UpdateSchedulerInKubeadmConfigMap(ctx context.Context, scheduler bootstrapv1.ControlPlaneComponent, version semver.Version) error

UpdateSchedulerInKubeadmConfigMap updates scheduler configuration in kubeadm config map.

func (*Workload) UpdateStaticPodConditions added in v0.3.11 

func (w *Workload) UpdateStaticPodConditions(ctx context.Context, controlPlane *ControlPlane)

UpdateStaticPodConditions is responsible for updating machine conditions reflecting the status of all the control plane components running in a static pod generated by kubeadm. This operation is best effort, in the sense that in case of problems in retrieving the pod status, it sets the condition to Unknown state without returning any error.

type WorkloadCluster 

type WorkloadCluster interface {
	// Basic health and status checks.
	ClusterStatus(ctx context.Context) (ClusterStatus, error)
	UpdateStaticPodConditions(ctx context.Context, controlPlane *ControlPlane)
	UpdateEtcdConditions(ctx context.Context, controlPlane *ControlPlane)
	EtcdMembers(ctx context.Context) ([]string, error)

	// Upgrade related tasks.
	ReconcileKubeletRBACBinding(ctx context.Context, version semver.Version) error
	ReconcileKubeletRBACRole(ctx context.Context, version semver.Version) error
	UpdateKubernetesVersionInKubeadmConfigMap(ctx context.Context, version semver.Version) error
	UpdateImageRepositoryInKubeadmConfigMap(ctx context.Context, imageRepository string, version semver.Version) error
	UpdateEtcdVersionInKubeadmConfigMap(ctx context.Context, imageRepository, imageTag string, version semver.Version) error
	UpdateEtcdExtraArgsInKubeadmConfigMap(ctx context.Context, extraArgs map[string]string, version semver.Version) error
	UpdateAPIServerInKubeadmConfigMap(ctx context.Context, apiServer bootstrapv1.APIServer, version semver.Version) error
	UpdateControllerManagerInKubeadmConfigMap(ctx context.Context, controllerManager bootstrapv1.ControlPlaneComponent, version semver.Version) error
	UpdateSchedulerInKubeadmConfigMap(ctx context.Context, scheduler bootstrapv1.ControlPlaneComponent, version semver.Version) error
	UpdateKubeletConfigMap(ctx context.Context, version semver.Version) error
	UpdateKubeProxyImageInfo(ctx context.Context, kcp *controlplanev1.KubeadmControlPlane, version semver.Version) error
	UpdateCoreDNS(ctx context.Context, kcp *controlplanev1.KubeadmControlPlane, version semver.Version) error
	RemoveEtcdMemberForMachine(ctx context.Context, machine *clusterv1.Machine) error
	RemoveMachineFromKubeadmConfigMap(ctx context.Context, machine *clusterv1.Machine, version semver.Version) error
	RemoveNodeFromKubeadmConfigMap(ctx context.Context, nodeName string, version semver.Version) error
	ForwardEtcdLeadership(ctx context.Context, machine *clusterv1.Machine, leaderCandidate *clusterv1.Machine) error
	AllowBootstrapTokensToGetNodes(ctx context.Context) error

	// State recovery tasks.
	ReconcileEtcdMembers(ctx context.Context, nodeNames []string, version semver.Version) ([]string, error)
}

WorkloadCluster defines all behaviors necessary to upgrade kubernetes on a workload cluster

TODO: Add a detailed description to each of these method definitions.

Source Files

View all Source files

Directories

Path Synopsis
Package controllers implements the Kubeadm controllers.
 Package controllers implements the Kubeadm controllers.
Package etcd provides a connection to an etcd member.
 Package etcd provides a connection to an etcd member.
fake
Package fake implements testing fakes.
 Package fake implements testing fakes.
util
Package util implements etcd utility functions.
 Package util implements etcd utility functions.
Package proxy implements kubeadm proxy functionality.
 Package proxy implements kubeadm proxy functionality.
Package webhooks provides the validating webhook for KubeadmControlPlane scale subresource.
 Package webhooks provides the validating webhook for KubeadmControlPlane scale subresource.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.