nodebootstrap

README

Nodebootstrap

Until recently, eksctl provided its own bootstrapping/userdata logic on disk, over-writing or ignore those which came with the AMIs. This caused numerous headaches after changes upstream and we got tired of maintaining these extra pieces.

Implementation

For Unmanaged nodes there is an interface:

type Bootstrapper interface {
  UserData() (string, error)
}

AMI families Ubuntu, AmazonLinux2, Bottlerocket and Windows all fulfil this.

As of eksctl version 0.45.0 unmanaged nodes of these families, as well as managed nodes (different interface), will defer to the native bootstrap script which comes built into the image.

This script is found on disk at /etc/eks/bootstrap.sh. UserData will provide a wrapper script which will set custom values and delegate to the official bootstrap script.

Ubuntu & AmazonLinux2

The bootstrapping "prep" for these 2 are fairly similar. Common setup lives in userdata.go. Individual scripts are prepped in ubuntu.go and al2.go.

Non-dynamic assets live in assets/.

Both bootstrappers add assets/bootstrap.helper.sh to the node along with either assets/bootstrap.ubuntu.sh or assets/bootstrap.al2.sh.

The call to UserData will also dynamically add the following:

• kubelet-extra.json - user configuration for kubelet
• docker-extra.json - extra config for docker daemon
• kubelet.env - env vars for kubelet

The bootstrap wrapper scripts will use jq and sed to get user and our config into various files, and then call /etc/eks/bootstrap.sh.

For AL2, enabling either SSM or EFA will add assets/install-ssm.al2.sh or assets/efa.al2.sh.

Troubleshooting

Ubuntu

sudo snap logs kubelet-eks [-n=all/20]
systemctl status docker.service

Files:

/etc/eks/bootstrap.sh
/var/lib/cloud/scripts/eksctl/bootstrap.ubuntu.sh
/etc/kubernetes/kubelet/kubelet-config.json
/etc/docker/daemon.json

AmazonLinux2

Status:

systemctl status kubelet
systemctl status docker

Logs:

journalctl -u kubelet.service

Files:

/etc/eks/bootstrap.sh
/var/lib/cloud/scripts/eksctl/bootstrap.al2.sh
/etc/kubernetes/kubelet/kubelet-config.json
/etc/docker/daemon.json
/var/lib/cloud/scripts/eksctl/efa.al2.sh
/var/lib/cloud/scripts/eksctl/install-ssm.sh

Documentation

Index

• func GetClusterDNS(clusterConfig *api.ClusterConfig) (string, error)
• func ProtectTOMLKeys(path []string, tree *toml.Tree)
• type AmazonLinux2
  • func NewAL2Bootstrapper(clusterConfig *api.ClusterConfig, ng *api.NodeGroup, clusterDNS string) *AmazonLinux2
  • func (b *AmazonLinux2) UserData() (string, error)
• type Bootstrapper
  • func NewBootstrapper(clusterConfig *api.ClusterConfig, ng *api.NodeGroup) (Bootstrapper, error)
  • func NewManagedBootstrapper(clusterConfig *api.ClusterConfig, ng *api.ManagedNodeGroup) (Bootstrapper, error)
• type Bottlerocket
  • func NewBottlerocketBootstrapper(clusterConfig *api.ClusterConfig, np api.NodePool) *Bottlerocket
  • func (b *Bottlerocket) UserData() (string, error)
• type ManagedAL2
  • func NewManagedAL2Bootstrapper(ng *api.ManagedNodeGroup) *ManagedAL2
  • func (m *ManagedAL2) UserData() (string, error)
• type ManagedBottlerocket
  • func NewManagedBottlerocketBootstrapper(clusterConfig *api.ClusterConfig, ng *api.ManagedNodeGroup) *ManagedBottlerocket
  • func (b *ManagedBottlerocket) UserData() (string, error)
• type ManagedWindows
  • func (w *ManagedWindows) UserData() (string, error)
• type Ubuntu
  • func NewUbuntuBootstrapper(clusterConfig *api.ClusterConfig, np api.NodePool, clusterDNS string) *Ubuntu
  • func (b *Ubuntu) UserData() (string, error)
• type Windows
  • func NewWindowsBootstrapper(clusterConfig *api.ClusterConfig, np api.NodePool, clusterDNS string) *Windows
  • func (b *Windows) UserData() (string, error)

Functions

func GetClusterDNS

func GetClusterDNS(clusterConfig *api.ClusterConfig) (string, error)

GetClusterDNS returns the DNS address to use

func ProtectTOMLKeys

func ProtectTOMLKeys(path []string, tree *toml.Tree)

ProtectTOMLKeys processes a tree finding and replacing dotted keys with quoted keys to retain the configured settings. This prevents TOML parsers from deserializing keys into nested key-value pairs at each dot encountered - which is not uncommon in the context of Kubernetes' labels, annotations, and taints.

Types

type AmazonLinux2

type AmazonLinux2 struct {
	// contains filtered or unexported fields
}

func NewAL2Bootstrapper

func NewAL2Bootstrapper(clusterConfig *api.ClusterConfig, ng *api.NodeGroup, clusterDNS string) *AmazonLinux2

func (*AmazonLinux2) UserData

func (b *AmazonLinux2) UserData() (string, error)

type Bootstrapper

type Bootstrapper interface {
	// UserData returns userdata for bootstrapping nodes
	UserData() (string, error)
}

func NewBootstrapper

func NewBootstrapper(clusterConfig *api.ClusterConfig, ng *api.NodeGroup) (Bootstrapper, error)

NewBootstrapper returns the correct bootstrapper for the AMI family

func NewManagedBootstrapper

func NewManagedBootstrapper(clusterConfig *api.ClusterConfig, ng *api.ManagedNodeGroup) (Bootstrapper, error)

NewManagedBootstrapper creates a new bootstrapper for managed nodegroups based on the AMI family

type Bottlerocket

type Bottlerocket struct {
	// contains filtered or unexported fields
}

func NewBottlerocketBootstrapper

func NewBottlerocketBootstrapper(clusterConfig *api.ClusterConfig, np api.NodePool) *Bottlerocket

func (*Bottlerocket) UserData

func (b *Bottlerocket) UserData() (string, error)

NewUserDataForBottlerocket generates TOML userdata for bootstrapping a Bottlerocket node.

type ManagedAL2

type ManagedAL2 struct {

	// UserDataMimeBoundary sets the MIME boundary for user data
	UserDataMimeBoundary string
	// contains filtered or unexported fields
}

ManagedAL2 is a bootstrapper for managed Amazon Linux 2 nodegroups

func NewManagedAL2Bootstrapper

func NewManagedAL2Bootstrapper(ng *api.ManagedNodeGroup) *ManagedAL2

NewManagedAL2Bootstrapper creates a new ManagedAL2 bootstrapper

func (*ManagedAL2) UserData

func (m *ManagedAL2) UserData() (string, error)

UserData returns user data for AL2 managed nodegroups

type ManagedBottlerocket

type ManagedBottlerocket struct {
	// contains filtered or unexported fields
}

func NewManagedBottlerocketBootstrapper

func NewManagedBottlerocketBootstrapper(clusterConfig *api.ClusterConfig, ng *api.ManagedNodeGroup) *ManagedBottlerocket

NewManagedBottlerocketBootstrapper returns a new bootstrapper for managed Bottlerocket.

func (*ManagedBottlerocket) UserData

func (b *ManagedBottlerocket) UserData() (string, error)

UserData generates TOML userdata for bootstrapping a Bottlerocket node.

type ManagedWindows

type ManagedWindows struct {
	NodeGroup *api.ManagedNodeGroup
}

ManagedWindows implements a bootstrapper for managed Windows nodegroups.

func (*ManagedWindows) UserData

func (w *ManagedWindows) UserData() (string, error)

UserData returns the userdata.

type Ubuntu

type Ubuntu struct {
	// contains filtered or unexported fields
}

func NewUbuntuBootstrapper

func NewUbuntuBootstrapper(clusterConfig *api.ClusterConfig, np api.NodePool, clusterDNS string) *Ubuntu

func (*Ubuntu) UserData

func (b *Ubuntu) UserData() (string, error)

type Windows

type Windows struct {
	// contains filtered or unexported fields
}

func NewWindowsBootstrapper

func NewWindowsBootstrapper(clusterConfig *api.ClusterConfig, np api.NodePool, clusterDNS string) *Windows

func (*Windows) UserData

func (b *Windows) UserData() (string, error)