vic

README

vSphere Integrated Containers virtual kubelet provider

This is a very early preview of the VMware vSphere Integrated Containers virtual kubelet provider.

Introduction to vSphere Integrated Containers

For those who are unfamiliar with vSphere Integrated Containers, it is a open source project that enables users to create a CaaS (container as a service) endpoint on a vSphere cluster. There are a few advantages of using vSphere Integrated Containers over using standard Docker in a VM:

1. Containers are wrapped in VMs and provides stronger isolation than a standard OS containers.
2. Use of vSphere Integrated Containers removes the need to deploy a host VM, OS, and docker software. It also removes the requirement to maintain that stack. An administrator can simply deploy a VCH (virtual container host) and get back a docker endpoint. With this endpoint, users can deploy containers. vSphere manages the endpoint.
3. The VCH can span a vCenter cluster, providing a much larger virtual host than any VM or physical machine on which users can deploy their containers. Users can deploy containers across the entire vCenter cluster using this single virtual host.

Virtual Kubelet in vSphere Integrated Containers

Within a deployed VCH, there is a daemon that provides a Docker compatible endpoint. We refer to this daemon as a Docker personality server. With the vSphere integrated Container virtual kubelet provider, we are introducing a second personality server that provides a Kubelet compatible endpoint for the VCH. With this endpoint, the VCH can join a kubernetes cluster, providing a robust virtual node for pods. Just as the VCH provides a CaaS endpoint for Docker users, it will now provide a Pod-as-a-Service endpoint for kubernetes users. vSphere manages the availability of the pods and remove the need for administrators to maintain the uptime of a normal node, it's software, and the pods running on it. For the user, it provides a much larger virtual node than any single VM or physical machine.

Pod VM in vSphere Integrated Containers

With the vSphere integrated Container virtual kubelet provider, we are also introducing a new isolation concept called the pod vm. With the original vSphere Integrated Containers, we introduced stronger container isolation through our container vm concept. Interested parties may go to the vSphere Integrated Containers github project page to read all about it. A standard pod in a standard kubernetes node is just a collection of containers with a parent container. The containers are isolated by the OS, but the pod is not. With vSphere Integrated Containers, pods are strongly isolated within a pod vm. Each pod runs within it's own pod vm.

Project Status

This is an early preview. The project relies on a feature branch within the vSphere Integrated Containers project. Currently, pods can be created and removed. Some node statuses can be queried. This provider reuses vSphere Integrated Containers to provision pods. As such, there is a dependency between this provider and the vSphere Integrated Containers project. All changes to that project are currently being done on a feature branch, named wolfpack.

There are a lot of kubelet features still undefined in the virtual-kubelet project, such as networking, volumes, exec, logging, etc. As these are defined, we will add this support into our provider and into vSphere Integrated Containers.

Building vSphere Integrated Containers with virtual kubelet support

This project and the associated vSphere Integrated Containers project are in active development. The simplest way to utilize this virtual kubelet provider, is to build the kubelet project and the wolfpack feature branch of vSphere Integrated Containers. The following is a step by step instructions on building both. There is an asciinema playback of this process below. The current instructions assumes your development environment is Linux.

First, ensure golang 1.8.x and git are installed on your machine. The vSphere Integrated Containers project checks for go 1.8. Next, perform the following steps

1. make sure you have go 1.8.x and prepare your GOPATH
   • $> go version
   • $> mkdir -p go/src/github.com/virtual-kubelet
2. get and build the virtual kubelet
   • $> cd go/src/github.com/virtual-kubelet
   • $> git clone https://github.com/virtual-kubelet/virtual-kubelet.git
   • $> cd virtual-kubelet
   • $> go build .
3. get wolfpack feature branch of vSphere Integrated Containers
   • $> cd
   • $> mkdir -p go/src/github.com/vmware
   • $> cd $GOPATH/src/github.com/vmware
   • $> git clone https://github.com/vmware/vic
   • $> cd vic
   • $> git checkout feature/wolfpack
4. build vSphere Integrated Containers as root
   • $> cd $GOPATH/src/github.com/vmware/vic
   • $> sudo su
   • $> export GOPATH=/home/[user]/go
   • $> export PATH=$PATH:/usr/local/go/bin:/home/[user]/go/bin
   • $> export VIRTUAL_KUBELET_PATH=$GOPATH/src/github.com/virtual-kubelet/virtual-kubelet/virtual-kubelet
   • $> make most-vkubelet
   • $> chown -R user:user *
   • $> exit (exit root)

You should now have a bin folder in the vic folder. This contains all the necessary assets to deploy a VCH using the vic-machine CLI.

Asciinema recording of the build steps

Usage

To get started with the vSphere Integrated Containers' virtual kubelet, users should first familiarize themselves with deploying vSphere Integrated Containers and using the VCH as a docker endpoint. We're leveraging the same deployment model with the virtual kubelet. Administrators deploy a VCH, using the vic-machine CLI or VIC Appliance, specifying the kubernetes cluster that the VCH should join.

Virtual Container Hosts should be deployed with the proper CLI flags to enable virtual kubelet, including both --k8s-server-address and --k8s-config.

Once the VCH is fully started, it automatically joins the cluster. Administrators can then apply taints or labels to the virtual node to create node affinity. Pods created with the right toleration or nodeSelector will then be deployed onto the VCH.

1. Get IP or FQDN of your kubernetes master node
2. Get location of your kube config you use for kubectl (usually in $HOME/.kube/config)
3. Deploy a virtual container host with vic-machine.
4. $> kubectl get nodes
5. Deploy a pod

Asciinema recording of an example usage

Documentation

Index

• Constants
• func KubeResourcesFromVchInfo(op trace.Operation, info *models.VCHInfo) v1.ResourceList
• func LocalInstance() bool
• func NilProxy(caller, proxyName string) error
• type CreateResponse
• type DockerClient
  • func NewVicDockerClient(personaAddr string) DockerClient
• type VicConfig
  • func NewVicConfig(op trace.Operation, configFile string) VicConfig
• type VicDockerClient
  • func (v *VicDockerClient) CreateContainer(op trace.Operation, config string) error
  • func (v *VicDockerClient) Ping(op trace.Operation) error
  • func (v *VicDockerClient) PullImage(op trace.Operation, image string) error
• type VicProvider
  • func NewVicProvider(configFile string, rm *manager.ResourceManager, ...) (*VicProvider, error)
  • func (v *VicProvider) Capacity() v1.ResourceList
  • func (v *VicProvider) CreatePod(pod *v1.Pod) error
  • func (v *VicProvider) DeletePod(pod *v1.Pod) error
  • func (p *VicProvider) ExecInContainer(name string, uid types.UID, container string, cmd []string, in io.Reader, ...) error
  • func (v *VicProvider) GetContainerLogs(namespace, podName, containerName string, tail int) (string, error)
  • func (v *VicProvider) GetPod(namespace, name string) (*v1.Pod, error)
  • func (p *VicProvider) GetPodFullName(namespace string, pod string) string
  • func (v *VicProvider) GetPodStatus(namespace, name string) (*v1.PodStatus, error)
  • func (v *VicProvider) GetPods() ([]*v1.Pod, error)
  • func (v *VicProvider) NodeAddresses() []v1.NodeAddress
  • func (v *VicProvider) NodeConditions() []v1.NodeCondition
  • func (v *VicProvider) NodeDaemonEndpoints() *v1.NodeDaemonEndpoints
  • func (v *VicProvider) OperatingSystem() string
  • func (v *VicProvider) UpdatePod(pod *v1.Pod) error

Constants

const (
	// Name of filename used in the endpoint vm
	LogFilename = "virtual-kubelet"

	// PanicLevel level, highest level of severity. Logs and then calls panic with the
	// message passed to Debug, Info, ...
	PanicLevel uint8 = iota
	// FatalLevel level. Logs and then calls `os.Exit(1)`. It will exit even if the
	// logging level is set to Panic.
	FatalLevel
	// ErrorLevel level. Logs. Used for errors that should definitely be noted.
	// Commonly used for hooks to send errors to an error tracking service.
	ErrorLevel
	// WarnLevel level. Non-critical entries that deserve eyes.
	WarnLevel
	// InfoLevel level. General operational entries about what's going on inside the
	// application.
	InfoLevel
	// DebugLevel level. Usually only enabled when debugging. Very verbose logging.
	DebugLevel
)

Functions

func KubeResourcesFromVchInfo

func KubeResourcesFromVchInfo(op trace.Operation, info *models.VCHInfo) v1.ResourceList

KubeResourcesFromVchInfo returns a K8s node resource list, given the VCHInfo

func LocalInstance

func LocalInstance() bool

func NilProxy

func NilProxy(caller, proxyName string) error

Types

type CreateResponse

type CreateResponse struct {
	Id       string `json:"Id"`
	Warnings string `json:"Warnings"`
}

type DockerClient

type DockerClient interface {
	Ping(op trace.Operation) error
	CreateContainer(op trace.Operation, config string) error
	PullImage(op trace.Operation, image string) error
}

Super simplistic docker client for the virtual kubelet to perform some operations

func NewVicDockerClient

func NewVicDockerClient(personaAddr string) DockerClient

type VicConfig

type VicConfig struct {
	PersonaAddr   string `yaml:"persona-server"`
	PortlayerAddr string `yaml:"portlayer-server"`
	HostUUID      string `yaml:"host-uuid"`
}

func NewVicConfig

func NewVicConfig(op trace.Operation, configFile string) VicConfig

type VicDockerClient

type VicDockerClient struct {
	// contains filtered or unexported fields
}

func (*VicDockerClient) CreateContainer

func (v *VicDockerClient) CreateContainer(op trace.Operation, config string) error

func (*VicDockerClient) Ping

func (v *VicDockerClient) Ping(op trace.Operation) error

func (*VicDockerClient) PullImage

func (v *VicDockerClient) PullImage(op trace.Operation, image string) error

type VicProvider

type VicProvider struct {
	// contains filtered or unexported fields
}

func NewVicProvider

func NewVicProvider(configFile string, rm *manager.ResourceManager, nodeName, operatingSystem string) (*VicProvider, error)

func (*VicProvider) Capacity

func (v *VicProvider) Capacity() v1.ResourceList

Capacity returns a resource list with the capacity constraints of the provider.

func (*VicProvider) CreatePod

func (v *VicProvider) CreatePod(pod *v1.Pod) error

CreatePod takes a Kubernetes Pod and deploys it within the provider.

func (*VicProvider) DeletePod

func (v *VicProvider) DeletePod(pod *v1.Pod) error

DeletePod takes a Kubernetes Pod and deletes it from the provider.

func (*VicProvider) ExecInContainer

func (p *VicProvider) ExecInContainer(name string, uid types.UID, container string, cmd []string, in io.Reader, out, err io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize, timeout time.Duration) error

ExecInContainer executes a command in a container in the pod, copying data between in/out/err and the container's stdin/stdout/stderr.

func (*VicProvider) GetContainerLogs

func (v *VicProvider) GetContainerLogs(namespace, podName, containerName string, tail int) (string, error)

GetContainerLogs retrieves the logs of a container by name from the provider.

func (*VicProvider) GetPod

func (v *VicProvider) GetPod(namespace, name string) (*v1.Pod, error)

GetPod retrieves a pod by name from the provider (can be cached).

func (*VicProvider) GetPodFullName

func (p *VicProvider) GetPodFullName(namespace string, pod string) string

Get full pod name as defined in the provider context TODO: Implementation

func (*VicProvider) GetPodStatus

func (v *VicProvider) GetPodStatus(namespace, name string) (*v1.PodStatus, error)

GetPodStatus retrieves the status of a pod by name from the provider. This function needs to return a status or the reconcile loop will stop running.

func (*VicProvider) GetPods

func (v *VicProvider) GetPods() ([]*v1.Pod, error)

GetPods retrieves a list of all pods running on the provider (can be cached).

func (*VicProvider) NodeAddresses

func (v *VicProvider) NodeAddresses() []v1.NodeAddress

NodeAddresses returns a list of addresses for the node status within Kubernetes.

func (*VicProvider) NodeConditions

func (v *VicProvider) NodeConditions() []v1.NodeCondition

NodeConditions returns a list of conditions (Ready, OutOfDisk, etc), which is polled periodically to update the node status within Kubernetes.

func (*VicProvider) NodeDaemonEndpoints

func (v *VicProvider) NodeDaemonEndpoints() *v1.NodeDaemonEndpoints

NodeDaemonEndpoints returns NodeDaemonEndpoints for the node status within Kubernetes.

func (*VicProvider) OperatingSystem

func (v *VicProvider) OperatingSystem() string

OperatingSystem returns the operating system the provider is for.

func (*VicProvider) UpdatePod

func (v *VicProvider) UpdatePod(pod *v1.Pod) error

UpdatePod takes a Kubernetes Pod and updates it within the provider.