libsnnet

package
v0.0.0-...-53c686c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 16, 2016 License: Apache-2.0 Imports: 29 Imported by: 0

README

Simple Node Network Library

Overview

The Simple Node Network Library (libsnnet) implements a simple SDN controller. The library implements all the networking setup primitives required in Ciao.

libsnnet currently provides the following capabilities

  • Creation of multiple isolated tenant overlay L2 networks
  • Auto assignment of IP Addresses for VM's and containers
  • Support for Virtual Machine (QEMU)and Container workloads
    • Containers and VMs that belong to the same tenant are placed in the same L2 overlay network and connect to each other seamlessly
  • Ability to perform inbound and outbound NAT to/from the workloads
  • A built in docker Network and IPAM plugin to allow interworking between Docker logical networks and the ciao L2 overlay Network

It tries to rely on interfaces directly exposed by the kernel vs using user space tools to ensure maximum portability. The implementation maintains local state on leaf nodes vs relying on centralized state. It also uses local state to perform any network re-configuration in the event of a launcher crash or restart

Currently the library supports creation of bridges, GRE tunnels, VM and Container compatible interfaces (VNICs) on nodes. It also provides and the ability to attach tunnels and VNICs to bridges.

The implementation also provides the ability to interconnect these bridges across nodes creating L2 Overlay networks.

Roles

The library supports node specific networking initialization capabilities. It currently supports setup of Compute Nodes (CN), Network Nodes (NN) and Compute Node Concentrator Instances (CNCI)

Compute Node

A compute node typically runs VM and Container workloads. The library provides API's to perform network initialization as well as network interface creation and overlay network linking.

Network Node

The tenant overlay networks are linked together to Network Nodes. The Network Node switch and route traffic between the tenant bridges and subnets distributed across multiple Compute Nodes.

CNCI

Compute Node Concentrators or CNCIs are Virtual Machines running on Network Nodes which handle subsets of traffic belonging to a single tenant. A single network node can run multiple CNCI's limited by the Compute and Network needs of the CNCIs. All tenant level switching and routing for a given tenant is handled isolated from other tenants using the CNCI's. The CNCIs also implement tenant specific firewall and NAT rules. In the future they may be extended to perform traffic shaping.

Testing

The libsnnet library exposes API's that are used by the launcher and other components of ciao. However the library also includes a reasonably comprehensive unit test framework. The tests can be run as follows

sudo ip link add testdummy type dummy
sudo ip addr add 198.51.100.1/24 dev testdummy
export SNNET_ENV=198.51.100.0/24
sudo -E go test --tags travis -v --short

Note: Some of the API's require Docker 1.11+ to be installed on the test system. Please install Docker to ensure that all the unit tests pass.

These tests also help identify any regressions due to changes in the netlink library or the docker Network and IPAM plugin frameworks.

Documentation

Index

Constants

View Source 
const (
	CnAPITimeout = 6
)

CnAPITimeout default controls the API timeout

View Source 
const (
	MACPrefix = "02:00" //Prefix for all private MAC addresses

)

Various configuration options

Variables

View Source 
var CnMaxAPIConcurrency = 8

CnMaxAPIConcurrency default controls internal API concurrency. It determines how many API's are being actively processed. Can be over-ridden prior to init.

View Source 
var DockerPluginCfg = struct {
	Name    string
	Dir     string
	Addr    string
	DataDir string
	Timeout time.Duration
}{
	Name:    "ciao",
	Dir:     "/etc/docker/plugins",
	Addr:    "127.0.0.1:9999",
	DataDir: "/var/lib/ciao/networking",
	Timeout: 1 * time.Second,
}

DockerPluginCfg controls plugin attributes these may be overridden by the caller if needed

Functions

func DebugSSHPortForIP 

func DebugSSHPortForIP(ip net.IP) (int, error)

DebugSSHPortForIP provides a utility routine that returns the ssh port on the tenant CNCI that can be used to reach a tenant instance with a given IP address

func DockerHandler 

func DockerHandler(d *DockerPlugin,
	fn func(*DockerPlugin, http.ResponseWriter, *http.Request)) http.HandlerFunc

DockerHandler is the default handler for unhandled events It returns error to the caller

func DumpIPTables 

func DumpIPTables() string

DumpIPTables provides a utility routine that returns the current state of the iptables

func EqualNetSlice 

func EqualNetSlice(slice1, slice2 []string) bool

EqualNetSlice compare 2 network slices

func Routing 

func Routing(action FwAction) error

Routing enable or disables routing echo 0 > /proc/sys/net/ipv4/ip_forward echo 1 > /proc/sys/net/ipv4/ip_forward

Types

type APIError 

type APIError struct {
	// contains filtered or unexported fields
}

APIError indicates that the networking call failed. However the system is still consistent and the networking layer has performed appropriate cleanup

func NewAPIError 

func NewAPIError(s string) APIError

NewAPIError is a recoverable error

func (APIError) Error 

func (e APIError) Error() string

type Attrs 

type Attrs struct {
	LinkName string // Locally unique device name
	TenantID string // UUID of the tenant the device belongs to
	// Auto generated. Combination of UUIDs and other params.
	// Typically assigned to the alias
	// It is both locally and globally unique
	// Fully qualifies the device and its role
	GlobalID string
	MACAddr  *net.HardwareAddr
}

Attrs contains fields common to all device types

type Bridge 

type Bridge struct {
	Attrs
	Link *netlink.Bridge
}

Bridge represents a ciao Bridge

type CNContainerEvent 

type CNContainerEvent int

CNContainerEvent to be generated when a Container VNIC is created 

const (
	//ContainerNetworkInfo Informative, no action needs to be taken by caller
	ContainerNetworkInfo CNContainerEvent = 0
	//ContainerNetworkAdd Caller has to create the logical docker network before starting
	//a container with this VNIC
	ContainerNetworkAdd CNContainerEvent = 1 << iota
	//ContainerNetworkDel Caller is responsible for logical docker network deletion
	//The corresponding physical network no longer exists
	ContainerNetworkDel
)

type CNSsntpEvent 

type CNSsntpEvent int

CNSsntpEvent to be generated in response to a VNIC creation This event is sent to the scheduler which will send it to the appropriate CNCI 

const (
	//SsntpNone : Non event
	SsntpNone CNSsntpEvent = 0
	//SsntpTunAdd : Local tunnel added, remote tunnel add required
	SsntpTunAdd CNSsntpEvent = 1 << iota
	//SsntpTunDel : Local tunnel deleted, remote tunnel delete required
	SsntpTunDel
)

type CnAPICtx 

type CnAPICtx struct {
	CancelChan chan interface{}
}

CnAPICtx contains API level context used to control the behaviour of the API, for example, cancellation by invoking close(CancelChan)

type Cnci 

type Cnci struct {
	*NetworkConfig
	MgtAddr     []netlink.Addr //TODO: Remove this and just use the link
	MgtLink     []netlink.Link
	ComputeAddr []netlink.Addr //TODO: Remove this and just use the link
	ComputeLink []netlink.Link

	ID     string // UUID of the concentrator generated by the Controller
	Tenant string // UUID of the tenant

	//APITimeout specifies the amount of time the API will wait for netlink
	//operations to complete. When multiple go routines  invoke the API
	//simultaneously certain netlink calls suffer higher latencies
	APITimeout time.Duration

	// IPAddress of the concentrator that is routable
	// The UUID to IP mapping in this case has to be
	// performed using the datacenter DHCP
	IP net.IP

	// Public IPAddress this concentrator is assigned
	PublicIPs   []net.IP
	PublicIPMap map[string]net.IP //Key is public IPNet
	// contains filtered or unexported fields
}

Cnci represents a Concentrator for a single tenant All subnets belonging to this tenant that are handled by this concentrator. A separate bridge will be setup for each subnet with its own dnsmasq service. Traffic is routable between tenant bridges

func (*Cnci) AddRemoteSubnet 

func (cnci *Cnci) AddRemoteSubnet(subnet net.IPNet, subnetKey int, cnIP net.IP) (string, error)

AddRemoteSubnet attaches a remote subnet to a local bridge on the CNCI If the bridge and DHCP server does not exist it will be created. If the tunnel exists and the bridge does not exist the bridge is created The bridge name interface name is returned if the bridge is newly created

func (*Cnci) DelRemoteSubnet 

func (cnci *Cnci) DelRemoteSubnet(subnet net.IPNet, subnetKey int, cnIP net.IP) error

DelRemoteSubnet detaches a remote subnet from the local bridge The bridge and DHCP server is kept around as they impose minimal overhead and helps in the case where instances keep getting added and deleted constantly

func (*Cnci) Init 

func (cnci *Cnci) Init() error

Init sets the CNCI configuration Discovers the physical interfaces and classifies them as management or compute Performs any node specific networking setup.

func (*Cnci) RebuildTopology 

func (cnci *Cnci) RebuildTopology() error

RebuildTopology CNCI network database using the information contained in the aliases. It can be called if the agent using the library crashes and loses network topology information. It can also be called, to rebuild the network topology on demand. TODO: Restarting the DNS Masq here - Define a re-attach method TODO: Log failures when making best effort progress

func (*Cnci) Shutdown 

func (cnci *Cnci) Shutdown() error

Shutdown stops all DHCP Servers. Tears down all links and tunnels It will continue even on encountering an error and perform as much cleanup as possible

type CnciVnic 

type CnciVnic struct {
	VnicAttrs
	Link *netlink.Macvtap
}

CnciVnic represents a ciao CNCI VNIC This is used to connect a CNCI instance to the network A CNCI VNIC will be directly attached to the data center network Currently we use MacVtap in VEPA mode. We can also use MacVtap in Bridge Mode

type ComputeNode 

type ComputeNode struct {
	ID string //UUID of CN
	*NetworkConfig
	MgtAddr     []netlink.Addr
	MgtLink     []netlink.Link
	ComputeAddr []netlink.Addr
	ComputeLink []netlink.Link
	//APITimeout specifies the amount of time the API will wait for netlink
	//operations to complete. When multiple go routines  invoke the API
	//simultaneously certain netlink calls suffer higher latencies
	APITimeout time.Duration
	// contains filtered or unexported fields
}

ComputeNode describes the high level networking setup of a compute node. The design allows for multiple links, however in phase 0 only the first link is chosen. The remaining are ignored. In the future this allows for backup links or link aggregation or failover

func (*ComputeNode) CreateCnciVnic 

func (cn *ComputeNode) CreateCnciVnic(cfg *VnicConfig) (*CnciVnic, error)

CreateCnciVnic creates a Cnci VNIC and sets all the underlying framework to ensure that the Vnic is active. The Cnci VNIC will bind to the first compute network interface.

func (*ComputeNode) CreateVnic 

func (cn *ComputeNode) CreateVnic(cfg *VnicConfig) (*Vnic, *SsntpEventInfo, *ContainerInfo, error)

CreateVnic creates a tenant VNIC that can be used by containers or VMs This will replace CreateVnic

If this is the first instance of a container Vnic belonging to the tenant on this subnet will provide the ContainerInfo which should be used by the caller of this API to logically create the Network in the Docker network database This is typically done using the command line or API equivalen of docker docker network create -d=ciao --ipam-driver=ciao --subnet=<ContainerInfo.Subnet> --gateway=<ContainerInfo.Gateway> --opt "bridge"=<ContainerInfo.Bridge> ContainerInfo.SubnetID

If this is the first instance of a Vnic belonging to the tenant, will provide a SSNTP message to be sent to the Scheduler to notify the CNCI of this instantiation. This message is processed by the CNCI which will setup the far side of the tunnel which is required to connect this CN tenant bridge to the tenant Subnet

Note: The caller of this function is responsible to send the message to the scheduler

func (*ComputeNode) DbRebuild 

func (cn *ComputeNode) DbRebuild(links []netlink.Link) error

DbRebuild the CN network database using the information contained in the aliases. It can be called if the agent using the library crashes and loses network topology information. It can also be called, to rebuild the network topology on demand.

func (*ComputeNode) DestroyCnciVnic 

func (cn *ComputeNode) DestroyCnciVnic(cfg *VnicConfig) error

DestroyCnciVnic destroys a Cnci VNIC.

func (*ComputeNode) DestroyVnic 

func (cn *ComputeNode) DestroyVnic(cfg *VnicConfig) (*SsntpEventInfo, *ContainerInfo, error)

DestroyVnic destroys a tenant VNIC. If this happens to be the last VNIC for this tenant subnet on this CN, the bridge and GRE tunnel will also be destroyed and SSNTP message generated.

This will replace the DestroyVnic method

This message needs to be sent to the CNCI which will teardown the tunnel. Note: The caller of this function is responsible to send the message to the scheduler or CNCI If the ContainerInfo is set, the container logical network has to be deleted using the command line or API equivalent of docker network rm ContainerInfo.SubnetID>

func (*ComputeNode) Init 

func (cn *ComputeNode) Init() error

Init sets the CN node configuration Discovers the physical interfaces and classifies them as management or compute Performs any node specific networking setup.

func (*ComputeNode) ResetNetwork 

func (cn *ComputeNode) ResetNetwork() error

ResetNetwork will attempt to clean up all network interfaces created. It will not clean up any interfaces created manually

type ContainerInfo 

type ContainerInfo struct {
	CNContainerEvent
	SubnetID string
	Subnet   net.IPNet
	Gateway  net.IP
	Bridge   string
}

ContainerInfo provides details that needed by docker to create the container associated with this VNIC. This event is used by the launcher to instantiate a logical docker network. This event is used synchronize the network state with the state maintained by docker

type DhcpEntry 

type DhcpEntry struct {
	MACAddr  net.HardwareAddr
	IPAddr   net.IP
	Hostname string // Optional
}

DhcpEntry is the fully qualified MAC address to IP mapping

type Dnsmasq 

type Dnsmasq struct {
	SubnetID    string                // UUID of the Tenant Subnet to which the  dnsmasq supports
	CNCIId      string                // UUID of the CNCI instance
	TenantID    string                // UUID of the Tenant to which the CNCI belongs to
	TenantNet   net.IPNet             // The tenant subnet served by this dnsmasq, has to be /29 or larger
	ReservedIPs int                   // Reserve IP at the start of subnet
	ConcIP      net.IP                // IP Address of the CNCI
	IPMap       map[string]*DhcpEntry // Static mac to IP map, key is macaddress
	Dev         *Bridge               // The bridge on which dnsmasq will attach
	MTU         int                   // MTU that takes into account the tunnel overhead
	DomainName  string                // Domain Name to be assigned to the subnet
	// contains filtered or unexported fields
}

Dnsmasq contains all the information required to spawn a dnsmasq process on behalf of a tenant on a concentrator

type DockerDBProvider 

type DockerDBProvider interface {
	//Initializes the Database
	DbInit(dir string) error
	//Populates the DockerPlugin cache from the database
	DbMapRebuild(table string, dockerMap interface{}) error
	//Closes the database
	DbClose() error
	//Creates the tables if the tables do not already exist in the database
	DbTableInit(tables []string) error
	//Adds the key value pair to the table
	DbAdd(table string, key string, value interface{}) error
	//Adds the key value pair to the table
	DbDelete(table string, key string) error
	//Retrives the value corresponding to the key from the table
	DbGet(table string, key string) (interface{}, error)
}

A DockerDBProvider represents a persistent data base provider that can be used by the DockerPlugin to store its internal state

func NewDockerBoltDBProvider 

func NewDockerBoltDBProvider() DockerDBProvider

NewDockerBoltDBProvider returns a bolt based database that conforms to the DockerDBProvider interface

type DockerEpMap 

type DockerEpMap struct {
	sync.Mutex
	// contains filtered or unexported fields
}

DockerEpMap maintains the Endpoint UUID to ciao VNIC mapping

type DockerEpVal 

type DockerEpVal struct {
	ID    string
	IP    string
	Cveth string
	Hveth string
}

DockerEpVal stores ciao VNIC info for a particular docker endpoint

type DockerNwMap 

type DockerNwMap struct {
	sync.Mutex
	// contains filtered or unexported fields
}

DockerNwMap maintains the Docker Network UUID to ciao Network mappings

type DockerNwVal 

type DockerNwVal struct {
	Bridge  string
	Gateway net.IPNet
}

DockerNwVal stores ciao CN tenant bridge mapping

type DockerPlugin 

type DockerPlugin struct {
	DockerDBProvider //Database used to persist the Docker to ciao Mapping
	//This is needed as the Docker Daemon and ciao have
	//different life cycles and UUIDs
	*mux.Router
	*http.Server
	DockerEpMap
	DockerNwMap
	// contains filtered or unexported fields
}

DockerPlugin describes a single instance of a docker plugin In the current design the plugin acts as an IPAM and Network Plugin

func NewDockerPlugin 

func NewDockerPlugin() *DockerPlugin

NewDockerPlugin instantiates a new Docker Plugin instance

func (*DockerPlugin) Close 

func (d *DockerPlugin) Close() error

Close the DockerPlugin This has to be called if the caller has ever performed an Init() failing to close may lead to database corruption

func (*DockerPlugin) Init 

func (d *DockerPlugin) Init() error

Init initializes the docker Plugin. Has to be called after init, but before start.

func (*DockerPlugin) Start 

func (d *DockerPlugin) Start() error

Start the DockerPlugin. This activates the HTTP/HTTPS server The DockerPlugin has to be started prior to the launch of the Docker Daemon

func (*DockerPlugin) Stop 

func (d *DockerPlugin) Stop() error

Stop the DockerPlugin The DockerPlugin has to be stopped after the Docker Daemon has been stopped. If the plugin is stopped when the docker daemon is still active the docker daemon has a timeout and retry mechanism. Hence if the docker plugin is restarted within the retry windows, the docker APIs will still succeed

type FatalError 

type FatalError struct {
	// contains filtered or unexported fields
}

FatalError indicates that the system may be in an inconsistent state due to the error. The caller needs to initiate some sort of recovery. No new workloads should be scheduled on this node until the error is resolved

func NewFatalError 

func NewFatalError(s string) FatalError

NewFatalError is a non recoverable error

func (FatalError) Error 

func (e FatalError) Error() string

type Firewall 

type Firewall struct {
	ExtInterfaces []string
	*iptables.IPTables
}

Firewall defines a single firewall instance

func InitFirewall 

func InitFirewall(devices ...string) (*Firewall, error)

InitFirewall Enables routing on the node and NAT on all external facing interfaces. Enable NAT right away to prevent tenant traffic escape TODO: Only enable external routing. Internal routing should always be enabled

func (*Firewall) ExtFwding 

func (f *Firewall) ExtFwding(action FwAction, extDevice string, intDevice string) error

ExtFwding enables or disables fwding between an externally connected interface and a tenant bridge (hence a tenant subnet) Each tenant subnet created needs explicit enabling/disabling

func (*Firewall) ExtPortAccess 

func (f *Firewall) ExtPortAccess(action FwAction, protocol string, extDevice string,
	externalPort int, internalIP net.IP, internalPort int) error

ExtPortAccess Enables/Disables port access via external device and port to an internal IP address and port for the specified protocol

func (*Firewall) ShutdownFirewall 

func (f *Firewall) ShutdownFirewall() error

ShutdownFirewall Disables routing and NAT TODO: Only external routing should be disabled.

type FwAction 

type FwAction int

FwAction defines firewall action to be performed 

const (
	//FwDisable disables
	FwDisable FwAction = iota
	//FwEnable enables
	FwEnable
)

type GreTunEP 

type GreTunEP struct {
	Attrs
	Link     *netlink.Gretap
	Key      uint32
	LocalIP  net.IP
	RemoteIP net.IP
	CNCIId   string // UUID of the CNCI
	CNId     string // UUID of the CN
}

GreTunEP ciao GRE Tunnel representation This represents one end of the tunnel

type Network 

type Network struct {
	ManagementNet []net.IPNet // Enumerates all possible management subnets
	ComputeNet    []net.IPNet // Enumerates all possible compute subnets
	FloatingPool  []net.IP    // Available floating IPs
	PublicGw      net.IP      // Public IP Gateway to reach the internet
	Mode          NetworkMode
}

Network describes the configuration of the data center network. This is the physical configuration of the data center. The Management Networks carry management/control SSNTP traffic The Compute Network carries tenant traffic. In a simplistic configuration the management network and the compute networks may be one and the same.

type NetworkConfig 

type NetworkConfig struct {
	ManagementNet []net.IPNet // Enumerates all possible management subnets
	ComputeNet    []net.IPNet // Enumerates all possible compute subnets
	Mode          NetworkMode //The data center networking mode
}

NetworkConfig from YAML. This is a subset of the top level data center configuration. Tenant Traffic is carried over the Compute Network. Management Traffic is carried over the Management Network. Both tenant and management traffic can also be carried over the same network if it is not possible to have separate management and compute networks.

type NetworkMode 

type NetworkMode int

NetworkMode describes the networking configuration of the data center 

const (
	// Routed means all traffic is routed with no tenant isolation except through firewall rules
	Routed NetworkMode = iota
	// GreTunnel means tenant instances interlinked using GRE tunnels. Full tenant isolation
	GreTunnel
)

type SsntpEventInfo 

type SsntpEventInfo struct {
	Event     CNSsntpEvent // TYPE: SSNTP Type
	CnciIP    string       // TO: IP Address of the concentrator
	CnIP      string       // FROM: Compute Network IP for this node
	Subnet    string       // Tenant Subnet
	TenantID  string       // Tenant UUID
	SubnetID  string       // Tenant Subnet UUID
	ConcID    string       // CNCI UUID
	CnID      string       // CN UUID
	SubnetKey int
	// contains filtered or unexported fields
}

SsntpEventInfo contains the event info that needs to be converted to YAML payload and sent to the CNCI via the scheduler

type Vnic 

type Vnic struct {
	VnicAttrs
	Link netlink.Link // TODO: Enhance netlink library to add specific tap type to libnetlink
}

Vnic represents a ciao VNIC (typically a tap or veth interface)

type VnicAttrs 

type VnicAttrs struct {
	Attrs
	Role       VnicRole
	InstanceID string // UUID of the instance to which it will attach
	BridgeID   string // ID of bridge it has attached to
	IPAddr     *net.IP
	MTU        int
}

VnicAttrs represent common Vnic attributes

type VnicConfig 

type VnicConfig struct {
	// Per API Context
	// TODO: Move this outside of the VNIC Cfg.
	// Currently placed within the VnicConfig for API backward compatibility
	CnAPICtx
	VnicRole
	VnicIP     net.IP
	ConcIP     net.IP
	VnicMAC    net.HardwareAddr
	MTU        int
	SubnetKey  int //optional: Currently set to SubnetIP
	Subnet     net.IPNet
	VnicID     string // UUID
	InstanceID string // UUID
	TenantID   string // UUID
	SubnetID   string // UUID
	ConcID     string // UUID
}

VnicConfig from YAML All these fields originate from the Controller These fields fully qualify a VNIC to be created on the NN or CN

type VnicRole 

type VnicRole int

VnicRole specifies the role of the VNIC 

const (
	//TenantVM role is assigned to tenant VM
	TenantVM VnicRole = iota //Attached to a VM in the tenant network
	//TenantContainer role is assigned to a tenant container
	TenantContainer //Attach to a container in the tenant network
	//DataCenter role is assigned to resources owned by the data center
	DataCenter //Attached to the data center network
)

Source Files

View all Source files

Directories

Path Synopsis
tests
cncicli
cncli
docker/plugin
parallel

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.