cf-networking-release

README

netman-release

A garden-runc add-on that provides container networking.

Overview

netman provides a batteries included container to container system and several APIs for swapping in third party components.

• IPAM and connectivity are provided by a swappable CNI plugin (flannel in the batteries included case).
• A swappable policy agent polls garden and the policy server for polices to enforce on the cell. In the provided solution, the VXLAN policy agent writes iptables rules to filter packets based on VXLAN gbp tags.
• Inbound traffic from the gorouter is port forwarded from the cell to the container via a NetIn rule. NetIn calls are made by garden to the external networker which then writes the iptables NAT rule.
• Application security groups are enforced by NetOut calls from garden. The external networker also writes iptables rules to enforce ASGs.

Project links

• Design doc for Container Networking Policy
• Engineering backlog
• Chat with us at the #container-networking channel on CloudFoundry Slack
• CI dashboard and config

Install the CF CLI Plugin

1. Get the binary

• Option 1: Download a precompiled binary of the network-policy-plugin for your operating system from our GitHub Releases

• Option 2: Build from source

  go build -o /tmp/network-policy-plugin ./src/cli-plugin
  

1. Install it

chmod +x ~/Downloads/network-policy-plugin
cf install-plugin ~/Downloads/network-policy-plugin

Deploy to bosh-lite

Follow the instructions here to install bosh-lite on your machine.

Ensure that br_netfilter is enabled on your vagrant box:

pushd ~/workspace/bosh-lite
  vagrant ssh -c 'sudo modprobe br_netfilter'
popd

Upload the latest bosh-lite stemcell

bosh upload stemcell https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent

Or download the stemcell and manually upload it to bosh-lite (potentially faster)

curl -L -o bosh-lite-stemcell-latest.tgz https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent
bosh upload stemcell bosh-lite-stemcell-latest.tgz

Then grab the required releases

pushd ~/workspace
  git clone https://github.com/cloudfoundry/diego-release
  git clone https://github.com/cloudfoundry/cf-release
  git clone https://github.com/cloudfoundry-incubator/netman-release
popd

Then deploy

pushd ~/workspace/netman-release
  ./scripts/deploy-to-bosh-lite
popd

Then follow the instructions for testing with the cats & dogs example.

Deploy to AWS

1. Upload stemcell with Linux kernel 4.4 to bosh director. Versions >= 3263.2 should work.
2. Create netman stubs

• netman requires additional information in several stubs.

• netman supports connecting to etcd over TLS. To generate the certs, use ./scripts/generate-etcd-certs in cf-release and add them to the properties in stubs/cf/properties.yml:

  etcd:
    require_ssl: true
    peer_require_ssl: false # Required if you don't want communication between etcd nodes to be secured via TLS
    ca_cert: <contents of cf-release/etcd-certs/etcd-ca.crt>
    server_cert: <contents of cf-release/etcd-certs/server.crt>
    server_key: <contents of cf-release/etcd-certs/server.key>
    client_cert: <contents of cf-release/etcd-certs/client.crt>
    client_key: <contents of cf-release/etcd-certs/client.key>
  

• Add under properties: uaa in stubs/cf/properties.yml:

  scim:
    users:
    - admin|<admin-password>|scim.write,scim.read,openid,cloud_controller.admin,doppler.firehose,network.admin
  clients:
    cf:
      scope: cloud_controller.read,cloud_controller.write,openid,password.write,cloud_controller.admin,scim.read,scim.write,doppler.firehose,uaa.user,routing.router_groups.read,network.admin
  

• Create a netman stub stubs/netman/stub.yml, fill in all REPLACE_ME fields with the appropriate credentials:

  ---
  netman_overrides:
    releases:
    - name: netman
      version: latest
    driver_templates:
    - name: garden-cni
      release: netman
    - name: cni-flannel
      release: netman
    - name: netmon
      release: netman
    - name: vxlan-policy-agent
      release: netman
    properties:
      vxlan-policy-agent:
        policy_server_url: http://policy-server.service.cf.internal:4002
      garden-cni:
        cni_plugin_dir: /var/vcap/packages/flannel/bin
        cni_config_dir: /var/vcap/jobs/cni-flannel/config/cni
      cni-flannel:
        flannel:
          etcd:
            require_ssl: true
        etcd_endpoints:
          - (( "https://" config_from_cf.etcd.advertise_urls_dns_suffix ":4001" ))
        etcd_ca_cert: REPLACE_WITH_ETCD_CA_CERT_FROM_CF_MANIFEST
        etcd_client_cert: REPLACE_WITH_ETCD_CLIENT_CERT_FROM_CF_MANIFEST
        etcd_client_key: REPLACE_WITH_ETCD_CLIENT_KEY_FROM_CF_MANIFEST
      policy-server:
        uaa_client_secret: REPLACE_WITH_UAA_CLIENT_SECRET
        uaa_url: (( "https://uaa." config_from_cf.system_domain ))
        skip_ssl_validation: true
        database:
          type: REPLACE_WITH_DB_TYPE # mysql or postgres
          connection_string: REPLACE_WITH_DB_CONNECTION_STRING
    garden_properties:
      network_plugin: /var/vcap/packages/runc-cni/bin/garden-external-networker
      network_plugin_extra_args:
      - --configFile=/var/vcap/jobs/garden-cni/config/adapter.json
    jobs:
    - name: policy-server
      instances: 1
      persistent_disk: 256
      templates:
      - name: policy-server
        release: netman
      - name: route_registrar
        release: cf
      - name: consul_agent
        release: cf
      - name: metron_agent
        release: cf
      resource_pool: database_z1
      networks:
        - name: diego1
      properties:
        nats:
          machines: (( config_from_cf.nats.machines ))
          user: (( config_from_cf.nats.user ))
          password: (( config_from_cf.nats.password ))
          port: (( config_from_cf.nats.port ))
        metron_agent:
          zone: z1
        route_registrar:
          routes:
          - name: policy-server
            port: 4002
            registration_interval: 20s
            uris:
            - (( "api." config_from_cf.system_domain "/networking" ))
        consul:
          agent:
            services:
              policy-server:
                name: policy-server
                check:
                  interval: 5s
                  script: /bin/true
  
  config_from_cf: (( merge ))
  

1. Generate diego with netman manifest:

• Run the following bash script. Set environment_path to the directory containing your stubs for cf, diego, and netman. Set output_path to the directory you want your manifest to be created in. Set diego_release_path to your local copy of the diego-release repository.

set -e -x -u

environment_path=
output_path=
diego_release_path=

pushd cf-release
  ./scripts/generate_deployment_manifest aws \
    ${environment_path}/stubs/director-uuid.yml \
    ${diego_release_path}/examples/aws/stubs/cf/diego.yml \
    ${environment_path}/stubs/cf/properties.yml \
    ${environment_path}/stubs/cf/instance-count-overrides.yml \
    ${environment_path}/stubs/cf/stub.yml \
    > ${output_path}/cf.yml
popd

pushd diego-release
  ./scripts/generate-deployment-manifest \
    -g \
    -c ${output_path}/cf.yml \
    -i ${environment_path}/stubs/diego/iaas-settings.yml \
    -p ${environment_path}/stubs/diego/property-overrides.yml \
    -n ${environment_path}/stubs/diego/instance-count-overrides.yml \
    -N ${environment_path}/stubs/netman/stub.yml \
    -v ${environment_path}/stubs/diego/release-versions.yml \
    > ${output_path}/diego.yml
popd

1. Deploy

• Target your bosh director.

bosh target <your-director>

• Set the deployment

bosh deployment ${output_path}/diego.yml

• Deploy

bosh deploy

Other infrastructures

We do not currently test this software on infrastructures other than BOSH-lite and AWS. With recent stemcells and the appropriate manifest changes, it should work. Let us know if you find issues.

To replace flannel with your own CNI plugin

1. Remove the following BOSH jobs:

• cni-flannel
• vxlan-policy-agent

1. Remove the following BOSH packages:

• flannel
• flannel-watchdog

1. Add in all packages and jobs required by your CNI plugin. At a minimum, you must provide a CNI binary program and a CNI config file.

• For more info on bosh packaging scripts read this.
• For more info on bosh jobs read this.

1. Update the deployment manifest properties

garden-cni:
  adapter:
    cni_plugin_dir: /var/vcap/packages/YOUR_PACKAGE/bin # your CNI binary goes in this directory
    cni_config_dir: /var/vcap/jobs/YOUR_JOB/config/cni  # your CNI config file goes in this directory

Remove any lingering references to flannel or cni-flannel in the deployment manifest.

To deploy a local-only (no-op) CNI plugin

As a baseline, you can deploy using only the basic bridge CNI plugin.

This plugin will provide connectivity between containers on the same Garden host (Diego cell) but will not provide a cross-host network. However, it can be a useful baseline configuration for testing and development.

cd bosh-lite
bosh target lite
bosh update cloud-config cloud-config.yml
bosh deployment local-only.yml
bosh deploy

To deploy diego with CNI but without cross-host container networking

For generating a cloudfoundry-diego deployment without container to container connectivity, but using the CNI bridge plugin for NAT'ed connectivity.

CNI_BRIDGE=true ./scripts/generate-bosh-lite-manifests
bosh deploy

Development

Running low-level tests

~/workspace/netman-release/scripts/docker-test

Running the full acceptance test on bosh-lite

WARNING: This test is taxing and has an aggressive timeout. It may fail on a laptop or other underpowered bosh-lite.

cd src/netman-cf-acceptance
./run-locally.sh

Referencing a new library from existing BOSH package

1. Add any new libraries into the submodule from the root of the repo

cd $GOPATH
git submodule add https://github.com/foo/bar src/github.com/foo/bar
./scripts/sync-package-specs

Adding a new BOSH package

1. Add any new libraries into the submodules from the root of the repo

cd $GOPATH
git submodule add https://github.com/foo/bar src/github.com/foo/bar

1. Update the package sync script:

vim $GOPATH/scripts/sync-package-specs

Find or create the sync_package line for baz

1. Run the sync script:

./scripts/sync-package-specs

Syslog forwarding

To forward all logs from your bosh-lite to a syslog destination (like Papertrail), add the following block to manifest-generation/stubs/bosh-lite-cf.yml:

  syslog_daemon_config:
    address: some-syslog-host.example.com
    port: 12345
    transport: udp