super_mode

README

Etherguard

中文版

This is the documentation of the super_mode of this example_config Before reading this, I'd like to suggest you read the static mode first.

Super mode

Super mode are inspired by n2n
We have two types of node, we called it super node and edge node.

All edge nodes have to connect to super node, exchange data and UDP hole punch each other by super node.
The super node runs the Floyd-Warshall Algorithm， and distribute the result to all edge node.

In the super mode of the edge node, the nexthoptable and peers section are useless. All infos are download from super node.
Meanwhile, super node will generate pre shared key for inter-edge communication(if usepskforinteredge enabled).

psk = shs256("PubkeyPeerA" + "PubkeyPeerB" + "Chef Special and Featured in the season see salt")[:32]

SuperMsg

There are new type of DstID called SuperMsg(65534). All packets sends to and receive from super node are using this packet type.
This packet will not send to any other edge node, just like DstID == self.NodeID

Control Message

In Super mode, Beside Normal Packet. We introduce a new packet type called Control Message. In Super mode, we will not relay any control message. We just receive or send it to target directly.
We list all the control message we use in the super mode below.

Register

This control message works like this picture:

1. edge node send Register to the super node
2. Supernode knows it's external IP and port number
3. Update it to database and distribute UpdatePeerMsg to all edges
4. Other edges get the notification, download the updated peer infos from supernode via HTTP API

Ping/Pong

While edges get the peer infos, edges will start trying to talk each other directly like this picture:

1. Send Ping to all other edges with local time with TTL=0
2. Received a Ping, Subtract the peer time from local time, we get a single way latency.
3. Send a Pong to supernode, let supernode calculate the NextHopTable
4. Wait the supernode push UpdateNhTable message and download it.

UpdateNhTable

While supernode get a Pong message, it will run the Floyd-Warshall Algorithm to calculate the NextHopTable
If there are any changes of this table, it will distribute UpdateNhTable to all edges to till then download the latest NextHopTable via HTTP API as soon as possible.

UpdateError

Notify edges that an error has occurred, and close the edge
It occurs when the version number is not match with supernode, or the NodeID of the edge is configured incorrectly, or the edge is deleted.

HTTP API

Why we use HTTP API instead of pack all information in the UpdateXXX?
Because UDP is an unreliable protocol, there is an limit on the amount of content that can be carried.
But the peer list contains all the peer information, the length is not fixed, it may exceed
So we use UpdateXXX to tell we have a update, please download the latest information from supernode via HTTP API as soon as possible. And UpdateXXX itself is not reliable, maybe it didn't reach the edge node at all.
So the information of UpdateXXX carries the state hash. Bring it when with HTTP API. When the super node receives the HTTP API and sees the state hash, it knows that the edge node has received the UpdateXXX.
Otherwise, it will send UpdateXXX to the node again after few seconds.

HTTP Guest API

HTTP also has some APIs for the front-end to help manage the entire network

peerstate

curl "http://127.0.0.1:3000/api/peerstate?Password=passwd"

It can show some information such as single way latency or last seen time.
We can visualize it by Force-directed graph drawing.

There is an Infinity section in the json response. It should be 9999. It means infinity if the number larger than it.
Cuz json can't present infinity so that I use this trick.
While we see the latency larger than this, we doesn't need to draw lines in this two nodes.

Example return value:

{
  "PeerInfo": {
    "1": {
      "Name": "hk",
      "LastSeen": "2021-09-29 11:23:22.854700559 +0000 UTC m=+28740.116476977"
    },
    "1001": {
      "Name": "relay_kr",
      "LastSeen": "2021-09-29 11:23:21.277417897 +0000 UTC m=+28738.539194315"
    },
    "121": {
      "Name": "za_north",
      "LastSeen": "0001-01-01 00:00:00 +0000 UTC"
    },
    "33": {
      "Name": "us_west",
      "LastSeen": "2021-09-29 11:23:13.257033252 +0000 UTC m=+28730.518809670"
    },
    "49": {
      "Name": "us_east",
      "LastSeen": "2021-09-29 11:23:16.606165241 +0000 UTC m=+28733.867941659"
    },
    "51": {
      "Name": "ca_central",
      "LastSeen": "0001-01-01 00:00:00 +0000 UTC"
    },
    "65": {
      "Name": "fr",
      "LastSeen": "2021-09-29 11:23:19.4084596 +0000 UTC m=+28736.670236018"
    },
    "81": {
      "Name": "au_central",
      "LastSeen": "0001-01-01 00:00:00 +0000 UTC"
    },
    "89": {
      "Name": "uae_north",
      "LastSeen": "0001-01-01 00:00:00 +0000 UTC"
    },
    "9": {
      "Name": "jp_east",
      "LastSeen": "2021-09-29 11:23:16.669505147 +0000 UTC m=+28733.931281565"
    },
    "97": {
      "Name": "br_south",
      "LastSeen": "0001-01-01 00:00:00 +0000 UTC"
    }
  },
  "Infinity": 99999,
  "Edges": {
    "1": {
      "1001": 0.033121187,
      "33": 0.075653164,
      "49": 0.100471502,
      "65": 0.065714769,
      "9": 0.022864241
    },
    "1001": {
      "1": 0.018561948,
      "33": 0.064077348,
      "49": 0.094459818,
      "65": 0.079481599,
      "9": 0.011163433
    },
    "33": {
      "1": 0.075263428,
      "1001": 0.070029457,
      "49": 0.032631349,
      "65": 0.045575061,
      "9": 0.050444255
    },
    "49": {
      "1": 0.100271358,
      "1001": 0.100182834,
      "33": 0.034563118,
      "65": 0.017950046,
      "9": 0.07510982
    },
    "65": {
      "1": 0.114219741,
      "1001": 0.132759205,
      "33": 0.095265063,
      "49": 0.067413235,
      "9": 0.127562362
    },
    "9": {
      "1": 0.026909699,
      "1001": 0.022555855,
      "33": 0.056469043,
      "49": 0.090400723,
      "65": 0.08525314
    }
  },
  "NhTable": {
    "1": {
      "1001": 1001,
      "33": 33,
      "49": 49,
      "65": 65,
      "9": 9
    },
    "1001": {
      "1": 1,
      "33": 33,
      "49": 49,
      "65": 65,
      "9": 9
    },
    "33": {
      "1": 1,
      "1001": 1001,
      "49": 49,
      "65": 65,
      "9": 9
    },
    "49": {
      "1": 1,
      "1001": 9,
      "33": 33,
      "65": 65,
      "9": 9
    },
    "65": {
      "1": 1,
      "1001": 1001,
      "33": 33,
      "49": 49,
      "9": 9
    },
    "9": {
      "1": 1,
      "1001": 1001,
      "33": 33,
      "49": 33,
      "65": 65
    }
  },
  "Dist": {
    "1": {
      "1": 0,
      "1001": 0.033121187,
      "33": 0.075119328,
      "49": 0.102236885,
      "65": 0.074688856,
      "9": 0.022473723
    },
    "1001": {
      "1": 0.018561948,
      "1001": 0,
      "33": 0.064077348,
      "49": 0.094459818,
      "65": 0.079481599,
      "9": 0.011163433
    },
    "33": {
      "1": 0.075263428,
      "1001": 0.070029457,
      "33": 0,
      "49": 0.032631349,
      "65": 0.045575061,
      "9": 0.050444255
    },
    "49": {
      "1": 0.100271358,
      "1001": 0.097665675,
      "33": 0.034563118,
      "49": 0,
      "65": 0.017950046,
      "9": 0.07510982
    },
    "65": {
      "1": 0.114219741,
      "1001": 0.132759205,
      "33": 0.095265063,
      "49": 0.067413235,
      "65": 0,
      "9": 0.127562362
    },
    "9": {
      "1": 0.026909699,
      "1001": 0.022555855,
      "33": 0.056469043,
      "49": 0.089100392,
      "65": 0.08525314,
      "9": 0
    }
  }
}

Section meaning:

1. PeerInfo: NodeID，Name，LastSeen
2. Edges: The Single way latency，9999 or missing means unreachable(UDP hole punching failed)
3. NhTable: Calculate result.
4. Dist: The latency of packet through Etherguard

peeradd

We can add new edges with this API without restart the supernode

Exanple:

curl -X POST "http://127.0.0.1:3000/api/peer/add?Password=passwd_addpeer" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "nodeid=100&name=Node_100&additionalcost=1000&pubkey=6SuqwPH9pxGigtZDNp3PABZYfSEzDaBSwuThsUUAcyM="

Parameter:

1. URL query: Password: Password. Configured in the config file.
2. Post body:
   1. nodeid: Node ID
   2. pubkey: Public Key
   3. pskey: Pre shared Key
   4. additionalcost: Additional cost for packet transfer. Unit: ms
   5. nexthoptable: If the graphrecalculatesetting of your super node is in static mode, you need to provide a new NextHopTable in json format in this parameter.

Return value:

1. http code != 200: Error reason
2. http code == 200，An example edge config.
   • generate by contents in edgetemplate with custom data (nodeid/name/pubkey)
   • Convenient for users to copy and paste

interface:
  itype: stdio
  name: tap1
  vppifaceid: 1
  vppbridgeid: 4242
  macaddrprefix: AA:BB:CC:DD
  mtu: 1416
  recvaddr: 127.0.0.1:4001
  sendaddr: 127.0.0.1:5001
  l2headermode: kbdbg
nodeid: 100
nodename: Node_100
defaultttl: 200
privkey: Your_Private_Key
listenport: 3001
loglevel:
  loglevel: normal
  logtransit: true
  logcontrol: true
  lognormal: true
  logntp: true
dynamicroute:
  sendpinginterval: 16
  peeralivetimeout: 30
  dupchecktimeout: 40
  conntimeout: 30
  connnexttry: 5
  savenewpeers: true
  supernode:
    usesupernode: true
    pskey: ""
    connurlv4: 127.0.0.1:3000
    pubkeyv4: LJ8KKacUcIoACTGB/9Ed9w0osrJ3WWeelzpL2u4oUic=
    connurlv6: ""
    pubkeyv6: HCfL6YJtpJEGHTlJ2LgVXIWKB/K95P57LHTJ42ZG8VI=
    apiurl: http://127.0.0.1:3000/api
    supernodeinfotimeout: 50
  p2p:
    usep2p: false
    sendpeerinterval: 20
    graphrecalculatesetting:
      jittertolerance: 20
      jittertolerancemultiplier: 1.1
      nodereporttimeout: 40
      recalculatecooldown: 5
  ntpconfig:
    usentp: true
    maxserveruse: 8
    synctimeinterval: 3600
    ntptimeout: 3
    servers:
    - time.google.com
    - time1.google.com
    - time2.google.com
    - time3.google.com
    - time4.google.com
    - time1.facebook.com
    - time2.facebook.com
    - time3.facebook.com
    - time4.facebook.com
    - time5.facebook.com
    - time.cloudflare.com
    - time.apple.com
    - time.asia.apple.com
    - time.euro.apple.com
    - time.windows.com
nexthoptable: {}
resetconninterval: 86400
peers: []

peerdel

Delete peer

There are two deletion modes, namely password deletion and private key deletion.
Designed to be used by administrators, or for people who join the network and want to leave the network.

Use Password to delete any node. Take the newly added node above as an example, use this API to delete the node

curl "http://127.0.0.1:3000/api/peer/del?Password=passwd_delpeer&nodeid=100"

We can also use privkey to delete, the same as above, but use privkey parameter only.

curl "http://127.0.0.1:3000/api/peer/del?privkey=IJtpnkm9ytbuCukx4VBMENJKuLngo9KSsS1D60BqonQ="

Parameter:

1. URL query:
   1. Password: Password: Password. Configured in the config file.
   2. nodeid: Node ID that you want to delete
   3. privkey: The private key of the edge

Return value:

1. http code != 200: Error reason
2. http code == 200: Success message

Config Parameters

Super mode of edge node

1. usesupernode: Whether to enable Super mode
2. pskey: Pre shared Key used to establish connection with supernode
3. connurlv4: IPv4 connection address of the Super node
4. pubkeyv4: IPv4 key of Super node
5. connurlv6: IPv6 connection address of the Super node
6. pubkeyv6: IPv6 key of Super node
7. apiurl: HTTP(S) API connection address of Super node
8. supernodeinfotimeout: Supernode Timeout

Super node it self

1. nodename: node name
2. privkeyv4: private key for ipv4
3. privkeyv6: private key for ipv6
4. listenport: listen udp port number
5. loglevel: Refer to README.md
6. repushconfiginterval: re-push interval of UpdateXXX messages
7. passwords: HTTP API password
   1. showstate: node information
   2. addpeer: add peer
   3. delpeer: delete peer
8. graphrecalculatesetting: Some parameters related to [Floyd-Warshall algorithm](https://zh.wikipedia.org/zh-tw/Floyd-Warshall algorithm)
   1. staticmode: Disable the Floyd-Warshall algorithm and only use the nexthoptable loaded at the beginning.
      Supernode is only used to assist hole punching
   2. recalculatecooldown: Floyd-Warshal is O(n^3) time complexity algorithm, which cannot be calculated too often. Set a cooling time
   3. jittertolerance: jitter tolerance, after receiving Pong, one 37ms and one 39ms will not trigger recalculation
   4. jittertolerancemultiplier: the same is the jitter tolerance, but high ping allows more errors https://www.desmos.com/calculator/raoti16r5n
   5. nodereporttimeout: The timeout of the received Pong packet. Change back to Infinity after timeout.
   6. timeoutcheckinterval: The interval to check if there any Pong packet timeouted, and recalculate the NhTable
9. nexthoptable: only works in staticmode==true, set nexthoptable manually
10. edgetemplate: for addpeer API. Refer to this configuration file and show a sample configuration file of the edge to the user
11. usepskforinteredge: Whether to enable pre-share key communication between edges. If enabled, supernode will generate PSKs for edges automatically
12. peers: Peer list, refer to README.md
    1. nodeid: Peer's node ID
    2. name: Peer name (displayed on the front end)
    3. pubkey: peer public key
    4. pskey: preshared key The PSK that this peer connects to this Supernode
    5. additionalcost: Additional cost for packet transfer. Unit: ms

V4 V6 Two Keys

Why we split IPv4 and IPv6 into two session? Because of this situation

In this case, SuperNode does not know the external ipv4 address of Node02 and cannot help Node1 and Node2 to UDP hole punch.

So like this, both V4 and V6 establish a session, so that both V4 and V6 can be taken care of at the same time.

UDP hole punch reachability

For different NAT type, the UDP hole punch reachability can refer this table.(Origin)

And if both sides are using ConeNAT, it's not gerenteed to punch success. It depends on the topology and the devices attributes.
Like the section 3.5 in this article, we can't punch success.

Notice for Relay node

Unlike n2n, our supernode do not relay any packet for edges.
If the edge punch failed and no any route available, it's just unreachable. In this case we need to setup a relay node.

Relay node is a regular edge in public network, but interface=dummy.

And we have to note that do not use 127.0.0.1 to connect to supernode.
Because supernode well distribute the source IP of the nodes to all other edges. But 127.0.0.1 is not accessible from other edge.

To avoid this issue, please use the external IP of the supernode in the edge config.

Quick start

Run this example_config (please open three terminals):

./etherguard-go -config example_config/super_mode/s1.yaml -mode super
./etherguard-go -config example_config/super_mode/n1.yaml -mode edge
./etherguard-go -config example_config/super_mode/n2.yaml -mode edge

Because it is in stdio mode, stdin will be read into the VPN network
Please type in one of the edge windows

b1aaaaaaaaaa

b1 will be converted into a 12byte layer 2 header, b is the broadcast address FF:FF:FF:FF:FF:FF, 1 is the ordinary MAC address AA:BB:CC:DD:EE:01, aaaaaaaaaa is the payload, and then feed it into the VPN
You should be able to see the string b1aaaaaaaaaa on another window. The first 12 bytes are converted back

Next: P2P Mode