README
¶
Simple and Secure Node Transfer Protocol
Overview
The Simple and Secure Node Transfer Protocol (SSNTP) is a custom, fully asynchronous and TLS based application layer protocol. All Cloud Integrated Advanced Orchestrator (CIAO) components communicate with each others over SSNTP.
SSNTP is designed with simplicity, efficiency and security in mind:
- All SSNTP entities are identified by a Universal Unique IDentifier (UUID).
- All SSNTP frame headers are identical for easy parsing.
- SSNTP payloads are optional.
- SSNTP payloads are YAML formatted.
- SSNTP is a one way protocol where senders do not receive a synchronous answer from the receivers.
- Any SSNTP entity can asynchronously send a command, status or event to one of its peers.
SSNTP clients and servers
The SSNTP protocol defines 2 entities: SSNTP clients and SSNTP servers.
A SSNTP server listens for and may accept connections from many SSNTP clients. It never initiates a connection to another SSNTP entity.
A SSNTP client initiates a connection to a SSNTP server and can only connect to one single server at a time. It does not accept incoming connections from another SSNTP entity.
Once connected, both clients and servers can initiate SSNTP transfers at any point in time without having to wait for any kind of SSNTP acknowledgement from the other end of the connection. SSNTP is a fully asynchronous protocol.
Roles
All SSNTP entities must declare their role at connection time, as part of their signed certificate extended key usage attributes.
SSNTP roles allow for:
- SSNTP frames filtering: Depending on the declared role of the sending entity, the receiving party can choose to discard frames and optionally send a frame rejection error back.
- SSNTP frames routing: A SSNTP server implementation can configure frame forwarding rules for multicasting specific received SSNTP frame types to all connected SSNTP clients with a given role.
There are currently 6 SSNTP different roles:
- SERVER (0x1): A generic SSNTP server.
- Controller (0x2): The CIAO Command and Status Reporting client.
- AGENT (0x4): The CIAO compute node Agent. It receives workload commands from the Scheduler and manages workload on a given compute node accordingly.
- SCHEDULER (0x8): The CIAO workload Scheduler. It receives workload related commands from the Controller and schedules them on the available compute nodes.
- NETAGENT (0x10): The CIAO networking compute node Agent. It receives networking workload commands from the Scheduler and manages workload on a given networking compute node accordingly.
- CNCIAGENT (0x20): A Compute Node Concentrator Instance Agent runs within the networking node workload and manages a specific tenant private network. All instances for this tenant will have a GRE tunnel established between them and the CNCI, and the CNCI acts as the tenant routing entity.
SSNTP connection
Before a SSNTP client is allowed to send any frame to a SSNTP server, or vice versa, both need to successfully go through the SSNTP connection protocol. The SSNTP connection is a mandatory step for the client and the server to verify each other's roles and also to retrieve each other's UUIDs.
-
SSNTP client sends a CONNECT command to the SSNTP server. This frame contains the advertised SSNTP client and this should match the client's certificate extended key usage attributes. The server will verify that both match and if they don't it will send a SSNTP error frame back with a ConnectionAborted (0x6) error code. The CONNECT frame destination UUID is the nil UUID as the client does not know the server UUID before getting its CONNECTED frame.
-
The server asynchronously sends a CONNECTED status frame to the client in order to notify him about a successful connection. The CONNECTED frame contains the server advertised role and the cluster configuration data in its payload. The client must verify that the server role matches its certificate extended key usage attributes. If that verification fails the client must send a SSNTP error frame to the server where the error code is ConnectionFailure (0x4), and then must close the TLS connection to the server. The client should also parse the cluster [configuration data] (https://github.com/01org/ciao/blob/master/payloads/configure.go) that comes in the CONNECTED payload and configure itself accordingly.
-
Connection is successfully established. Both ends of the connection can now asynchronously send SSNTP frames.
SSNTP frames
Each SSNTP frame is composed of a fixed length, 8 bytes long header and an an optional YAML formatted payload.
SSNTP header
+----------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length |
| (1 byte) | (1 byte) | (1 byte) | (1 byte) | or Role (4 bytes) |
+----------------------------------------------------------------+
- Major is the SSNTP version major number. It is currently 0.
- Minor is the SSNTP version minor number. It is currently 1.
- Type is the SSNTP frame type. There are 4 different frame types: COMMAND, STATUS, EVENT and ERROR.
- Operand is the SSNTP frame sub-type.
- Payload length is the optional YAML formatted SSNTP payload length in bytes. It is set to zero for payload less frames.
- Role is the SSNTP entity role. Only the CONNECT command and CONNECTED status frames are using this field as a role descriptor.
SSNTP COMMAND frames
There are 10 different SSNTP COMMAND frames:
CONNECT
CONNECT must be the first frame SSNTP clients send when trying to connect to a SSNTP server. Any frame sent to a SSNTP server from a client that did not initially sent a CONNECT frame will be discarded and the TLS connection to the client will be closed.
The purpose of the CONNECT command frame is for the client to advertise its role and for the server to verify that the advertised role matches the client's certificate extended key usage attributes.
The CONNECT frame is payloadless and its Destination UUID is the nil UUID:
+--------------------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Role | Client UUID | Nil UUID |
| | | (0x0) | (0x0) | (bitmask of client roles) | | |
+--------------------------------------------------------------------------------------+
START
The CIAO Controller client sends the START command to the Scheduler in order to schedule a new workload. The [START command YAML payload] (https://github.com/01org/ciao/blob/master/payloads/start.go) is mandatory and contains a full workload description.
If the Scheduler finds a compute node (CN) with enough capacity to run this workload, it will then send a START command to the given Agent UUID managing this CN with the same payload.
If the Scheduler cannot find a suitable CN for this workload, it will asynchronously send a SSNTP ERROR frame back to the Controller. The error code should be StartFailure (0x2) and the payload must comply with the [StartFailure YAML schema] (https://github.com/01org/ciao/blob/master/payloads/startfailure.go) so that the Controller eventually knows that a given instance/workload UUID could not start.
Once the Scheduler has sent the START command to an available CN Agent, it is up to this Agent to actually initialize and start an instance that matches the START YAML payload. If that fails the Agent should asynchronously sends a SSNTP ERROR back to the Scheduler and the error code should be StartFailure (0x2). The Scheduler must then forward that error frame to the Controller.
The START command payload is mandatory:
+--------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted |
| | | (0x0) | (0x1) | | workload description |
+--------------------------------------------------------------------------+
STOP
The CIAO Controller client sends the STOP command to the Scheduler in order to stop a running instance on a given CN. The [STOP command YAML payload] (https://github.com/01org/ciao/blob/master/payloads/stop.go) is mandatory and contains the instance UUID to be stopped and the agent UUID that manages this instance.
STOPping an instance means shutting it down. Non persistent instances are deleted as well when being STOPped. Persistent instances metadata and disks images are stored and can be started again through the RESTART SSNTP command.
There are several error cases related to the STOP command:
-
If the Scheduler cannot find the Agent identified in the STOP command payload, it should send a SSNTP error with the StopFailure (0x3) error code back to the Controller.
-
If the Agent cannot actually stop the instance (Because e.g. it's already finished), it should also send a SSNTP error with the StopFailure (0x3) error code back to the Scheduler. It is then the Scheduler responsibility to notify the Controller about it by forwarding this error frame.
+--------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted |
| | | (0x0) | (0x2) | | payload |
+--------------------------------------------------------------------+
STATS
CIAO CN Agents periodically send the STATS command to the Scheduler in order to provide a complete view of the compute node status. It is up to the CN Agent implementation to define the STATS sending period.
Upon reception of Agent STATS commands, the Scheduler must forward it to the Controller so that it can provide a complete cloud status report back to the users.
The STATS command comes with a mandatory [YAML formatted payload] (https://github.com/01org/ciao/blob/master/payloads/stats.go).
+----------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted compute |
| | | (0x0) | (0x3) | | node statistics |
+----------------------------------------------------------------------------+
EVACUATE
The CIAO Controller client sends EVACUATE commands to the Scheduler to ask a specific CIAO Agent to evacuate its compute node, i.e. stop and migrate all of the current workloads it is monitoring on its node.
The [EVACUATE YAML payload] (https://github.com/01org/ciao/blob/master/payloads/evacuate.go) is mandatory and describes the next state to reach after evacuation is done. It could be 'shutdown' for shutting the node down, 'update' for having it run a software update, 'reboot' for rebooting the node or 'maintenance' for putting the node in maintenance mode:
+---------------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted compute |
| | | (0x0) | (0x4) | | node next state description |
+---------------------------------------------------------------------------------+
DELETE
The CIAO Controller client may send DELETE commands in order to completely remove an already STOPped instance from the cloud. This command is only relevant for persistent workload based instances as non persistent instances are implicitly deleted when being STOPed.
Deleting a persistent instance means completely removing it from the cloud and thus it should no longer be reachable for e.g. a RESTART command.
When asked to delete a non existing instance the CN Agent must reply with a DeleteFailure error frame.
The [DELETE YAML payload schema] (https://github.com/01org/ciao/blob/master/payloads/stop.go) is the same as the STOP one.
+--------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted |
| | | (0x0) | (0x5) | | payload |
+--------------------------------------------------------------------+
RESTART
The CIAO Controller client may send RESTART commands in order to restart previously STOPped persistent instances. Non persistent instances cannot be RESTARTed as they are implicitly deleted when being STOPped.
When asked to restart a non existing instance the CN Agent must reply with a RestartFailure error frame.
The [RESTART YAML payload schema] (https://github.com/01org/ciao/blob/master/payloads/start.go) is the same as the STOP one.
+--------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted |
| | | (0x0) | (0x6) | | payload |
+--------------------------------------------------------------------+
AssignPublicIP
AssingPublicIP is a command sent by the Controller to assign a publicly routable IP to a given instance. It is sent to the Scheduler and must be forwarded to the right CNCI.
The public IP is fetched from a pre-allocated pool managed by the Controller.
The [AssignPublicIP YAML payload schema] (https://github.com/01org/ciao/blob/master/payloads/assignpublicIP.go) is made of the CNC, the tenant and the instance UUIDs, the allocated public IP and the instance private IP and MAC.
+--------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted |
| | | (0x0) | (0x7) | | payload |
+--------------------------------------------------------------------+
ReleasePublicIP
ReleasePublicIP is a command sent by the Controller to release a publicly routable IP from a given instance. It is sent to the Scheduler and must be forwarded to the right CNCI.
The released public IP is added back to the Controller managed IP pool.
The [ReleasePublicIP YAML payload schema] (https://github.com/01org/ciao/blob/master/payloads/assignpublicIP.go) is made of the CNCI and a tenant UUIDs, the released public IP, the instance private IP and MAC.
+--------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted |
| | | (0x0) | (0x8) | | payload |
+--------------------------------------------------------------------+
CONFIGURE
CONFIGURE commands are sent to request any SSNTP entity to configure itself according to the CONFIGURE command payload. Controller or any SSNTP client handling user interfaces defining any cloud setting (image service, networking configuration, identity management...) must send this command for any configuration change and for broadcasting the initial cloud configuration to all CN and NN agents.
CONFIGURE commands should be sent in the following cases:
- At cloud boot time, as a broadcast command.
- For every cloud configuration change.
- Every time a new agent joins the SSNTP network.
The [CONFIGURE YAML payload] (https://github.com/01org/ciao/blob/master/payloads/configure.go) always includes the full cloud configuration and not only changes compared to the last CONFIGURE command sent.
+-----------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted payload |
| | | (0x0) | (0x9) | | |
+-----------------------------------------------------------------------------+
SSNTP STATUS frames
There are 5 different SSNTP STATUS frames:
CONNECTED
CONNECTED is sent by SSNTP servers back to a client to notify it that the connection successfully completed.
From the CONNECTED frame the client will gather 2 pieces of information:
- The server UUID. This UUID will be used as the destination UUID for every frame the client sends going forward.
- The server Role. The client must verify that the server TLS certificate extended key usages attributes match the advertise server Role. If it does not, the client must discard and close the TLS connection to the server.
The CONNECTED frame payload is the same as the CONFIGURE one and contains cluster configuration data.
+--------------------------------------------------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Role | Server UUID | Client UUID | Payload | YAML formatted |
| | | (0x1) | (0x0) | (bitmask of server roles) | | | Length | payload |
+--------------------------------------------------------------------------------------------------------------------+
READY
SSNTP compute node Agents send READY status frames to let the Scheduler know that:
- Their CN capacity has changed. The new capacity is described in the [READY YAML payload] (https://github.com/01org/ciao/blob/master/payloads/ready.go). This is the main piece of information the Scheduler uses to make its instances scheduling decisions.
- They are ready to take further commands, and in particular to start new workloads on the CN they manage. It is important to note that a Scheduler should not send a new START commands to a given Agent until it receives the next READY status frame from it. Some Scheduler implementations may implement opportunistic heuristics and send several START commands after receiving a STATUS frame, by forecasting CN capacities based on the START command payloads they previously sent. This allow them to reach shorter average instances startup times at the risk of hitting higher than expected cloud overcommit ratios.
The READY status payload is almost a subset of the STATS command one as it does describe the CN capacity status without providing any details about the currently running instances. There are several differences between READY and STATS:
- READY frames are asynchronous while STATS frames are periodic. Agent implementations will typically send READY status to the Scheduler after successfully starting a new instance on the CN while they send STATS command frames to the Controller every so often.
- READY frames are typically much smaller than STATS ones as their payload does not contain any instance related status. On CNs running thousands of instances, STATS payloads can be significantly larger than READY ones.
- Sending a STATS command does explicitly provide information about the Agent's readiness to process any further instance related commands. For example, an Agent may be busy starting an instance while at the same time sending a STATS command.
As a consequence SSNTP compute node Agents must use the READY and FULL status frames, rather than STATs frames, to notify the scheduler about their availability and capacity.
The READY status frame payload is mandatory:
+----------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted compute |
| | | (0x1) | (0x1) | | node new capacity |
+----------------------------------------------------------------------------+
FULL
Whenever the CN they manage runs out of capacity, SSNTP Agents must send a FULL status frame to the Scheduler.
The Scheduler must not send any START command to an Agent whose latest status is reported to be FULL. FULL Agents who receive such commands should reply with an SSNTP error frame to the Scheduler. The error code should be StartFailure (0x2)
The Scheduler may decide to resume sending START commands to a FULL Agent after receiving the next READY status frame from it. Any SSNTP command except for the START and CONNECT ones can be sent to a FULL Agent.
The FULL status frame is payloadless:
+---------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length |
| | | (0x1) | (0x2) | (0x0) |
+---------------------------------------------------+
OFFLINE
OFFLINE is a compute node status frame sent by SSNTP Agents to let the Scheduler know that although they're running and still connected to the SSNTP network, they are not ready to process any kind of SSNTP commands. Agents should reply with a SSNTP error frame to any received frame while they are OFFLINE.
The Scheduler should forward OFFLINE status frames to the Controller for it to immediately know about a CN not being able to process any further commands.
SSNTP Agents in OFFLINE mode should continue sending periodic STATS frame.
The OFFLINE status frame is payloadless:
+---------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length |
| | | (0x1) | (0x3) | (0x0) |
+---------------------------------------------------+
MAINTENANCE
TBD
+-----------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted payload |
| | | (0x1) | (0x4) | | |
+-----------------------------------------------------------------------------+
SSNTP EVENT frames
Unlike STATUS frames, EVENT frames are not necessarily related to a particular compute node's status. They allow SSNTP entities to notify each other about important events.
There are 6 different SSNTP EVENT frames: TenantAdded, TenantRemoved, InstanceDeleted, ConcentratorInstanceAdded, PublicIPAssigned and TraceReport.
TenantAdded
TenantAdded is used by CN Agents to notify Networking Agents that the first workload for a given tenant has just started. Networking agents need to be notified about this so that they can forward the notification to the right CNCI (Compute Node Concentrator Instance), i.e. the CNCI running the tenant workload.
A [TenantAdded event payload] (https://github.com/01org/ciao/blob/master/payloads/tenantadded.go) is a YAML formatted one containing the tenant, the agent and the concentrator instance (CNCI) UUID, the tenant subnet, the agent and the CNCI IPs, the subnet key and the CNCI MAC.
The Scheduler receives TenantAdded events from the CN Agent and must forward them to the appropriate CNCI Agent.
+---------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted tenant |
| | | (0x3) | (0x0) | | information |
+---------------------------------------------------------------------------+
TenantRemoved
TenantRemoved is used by CN Agents to notify Networking Agents that the last workload for a given tenant has just terminated. Networking agents need to be notified about it so that they can forward it to the right CNCI (Compute Node Concentrator Instance), i.e. the CNCI running the tenant workload.
A [TenantRemoved event payload] (https://github.com/01org/ciao/blob/master/payloads/tenantadded.go) is a YAML formatted one containing the tenant, the agent and the concentrator instance (CNCI) UUID, the tenant subnet, the agent and the CNCI IPs, and the subnet key.
The Scheduler receives TenantRemoved events from the CN Agent and must forward them to the appropriate CNCI Agent.
+---------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted tenant |
| | | (0x3) | (0x1) | | information |
+---------------------------------------------------------------------------+
InstanceDeleted
InstanceDeleted is sent by workload agents to notify the scheduler and the Controller that a previously running instance has been deleted. While the scheduler and the Controller could infer that piece of information from the next STATS command (The deleted instance would no longer be there) it is safer, simpler and less error prone to explicitly send this event.
A [InstanceDeleted event payload] (https://github.com/01org/ciao/blob/master/payloads/instancedeleted.go) is a YAML formatted one containing the deleted instance UUID.
The Scheduler receives InstanceDeleted events from the payload agents and must forward them to the Controller.
+---------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted tenant |
| | | (0x3) | (0x2) | | information |
+---------------------------------------------------------------------------+
ConcentratorInstanceAdded
Networking node agents send this event to the Scheduler to notify the SSNTP network that a networking concentrator instance (CNCI) is now running on this node. A CNCI handles the GRE tunnel concentrator for a given tenant. Each instance started by this tenant will have a GRE tunnel established between it and the CNCI allowing all instances for a given tenant to be on the same private network.
The Scheduler must forward that event to all Controllers. The Controller needs to know about it as it will fetch the CNCI IP and the tenant UUID from this event's payload and pass that through the START payload when scheduling a new instance for this tenant. A tenant instances can not be scheduled until Controller gets a ConcentratorInstanceAdded event as instances will be isolated as long as the CNCI for this tenant is not running.
A [ConcentratorInstanceAdded event payload] (https://github.com/01org/ciao/blob/master/payloads/concentratorinstanceadded.go) is a YAML formatted one containing the CNCI IP and the tenant UUID on behalf of which the CNCI runs.
+---------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted CNCI |
| | | (0x3) | (0x3) | | information |
+---------------------------------------------------------------------------+
PublicIPAssigned
Networking concentrator instances (CNCI) send PublicIPAssigned to the Scheduler when they successfully assigned a public IP to a given instance. The public IP can either come from a Controller pre-allocated pool, or from a control network DHCP server.
The Scheduler must forward those events to the Controller.
The [PublicIPAssigned event payload] (https://github.com/01org/ciao/blob/master/payloads/concentratorinstanceadded.go) contains the newly assigned public IP, the instance private IP, the instance UUID and the concentrator UUID.
+--------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted |
| | | (0x3) | (0x4) | | payload |
+--------------------------------------------------------------------+
TraceReport
Any SSNTP entity can decide to send a TraceReport event in order to let the CIAO controller know about any kind of frame traces.
It is then up to the Controller to interpret and store those traces.
The [TraveReport event payload] (https://github.com/01org/ciao/blob/master/payloads/tracereport.go) contains a set of frame traces.
+----------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted payload |
| | | (0x3) | (0x5) | | |
+----------------------------------------------------------------------------+
NodeConnected
NodeConnected events are sent by the Scheduler to notify e.g. the Controllers about a new compute or networking node being connected. The [NodeConnected event payload] (https://github.com/01org/ciao/blob/master/payloads/nodeconnected.go) contains the connected node UUID and the node type (compute or networking)
+----------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted payload |
| | | (0x3) | (0x6) | | |
+----------------------------------------------------------------------------+
NodeDisconnected
NodeDisconnected events are sent by the Scheduler to notify e.g. the Controllers about a compute or networking node disconnection. The [NodeDisconnected event payload] (https://github.com/01org/ciao/blob/master/payloads/nodeconnected.go) contains the disconnected node UUID and the node type (compute or networking)
+----------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted payload |
| | | (0x3) | (0x7) | | |
+----------------------------------------------------------------------------+
SSNTP ERROR frames
SSNTP being a fully asynchronous protocol, SSNTP entities are not expecting specific frames to be acknowledged or rejected. Instead they must be ready to receive asynchronous error frames notifying them about an application level error, not a frame level one.
There are 7 different SSNTP ERROR frames:
InvalidFrameType
When a SSNTP entity receives a frame whose type it does not support, it should send an InvalidFrameType error back to the sender.
The [InvalidFrameType error payload] (https://github.com/01org/ciao/blob/master/payloads/invalidframetype.go) only contains the SSNTP frame type that the receiver could not process:
+-----------------------------------------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | Source UUID | Destination UUID | YAML formatted frame |
| | | (0x4) | (0x0) | | | | type information |
+-----------------------------------------------------------------------------------------------------------+
StartFailure
The StartFailure SSNTP error frames must be sent when an instance could not be started. For example:
-
The Scheduler receives a START command from the Controller but all its CN Agents are busy or full. In that case the Scheduler must send a StartFailure error frame back to the Controller
-
An Agent receives a START command from the Scheduler but it cannot start the instance. This could happen for many reasons:
- Malformed START YAML payload
- Compute node is full In that case the Agent must send a StartFailure error frame back to the Scheduler and the Scheduler must forward it to the Controller.
The [StartFailure YAML payload] (https://github.com/01org/ciao/blob/master/payloads/startfailure.go) contains the instance UUID that failed to be started together with an additional error string.
+--------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted frame |
| | | (0x4) | (0x1) | | error information |
+--------------------------------------------------------------------------+
StopFailure
When the Controller client needs to stop a running instance on a given CN, it sends a STOP SSNTP command to the Scheduler. The STOP command payload contains the instance UUID and the CN Agent UUID where that instance is running.
-
If the Scheduler can no longer find the CN Agent, it must send a StopFailure error frame back to the Controller.
-
If the CN Agent cannot stop the instance because, for example, it is no longer running, it must send a StopFailure error frame back to the Scheduler and the Scheduler must forward it to the Controller.
The [StopFailure YAML payload] (https://github.com/01org/ciao/blob/master/payloads/startfailure.go) contains the instance UUID that failed to be stopped together with an additional error string.
+--------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted frame |
| | | (0x4) | (0x2) | | error information |
+--------------------------------------------------------------------------+
ConnectionFailure
Both SSNTP clients and servers can send a ConnectionFailure error frame when the initial connection could not be completed but should be retried. ConnectionFailure is not a fatal error but represents a transient connection error.
The ConnectionFailure error frame is payloadless:
+---------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length |
| | | (0x4) | (0x3) | (0x0) |
+---------------------------------------------------+
DeleteFailure
When the Controller client wants to delete a stopped instance on a given CN, it sends a DELETE SSNTP command to the Scheduler.
-
If the Scheduler can no longer find the CN Agent, it must send a DeleteFailure error frame back to the Controller.
-
If the CN Agent cannot delete the instance because, for example, it is no longer present, it must send a DeleteFailure error frame back to the Scheduler and the Scheduler must forward it to the Controller.
The [DeleteFailure YAML payload] (https://github.com/01org/ciao/blob/master/payloads/deletefailure.go) contains the instance UUID that failed to be stopped together with an additional error string.
+--------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted frame |
| | | (0x4) | (0x4) | | error information |
+--------------------------------------------------------------------------+
RestartFailure
When the Controller client wants to restart a stopped instance on a given CN, it sends a RESTART SSNTP command to the Scheduler.
-
If the Scheduler can no longer find the CN Agent, it must send a RestartFailure error frame back to the Controller.
-
If the CN Agent cannot restart the instance because, for example, it is no longer present, it must send a RestartFailure error frame back to the Scheduler and the Scheduler must forward it to the Controller.
The [RestartFailure YAML payload] (https://github.com/01org/ciao/blob/master/payloads/startfailure.go) contains the instance UUID that failed to be stopped together with an additional error string.
+--------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted frame |
| | | (0x4) | (0x5) | | error information |
+--------------------------------------------------------------------------+
ConnectionAborted
Both SSNTP clients and servers can send a ConnectionAborted error frame when either the CONNECT command frame or the CONNECTED status frame contain an advertised role that does not match the peer's certificate extended key usage attribute.
Sending ConnectionAborted means that for security reasons the connection will not be retried.
The ConnectionAborted error frame is payloadless:
+---------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length |
| | | (0x4) | (0x6) | (0x0) |
+---------------------------------------------------+
InvalidConfiguration
The InvalidConfiguration error is either sent by the Scheduler to report an invalid CONFIGURE payload back to the sender, or by the clients to which a CONFIGURE command has been forwarded to and that leads to configuration errors on their side. When the scheduler receives such error back from any client it should revert back to the previous valid configuration.
The InvalidConfiguration error frame contain the invalid configuration data payload.
+------------------------------------------------------------------------+
| Major | Minor | Type | Operand | Payload Length | YAML formatted |
| | | (0x4) | (0x7) | | configuration data |
+------------------------------------------------------------------------+
Documentation
¶
Index ¶
- Constants
- Variables
- type Client
- func (client *Client) Close()
- func (client *Client) ClusterConfiguration() (payload []byte)
- func (client *Client) Dial(config *Config, ntf ClientNotifier) error
- func (client *Client) SendCommand(cmd Command, payload []byte) (int, error)
- func (client *Client) SendError(error Error, payload []byte) (int, error)
- func (client *Client) SendEvent(event Event, payload []byte) (int, error)
- func (client *Client) SendStatus(status Status, payload []byte) (int, error)
- func (client *Client) SendTracedCommand(cmd Command, payload []byte, trace *TraceConfig) (int, error)
- func (client *Client) SendTracedError(error Error, payload []byte, trace *TraceConfig) (int, error)
- func (client *Client) SendTracedEvent(event Event, payload []byte, trace *TraceConfig) (int, error)
- func (client *Client) SendTracedStatus(status Status, payload []byte, trace *TraceConfig) (int, error)
- func (client *Client) UUID() string
- type ClientNotifier
- type Command
- type CommandForwarder
- type Config
- type ConnectFrame
- type ConnectedFrame
- type Error
- type ErrorForwarder
- type Event
- type EventForwarder
- type ForwardDecision
- type ForwardDestination
- type Frame
- type FrameForwardRule
- type FrameTrace
- type Logger
- type Node
- type Role
- type Server
- func (server *Server) SendCommand(uuid string, cmd Command, payload []byte) (int, error)
- func (server *Server) SendError(uuid string, error Error, payload []byte) (int, error)
- func (server *Server) SendEvent(uuid string, event Event, payload []byte) (int, error)
- func (server *Server) SendStatus(uuid string, status Status, payload []byte) (int, error)
- func (server *Server) SendTracedCommand(uuid string, cmd Command, payload []byte, trace *TraceConfig) (int, error)
- func (server *Server) SendTracedError(uuid string, error Error, payload []byte, trace *TraceConfig) (int, error)
- func (server *Server) SendTracedEvent(uuid string, event Event, payload []byte, trace *TraceConfig) (int, error)
- func (server *Server) SendTracedStatus(uuid string, status Status, payload []byte, trace *TraceConfig) (int, error)
- func (server *Server) Serve(config *Config, ntf ServerNotifier) error
- func (server *Server) Stop()
- func (server *Server) UUID() string
- type ServerNotifier
- type Status
- type StatusForwarder
- type TraceConfig
- type Type
Examples ¶
Constants ¶
const ( UNKNOWN Role = 0x0 SERVER = 0x1 // The Command and Status Reporter. This is a client role. Controller = 0x2 // The cloud compute node agent. This is a client role. AGENT = 0x4 // The workload scheduler. This is a server role. SCHEDULER = 0x8 // The networking compute node agent. This is a client role. NETAGENT = 0x10 // The networking compute node concentrator instance (CNCI) agent. This is a client role. CNCIAGENT = 0x20 )
SSNTP clients and servers can have one or several roles and are expected to declare their roles during the SSNTP connection procedure.
Variables ¶
var ( // RoleAgentOID is the SSNTP Agent Role Object ID. RoleAgentOID = asn1.ObjectIdentifier{1, 3, 6, 1, 4, 1, 343, 8, 1} // RoleSchedulerOID is the SSNTP Scheduler Role Object ID. RoleSchedulerOID = asn1.ObjectIdentifier{1, 3, 6, 1, 4, 1, 343, 8, 2} // RoleControllerOID is the SSNTP Controller Role Object ID. RoleControllerOID = asn1.ObjectIdentifier{1, 3, 6, 1, 4, 1, 343, 8, 3} // RoleNetAgentOID is the SSNTP Networking Agent Role Object ID. RoleNetAgentOID = asn1.ObjectIdentifier{1, 3, 6, 1, 4, 1, 343, 8, 4} // RoleAgentOID is the SSNTP Server Role Object ID. RoleServerOID = asn1.ObjectIdentifier{1, 3, 6, 1, 4, 1, 343, 8, 5} // RoleCNCIAgentOID is the SSNTP Compute Node Concentrator Instance Agent Role Object ID. RoleCNCIAgentOID = asn1.ObjectIdentifier{1, 3, 6, 1, 4, 1, 343, 8, 6} )
We use SSL extended key usage attributes for specifying and verifying SSNTP client and server claimed roles. For example if a client claims to be a Controller, then its client certificate extended key usage attribute should contain the right OID for that role.
var Log glogLog
Log is a glog based SSNTP Logger implementation. Error message will be logged unconditionally. Warnings are logged if glog's V >= 1. Info messages are logged if glog's V >= 2.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is the SSNTP client structure. This is an SSNTP client handle to connect to and disconnect from an SSNTP server, and send SSNTP frames to it. It is an entirely opaque structure, only accessible through its public methods.
func (*Client) ClusterConfiguration ¶
ClusterConfiguration returns the latest cluster configuration payload a client received. Client should use that payload to configure themselves based on the information provided to them by the Scheduler or the Controller. Cluster configuration payloads can come from either a CONNECTED status frame or a CONFIGURE command one.
func (*Client) Dial ¶
func (client *Client) Dial(config *Config, ntf ClientNotifier) error
Dial attempts to connect to a SSNTP server, as specified by the config argument. Dial will try and retry to connect to this server and will wait for it to show up if it's temporarily unavailable. A client can be closed while it's still trying to connect to the SSNTP server, so that one can properly kill a client if e.g. no server will ever come alive. Once connected a separate routine will listen for server commands, statuses or errors and report them back through the SSNTP client notifier interface.
Example ¶
package main
import (
"fmt"
"time"
)
type ssntpEchoClient struct {
ssntp Client
name string
}
func (client *ssntpEchoClient) ConnectNotify() {
fmt.Printf("%s Connected\n", client.name)
}
func (client *ssntpEchoClient) DisconnectNotify() {
fmt.Printf("%s disconnected\n", client.name)
}
func (client *ssntpEchoClient) StatusNotify(status Status, frame *Frame) {
n, err := client.ssntp.SendStatus(status, frame.Payload)
if err != nil {
fmt.Printf("%s\n", err)
}
fmt.Printf("Echoed %d status bytes\n", n)
}
func (client *ssntpEchoClient) CommandNotify(command Command, frame *Frame) {
n, err := client.ssntp.SendCommand(command, frame.Payload)
if err != nil {
fmt.Printf("%s\n", err)
}
fmt.Printf("Echoed %d command bytes\n", n)
}
func (client *ssntpEchoClient) EventNotify(event Event, frame *Frame) {
n, err := client.ssntp.SendEvent(event, frame.Payload)
if err != nil {
fmt.Printf("%s\n", err)
}
fmt.Printf("Echoed %d event bytes\n", n)
}
func (client *ssntpEchoClient) ErrorNotify(error Error, frame *Frame) {
fmt.Printf("ERROR %s\n", error)
}
func main() {
var config Config
client := &ssntpEchoClient{
name: "CIAO Agent",
}
config.URI = "myCIAOserver.local"
config.CAcert = "CIAOCA.crt"
config.Cert = "agent.pem"
config.Role = uint32(AGENT)
if client.ssntp.Dial(&config, client) != nil {
fmt.Printf("Could not connect to an SSNTP server\n")
return
}
// Loop and wait for notifications
for {
time.Sleep(time.Duration(10) * time.Second)
}
}
Output:
func (*Client) SendCommand ¶
SendCommand sends a specific command and its payload to the SSNTP server.
func (*Client) SendError ¶
SendError sends an error back to the SSNTP server. This is just for notification purposes, to let e.g. the server know that it sent an unexpected frame.
func (*Client) SendStatus ¶
SendStatus sends a specific status and its payload to the SSNTP server.
func (*Client) SendTracedCommand ¶
func (client *Client) SendTracedCommand(cmd Command, payload []byte, trace *TraceConfig) (int, error)
SendTracedCommand sends a specific command and its payload to the SSNTP server. The SSNTP command frame will be traced according to the trace argument.
func (*Client) SendTracedError ¶
SendTracedError sends an error back to the SSNTP server. This is just for notification purposes, to let e.g. the server know that it sent an unexpected frame. The SSNTP error frame will be traced according to the trace argument.
func (*Client) SendTracedEvent ¶
SendTracedEvent sends a specific status and its payload to the SSNTP server. The SSNTP event frame will be traced according to the trace argument.
func (*Client) SendTracedStatus ¶
func (client *Client) SendTracedStatus(status Status, payload []byte, trace *TraceConfig) (int, error)
SendTracedStatus sends a specific status and its payload to the SSNTP server. The SSNTP status frame will be traced according to the trace argument.
type ClientNotifier ¶
type ClientNotifier interface {
// ConnectNotify notifies of a successful connection to an SSNTP server.
// This notification is mostly useful for clients to know when they're
// being re-connected to the SSNTP server.
ConnectNotify()
// DisconnectNotify notifies of a SSNTP server disconnection.
// SSNTP Client implementations are not supposed to explicitly
// reconnect, the SSNTP protocol will handle the reconnection.
DisconnectNotify()
// StatusNotify notifies of a pending status frame from the SSNTP server.
StatusNotify(status Status, frame *Frame)
// CommandNotify notifies of a pending command frame from the SSNTP server.
CommandNotify(command Command, frame *Frame)
// EventNotify notifies of a pending event frame from the SSNTP server.
EventNotify(event Event, frame *Frame)
// ErrorNotify notifies of a pending error frame from the SSNTP server.
// Error frames are always related to the last sent frame.
ErrorNotify(error Error, frame *Frame)
}
ClientNotifier is the SSNTP client notification interface. Any SSNTP client must implement this interface. IMPORTANT: All ClientNotifier implementations must be thread safe, i.e. they will very likely be called by multiple go routines.
type Command ¶
type Command uint8
Command is the SSNTP Command operand. It can be CONNECT, START, STOP, STATS, EVACUATE, DELETE, RESTART, AssignPublicIP, ReleasePublicIP or CONFIGURE.
const ( // CONNECT is the first frame sent by an SSNTP client to establish the SSNTP // connection. A server will ignore any clients until it sends its first CONNECT // frame: // SSNTP CONNECT Command frame // // +-------------------------------------------------------------+ // | Major | Minor | Type | Operand | Role | // | | | (0x0) | (0x0) | (bitmask of client roles) | // +-------------------------------------------------------------+ CONNECT Command = iota // START is a command that should reach CIAO agents for scheduling a new // on the compute node (CN) they manage. It should typically come from the Controller // entity directly or via the main server: // SSNTP START Command frame // // +-----------------------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted workload description | // | | | (0x0) | (0x1) | | | // +-----------------------------------------------------------------------------------------+ START // STOP is used to ask a CIAO agent to stop a running workload. The workload // is identified by its UUID, as part of the YAML formatted payload: // SSNTP STOP Command frame // // +----------------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted workload UUID | // | | | (0x0) | (0x2) | | | // +----------------------------------------------------------------------------------+ STOP // STATS is a command sent by CIAO agents to update the SSNTP network // about their compute node statistics. Agents can send that command to either // the main server or to the Controllers directly. In the former case the server will // be responsible for forwarding it to the known Controllers. // The conpute node statistics form the YAML formatted payload for this command: // SSNTP STATS Command frame // // +----------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted compute | // | | | (0x0) | (0x3) | | node statistics | // +----------------------------------------------------------------------------+ STATS // EVACUATE is intended to ask a specific CIAO agent to evacuate its compute // node, i.e. stop and migrate all of the current workloads he's monitoring on // this node. The payload for this command is a YAML formatted description of the // next state to reach after evacuation is done. It could be 'shutdown' for shutting // the node down, 'update' for having it run a software update, 'reboot' for rebooting // the node or 'maintenance' for putting the node in maintenance mode: // +---------------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted compute | // | | | (0x0) | (0x4) | | node next state description | // +---------------------------------------------------------------------------------+ EVACUATE // DELETE is a command sent to a CIAO CN Agent in order to completely delete a // running instance. This is only relevant for persistent workloads after they were // STOPPED. Non persistent workload get deleted when they are STOPPED. // It is up to the CN Agent implementation to decide what exactly needs to be deleted // on the CN but a deleted instance will no longer be able to boot. // The DELETE command payload uses the same YAML schema as the STOP command one, i.e. // an instance UUID and an agent UUID. // SSNTP DELETE Command frame // +------------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted payload | // | | | (0x0) | (0x5) | | instance and agent UUIDs | // +------------------------------------------------------------------------------+ DELETE // RESTART is a command sent to CIAO CN Agents for restarting an instance that was // previously STOPped. This command is only relevant for persistent workloads since // non persistent ones are implicitly deleted when STOPped and thus can not be // RESTARTed. // The RESTART command payload uses the same YAML schema as the STOP command one, i.e. // an instance UUID and an agent UUID. // SSNTP DELETE Command frame // +------------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted payload | // | | | (0x0) | (0x6) | | instance and agent UUIDs | // +------------------------------------------------------------------------------+ RESTART // AssignPublicIP is a command sent by the Controller to assign // a publicly routable IP to a given instance. It is sent // to the Scheduler and must be forwarded to the right CNCI. // // The public IP is fetched from a pre-allocated pool // managed by the Controller. // // The AssignPublicIP YAML payload schema is made of the // CNCI and a tenant UUIDs, the allocated public IP, the // instance private IP and MAC. // // SSNTP AssignPublicIP Command frame // +----------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted payload | // | | | (0x0) | (0x7) | | | // +----------------------------------------------------------------------------+ AssignPublicIP // ReleasePublicIP is a command sent by the Controller to release // a publicly routable IP from a given instance. It is sent // to the Scheduler and must be forwarded to the right CNCI. // // The released public IP is added back to the Controller managed // IP pool. // // The ReleasePublicIP YAML payload schema is made of the // CNCI and a tenant UUIDs, the released public IP, the // instance private IP and MAC. // // SSNTP ReleasePublicIP Command frame // +--------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted | // | | | (0x0) | (0x8) | | payload | // +--------------------------------------------------------------------+ ReleasePublicIP // CONFIGURE commands are sent to request any SSNTP entity to // configure itself according to the CONFIGURE command payload. // Controller or any SSNTP client handling user interfaces defining any // cloud setting (image service, networking configuration, identity // management...) must send this command for any configuration // change and for broadcasting the initial cloud configuration to // all CN and NN agents. // // The CONFIGURE command payload always include the full cloud // configuration and not only changes compared to the last CONFIGURE // command sent. // // SSNTP CONFIGURE Command frame // +-----------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted payload | // | | | (0x0) | (0x9) | | | // +-----------------------------------------------------------------------------+ CONFIGURE )
type CommandForwarder ¶
type CommandForwarder interface {
CommandForward(uuid string, command Command, frame *Frame) ForwardDestination
}
CommandForwarder is the SSNTP Command forwarding interface. The uuid argument is the sender's UUID.
type Config ¶
type Config struct {
// UUID is the client or server UUID string. If set to "",
// the SSNTP package will generate a random one.
UUID string
// URI semantic differs between servers and clients.
// For clients it represents the the SSNTP server URI
// they want to connect to.
// For servers it represents the URI they will be
// listening on.
// When set to "" SSNTP servers will listen on all interfaces
// and IPs on the running host.
URI string
// Role is a bitmask of SSNTP roles the client or server intends
// to run.
Role uint32
// CACert is the Certification Authority certificate path
// to use when verifiying the peer identity.
// If set to "", /etc/pki/ciao/ciao_ca_cert.crt will be used.
CAcert string
// Cert is the client or server x509 signed certificate path.
// If set to "", /etc/pki/ciao/client.pem and /etc/pki/ciao/ciao.pem
// will be used for SSNTP clients and server, respectively.
Cert string
// Transport is the underlying transport protocol. Only "tcp" and "unix"
// transports are supported. The default is "tcp".
Transport string
// ForwardRules is optional and contains a list of frame forwarding rules.
ForwardRules []FrameForwardRule
// Log is the SSNTP logging interface.
// If not set, only error messages will be logged.
// The SSNTP Log implementation provides a default logger.
Log Logger
// When RoleVerification is true the peer declared role will be
// verified by checking that the received certificate extended
// key usage attributes contains the right OID.
RoleVerification bool
// TCP port to connect (Client) or to listen to (Server).
// This is optional, the default SSNTP port is 8888.
Port uint32
// Trace configures the desired level of SSNTP frame tracing.
Trace *TraceConfig
}
A Config structure is used to configure a SSNTP client or server. It is mandatory to provide an SSNTP configuration when starting an SSNTP server or when connecting to one as a client.
type ConnectFrame ¶
type ConnectFrame struct {
Major uint8
Minor uint8
Type Type
Operand uint8
Role uint32
Source []byte
Destination []byte
}
ConnectFrame is the SSNTP connection frame structure.
func (ConnectFrame) String ¶
func (f ConnectFrame) String() string
type ConnectedFrame ¶
type ConnectedFrame struct {
Major uint8
Minor uint8
Type Type
Operand uint8
Role uint32
Source []byte
Destination []byte
PayloadLength uint32
Payload []byte
}
ConnectedFrame is the SSNTP connected frame structure.
func (ConnectedFrame) String ¶
func (f ConnectedFrame) String() string
type Error ¶
type Error uint8
Error is the SSNTP Error operand. It can be InvalidFrameType Error, StartFailure, StopFailure, ConnectionFailure, RestartFailure, DeleteFailure, ConnectionAborted or InvalidConfiguration.
const ( // InvalidFrameType is sent when receiving an unsupported frame type. InvalidFrameType Error = iota // StartFailure is sent by launcher agents to report a workload start failure. StartFailure // StopFailure is sent by launcher agents to report a workload pause failure. StopFailure // ConnectionFailure is sent to report an SSNTP connection failure. // It can be sent by servers and clients. ConnectionFailure // RestartFailure is sent by launcher agents to report a workload re-start failure. RestartFailure // DeleteFailure is sent by launcher agents to report a workload deletion failure. DeleteFailure // ConnectionAborted is sent to report an SSNTP connection abortion. // This is used for example when receiving bad certificates. ConnectionAborted // InvalidConfiguration is either sent by the Scheduler to report an invalid // CONFIGURE payload back to the sender, or by the clients to which a CONFIGURE // command has been forwarded to and that leads to configuration errors on their // side. // When the scheduler receives such error back from any client it should revert // back to the previous valid configuration. InvalidConfiguration )
type ErrorForwarder ¶
type ErrorForwarder interface {
ErrorForward(uuid string, error Error, frame *Frame) ForwardDestination
}
ErrorForwarder is the SSNTP Error forwarding interface. The uuid argument is the sender's UUID.
type Event ¶
type Event uint8
Event is the SSNTP Event operand. It can be TenantAdded, TenantRemoval, InstanceDeleted, ConcentratorInstanceAdded, PublicIPAssigned, TraceReport, NodeConnected or NodeDisconnected
const ( // TenantAdded is used by workload agents to notify networking agents that the first // workload for a given tenant has just started. Networking agents need to know about that // so that they can forward it to the right CNCI (Compute Node Concentrator Instance), i.e. // the CNCI running the tenant workload. // SSNTP TenantAdded Event frame // // +---------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted tenant | // | | | (0x3) | (0x0) | | information | // +---------------------------------------------------------------------------+ TenantAdded Event = iota // TenantRemoved is used by workload agents to notify networking agents that the last // workload for a given tenant has just terminated. Networking agents need to know about that // so that they can forward it to the right CNCI (Compute Node Concentrator Instance), i.e. // the CNCI running the tenant workload. // SSNTP TenantRemoved Event frame // // +--------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted tenant | // | | | (0x3) | (0x1) | | information | // +---------------------------------------------------------------------------+ TenantRemoved // InstanceDeleted is sent by workload agents to notify the scheduler and the Controller that a // previously running instance has been deleted. While the scheduler and the Controller could infer // that information from the next STATS command (The deleted instance would no longer be there) // it is safer, simpler and less error prone to explicitly send this event. // // SSNTP InstanceDeleted Event frame // // +---------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted | // | | | (0x3) | (0x2) | | instance information | // +---------------------------------------------------------------------------+ InstanceDeleted // ConcentratorInstanceAdded events are sent by networking nodes // agents to the Scheduler in order to notify the SSNTP network // that a networking concentrator instance (CNCI) is now running // on this node. // A CNCI handles the GRE tunnel concentrator for a given // tenant. Each instance started by this tenant will have a // GRE tunnel established between it and the CNCI allowing all // instances for a given tenant to be on the same private // network. // // The Scheduler must forward that event to all Controllers. The Controller // needs to know about it as it will fetch the CNCI IP and the // tenant UUID from this event's payload and pass that through // the START payload when scheduling a new instance for this // tenant. A tenant instances can not be scheduled until Controller gets // a ConcentratorInstanceAdded event as instances will be // isolated as long as the CNCI for this tenant is not running. // // SSNTP ConcentratorInstanceAdded Event frame // // +--------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted | // | | | (0x3) | (0x3) | | CNCI information | // +--------------------------------------------------------------------------+ ConcentratorInstanceAdded // PublicIPAssigned events are sent by Networking concentrator // instances (CNCI) to the Scheduler when they successfully // assigned a public IP to a given instance. // The public IP can either come from a Controller pre-allocated pool, // or from a control network DHCP server. // // The Scheduler must forward those events to the Controller. // // The PublicIPAssigned event payload contains the newly assigned // public IP, the instance private IP and the instance UUID. // // SSNTP PublicIPAssigned Event frame // // +----------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted payload | // | | | (0x3) | (0x4) | | | // +----------------------------------------------------------------------------+ PublicIPAssigned // TraceReport events carry a tracing report payload from one // of the SSNTP clients. // // SSNTP TraceReport Event frame // // +----------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted payload | // | | | (0x3) | (0x5) | | | // +----------------------------------------------------------------------------+ TraceReport // NodeConnected events are sent by the Scheduler to notify e.g. the Controllers about // a new compute or networking node being connected. // The NodeConnected event payload contains the connected node UUID and the node type // (compute or networking) // // SSNTP NodeConnected Event frame // // +----------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted payload | // | | | (0x3) | (0x6) | | | // +----------------------------------------------------------------------------+ NodeConnected // NodeDisconnected events are sent by the Scheduler to notify e.g. the Controllers about // a new compute or networking node disconnection. // The NodeDisconnected event payload contains the discconnected node UUID and the node // type (compute or networking) // // SSNTP NodeDisconnected Event frame // // +----------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted payload | // | | | (0x3) | (0x7) | | | // +----------------------------------------------------------------------------+ NodeDisconnected )
type EventForwarder ¶
type EventForwarder interface {
EventForward(uuid string, event Event, frame *Frame) ForwardDestination
}
EventForwarder is the SSNTP Event forwarding interface. The uuid argument is the sender's UUID.
type ForwardDecision ¶
type ForwardDecision uint8
ForwardDecision tells SSNTP how it should forward a frame. Callers set that value as part of the ForwardDestination structure.
const ( // Forward the frame. The recipients are defined by the ForwardDecision // UUIDs field. Forward ForwardDecision = iota // Discard the frame. The frame will be discarded by SSNTP. Discard // Queue the frame. SSNTP will queue the frame and the caller will have to call // into the SSNTP Server queueing API to fetch it back. Queue )
type ForwardDestination ¶
type ForwardDestination struct {
// contains filtered or unexported fields
}
ForwardDestination is returned by the forwading interfaces and allows the interface implementer to let SSNTP know what to do next with a received frame. The interface implementer needs to specify if the frame should be forwarded, discarded or queued (Decision). If the implementer decision is to forward the frame, it should also provide a list of recipients to forward it to (UUIDs)
func (*ForwardDestination) AddRecipient ¶
func (d *ForwardDestination) AddRecipient(uuid string)
AddRecipient adds a recipient to a ForwardDestination structure. AddRecipient implicitly sets the forwarding decision to Forward since adding a recipient means the frame must be forwarded.
func (*ForwardDestination) SetDecision ¶
func (d *ForwardDestination) SetDecision(decision ForwardDecision)
SetDecision is a helper for setting the ForwardDestination Decision field.
type Frame ¶
type Frame struct {
Major uint8
Minor uint8
Type Type
Operand uint8
PayloadLength uint32
Trace *FrameTrace
Payload []byte
}
Frame represents an SSNTP frame structure.
func (Frame) DumpTrace ¶
func (f Frame) DumpTrace() (*payloads.FrameTrace, error)
DumpTrace builds SSNTP frame tracing data into a FrameTrace payload. Callers typically marshall this structure into a TraceReport YAML payload.
func (Frame) Duration ¶
Duration returns the time spent between the first frame transmission and its last reception.
func (*Frame) SetEndStamp ¶
func (f *Frame) SetEndStamp()
SetEndStamp adds the final timestamp to an SSNTP frame. This is called by the SSNTP node that believes it's the last frame receiver. It provides information to build the complete duration of the operation related to an SSNTP frame.
type FrameForwardRule ¶
type FrameForwardRule struct {
// Operand is the SSNTP frame operand to which this rule applies.
Operand interface{}
// A frame which operand is Operand will be forwarded to all
// SSNTP clients playing the Dest SSNTP role.
// This field is ignored if a forwarding interface is provided.
Dest Role
// The SSNTP Command forwarding interface implementation for this SSNTP frame.
CommandForward CommandForwarder
// The SSNTP Status forwarding interface implementation for this SSNTP frame.
StatusForward StatusForwarder
// The SSNTP Error forwarding interface implementation for this SSNTP frame.
ErrorForward ErrorForwarder
// The SSNTP Event forwarding interface implementation for this SSNTP frame.
EventForward EventForwarder
}
FrameForwardRule defines a forwarding rule for a SSNTP frame. The rule creator can either choose to forward this frame to all clients playing a specified SSNTP role (Dest), or can return a forwarding decision back to SSNTP depending on the frame payload (*Forwarder). If a frame forwarder interface implementation is provided, the Dest field will be ignored.
type FrameTrace ¶
type FrameTrace struct {
Label []byte
StartTimestamp time.Time
EndTimestamp time.Time
PathLength uint8
Path []Node
}
FrameTrace gathers all SSNTP frame tracing information, including frame labelling, per Node timestamping and both start and end timestamps as provided by the frame API callers.
type Logger ¶
type Logger interface {
Errorf(format string, args ...interface{})
Warningf(format string, args ...interface{})
Infof(format string, args ...interface{})
}
Logger is an interface for SSNTP users to define their own SSNTP tracing routines. By default we use errLog and we also provide Log, a glog based SSNTPLogger implementation.
type Role ¶
type Role uint32
Role describes the SSNTP role for the frame sender. It can be UNKNOWN, SERVER, Controller, AGENT, SCHEDULER, NETAGENT or CNCIAGENT.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is the SSNTP server structure. This is an SSNTP server handle to start listening and handling SSNTP client connections, and send SSNTP frames to them. It is an entirely opaque structure, only accessible through its public methods.
func (*Server) SendCommand ¶
SendCommand sends a specific command and its payload to a client. The client is specified by its uuid
func (*Server) SendError ¶
SendError sends an error back to a client. The client is specified by its uuid
func (*Server) SendEvent ¶
SendEvent sends a specific status and its payload to a client. The client is specified by its uuid
func (*Server) SendStatus ¶
SendStatus sends a specific status and its payload to a client. The client is specified by its uuid
func (*Server) SendTracedCommand ¶
func (server *Server) SendTracedCommand(uuid string, cmd Command, payload []byte, trace *TraceConfig) (int, error)
SendTracedCommand sends a specific command and its payload to a client. The SSNTP command frame will be traced according to the trace argument. The client is specified by its uuid
func (*Server) SendTracedError ¶
func (server *Server) SendTracedError(uuid string, error Error, payload []byte, trace *TraceConfig) (int, error)
SendTracedError sends an error back to a client. The SSNTP error frame will be traced according to the trace argument. The client is specified by its uuid
func (*Server) SendTracedEvent ¶
func (server *Server) SendTracedEvent(uuid string, event Event, payload []byte, trace *TraceConfig) (int, error)
SendTracedEvent sends a specific event and its payload to a client. The SSNTP event frame will be traced according to the trace argument. The client is specified by its uuid
func (*Server) SendTracedStatus ¶
func (server *Server) SendTracedStatus(uuid string, status Status, payload []byte, trace *TraceConfig) (int, error)
SendTracedStatus sends a specific status and its payload to a client. The SSNTP status frame will be traced according to the trace argument. The client is specified by its uuid
func (*Server) Serve ¶
func (server *Server) Serve(config *Config, ntf ServerNotifier) error
Serve starts an SSNTP server that will listen and serve SSNTP client connections. Notifiers will be called when new clients connect and disconnect. And also when statuses, payloads and errors are received.
Example ¶
package main
import (
"fmt"
)
type logger struct{}
func (l logger) Infof(format string, args ...interface{}) {
fmt.Printf("INFO: SSNTP Server: "+format, args...)
}
func (l logger) Errorf(format string, args ...interface{}) {
fmt.Printf("ERROR: SSNTP Server: "+format, args...)
}
func (l logger) Warningf(format string, args ...interface{}) {
fmt.Printf("WARNING: SSNTP Server: "+format, args...)
}
type ssntpDumpServer struct {
ssntp Server
name string
}
func (server *ssntpDumpServer) ConnectNotify(uuid string, role uint32) {
fmt.Printf("%s: %s connected (role 0x%x)\n", server.name, uuid, role)
}
func (server *ssntpDumpServer) DisconnectNotify(uuid string, role uint32) {
fmt.Printf("%s: %s disconnected (role 0x%x)\n", server.name, uuid, role)
}
func (server *ssntpDumpServer) StatusNotify(uuid string, status Status, frame *Frame) {
fmt.Printf("%s: STATUS %s from %s\n", server.name, status, uuid)
}
func (server *ssntpDumpServer) CommandNotify(uuid string, command Command, frame *Frame) {
fmt.Printf("%s: COMMAND %s from %s\n", server.name, command, uuid)
}
func (server *ssntpDumpServer) EventNotify(uuid string, event Event, frame *Frame) {
fmt.Printf("%s: EVENT %s from %s\n", server.name, event, uuid)
}
func (server *ssntpDumpServer) ErrorNotify(uuid string, error Error, frame *Frame) {
fmt.Printf("%s: ERROR (%s) from %s\n", server.name, error, uuid)
}
const agentUUID = "3390740c-dce9-48d6-b83a-a717417072ce"
func (server *ssntpDumpServer) CommandForward(uuid string, command Command, frame *Frame) (dest ForwardDestination) {
dest.AddRecipient(agentUUID)
return
}
func main() {
var config Config
server := &ssntpDumpServer{
name: "CIAO Echo Server",
}
config.Log = logger{}
config.CAcert = "MyServer.crt"
config.ForwardRules = []FrameForwardRule{
/* All STATS commands forwarded to Controllers. */
{
Operand: STATS,
Dest: Controller,
},
/* For START commands, server.CommandForward will decide where to forward them. */
{
Operand: START,
CommandForward: server,
},
}
server.ssntp.Serve(&config, server)
}
Output:
type ServerNotifier ¶
type ServerNotifier interface {
// ConnectNotify notifies of a new SSNTP client connection.
ConnectNotify(uuid string, role uint32)
// DisconnectNotify notifies of a SSNTP client having
// disconnected from us.
DisconnectNotify(uuid string, role uint32)
// StatusNotify notifies of a pending status frame.
// The frame comes from a SSNTP client identified by uuid.
StatusNotify(uuid string, status Status, frame *Frame)
// CommandNotify notifies of a pending command frame.
// The frame comes from a SSNTP client identified by uuid.
CommandNotify(uuid string, command Command, frame *Frame)
// EventNotify notifies of a pending event frame.
// The frame comes from a SSNTP client identified by uuid.
EventNotify(uuid string, event Event, frame *Frame)
// ErrorNotify notifies of a pending error frame.
// The frame comes from a SSNTP client identified by uuid.
ErrorNotify(uuid string, error Error, frame *Frame)
}
ServerNotifier is the SSNTP server notification interface. Any SSNTP server must implement this interface.
type Status ¶
type Status uint8
Status is the SSNTP Status operand. It can be CONNECTED, READY, FULL, OFFLINE or MAINTENANCE
const ( // CONNECTED is the reply to a client CONNECT command and thus only SSNTP servers can // send such frame. The CONNECTED status confirms the client that it's connected and // that it should be prepared to process and send commands and statuses. // The CONNECTED payload contains the cloud configuration data. Please refer to the // CONFIGURE command frame for more details. // // SSNTP CONNECTED Status frame // // +--------------------------------------------------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Role | Server UUID | Client UUID | Payload | YAML formatted | // | | | (0x1) | (0x0) | (bitmask of server roles) | | | Length | payload | // +--------------------------------------------------------------------------------------------------------------------+ CONNECTED Status = iota // READY is a status command CIAO agents send to the scheduler to notify them about // their readiness to launch some more work (Virtual machines, containers or bare metal // ones). It is the only way for an agent to notify the CIAO scheduler about its // compute node capacity change and thus its readiness to take some more work. The new // CN capacity is described in this frame's payload: // SSNTP READY Status frame // // +----------------------------------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | YAML formatted compute | // | | | (0x1) | (0x1) | | node new capacity | // +----------------------------------------------------------------------------+ READY // FULL is a status command CIAO agents send to the scheduler to let it know that // the compute node they control is now running at full capacity, i.e. it can temporarily // not run any additional work. The scheduler should stop sending START commands to such // agent until it receives a new READY status with some available capacity from it. // SSNTP FULL Status frame // // +---------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | // | | | (0x1) | (0x2) | (0x0) | // +---------------------------------------------------+ FULL // OFFLINE is used by agents to let everyone know that although they're still running // and connected to the SSNTP network they are not ready to receive any kind of command, // be it START, STOP or EVACUATE ones. // // SSNTP OFFLINE Status frame // // +---------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | // | | | (0x1) | (0x3) | (0x0) | // +---------------------------------------------------+ OFFLINE // MAINTENANCE is used by agents to let the scheduler know that it entered maintenance // mode. // // SSNTP MAINTENANCE Status frame // // +---------------------------------------------------+ // | Major | Minor | Type | Operand | Payload Length | // | | | (0x1) | (0x4) | (0x0) | // +---------------------------------------------------+ MAINTENANCE )
type StatusForwarder ¶
type StatusForwarder interface {
StatusForward(uuid string, status Status, frame *Frame) ForwardDestination
}
StatusForwarder is the SSNTP Status forwarding interface. The uuid argument is the sender's UUID.
type TraceConfig ¶
type TraceConfig struct {
// Label places a a label in the SSNTP frame sent
// using this config.
Label []byte
// Start is defined by the API caller to specify when
// operations related to that frames actually started.
// Together with SetEndStamp, this allows for an
// end-to-end timestamping.
Start time.Time
// PathTrace turns frame timestamping on or off.
PathTrace bool
}
TraceConfig is the SSNTP tracing configuration to be used when calling into the client SendTraced* APIs.
type Type ¶
type Type uint8
Type is the SSNTP frame type. It can be COMMAND, STATUS, ERROR or EVENT.
const ( // COMMAND frames are meant for SSNTP clients to send commands. // For example the Controller sends START or STOP commands to launch and // pause workloads. // SSNTP being asynchronous SSNTP commands are not replied to. COMMAND Type = iota // STATUS frames are mostly used by the launcher agent to report // about the node status. It is used by the scheduler as an indication // for its next scheduling decisions. Status frames can be seen as // a way of building flow control between the scheduler and the launchers. STATUS // ERROR frames contain error reports. Combining the error operand together // with the Error frame YAML payload allows for building a complete error // interpretation and description. // ERROR frames are typically sent for command failures. ERROR // EVENT frames carry asynchronous events that the receiver can decide to // broadcast or not. // EVENT frames describe a general, non erratic cluster event. EVENT )
Source Files