fdcache

Documentation

Overview

Package fdcache contains a mechanism to obtain the file descriptor bound to a websocket.Conn. We initially implemented this mechanism for BBR hence the following explanation is mainly tailored on the BBR case. But we also have other use cases for this feature in tree, e.g. reading TCP_INFO.

To read BBR variables, we need a file descriptor. When serving a WebSocket client we have a websocket.Conn instance. The UnderlyingConn() method allows us to get the corresponding net.Conn, which typically is a tls.Conn. Yet, obtaining a file descriptor from a tls.Conn seems complex, because the underlying socket connection is private. Still, we've a custom listener that is required to turn on BBR (see tcpListenerEx). At that point, we can obtain a *os.File from the *net.TCPConn. From such *os.File, we can then get a file descriptor. However, there are some complications:

a) the returned *os.File is bound to another file descriptor that is

a dup() of the one inside the *net.Conn, so, we should keep that
*os.File alive during the whole ndt7 measurement;

b) in Go < 1.11, using this technique makes the file descriptor use

blocking I/O, which spawns more threads (see below).

For these reasons, we're keeping a cache mapping between the socket's four tuple (i.e. local and remote address and port) and the *os.File. We use the four tuple because, in principle, a server can be serving on more than a single local IP, so only using the remote endpoint may not be enough.

In the good case, this is what is gonna happen:

1. a connection is accepted in tcpListenerEx, so we have a *net.TCPConn;

2. using the *net.Conn, we turn on BBR and cache the *os.File using bbr.EnableAndRememberFile() with the four tuple as the key;

3. WebSocket negotiation is successful, so we have a websocket.Conn, from which we can get the underlying connection and hence the four tuple;

4. using the four tuple, we can retrieve the *os.File, removing it from the cache using bbr.GetAndForgetFile();

5. we defer *os.File.Close() until the end of the WebSocket serving loop and periodically we use such file to obtain the file descriptor and read the BBR variables using bbr.GetMaxBandwidthAndMinRTT().

Because a connection might be closed between steps 2. and 3. (i.e. after the connection is accepted and before the HTTP layer finishes reading the request and determines that it should be routed to the handler that we have configured), we also need a stale entry management mechanism so that we delete *os.File instances cached for too much time.

Depending on whether Golang calls shutdown() when a socket is closed or not, it might be that this caching mechanism keeps connections alive for more time than expected. The specific case where we can have this issue is the one where we receive a HTTP connection that is not a valid UPGRADE request, but a valid HTTP request. To avoid this issue, we SHOULD make sure to remove the *os.File from the cache basically everytime we got our handler called, regardless of whether the request is a valid UPGRADE.

Index

• func GetAndForgetFile(conn net.Conn) *os.File
• func GetUUID(conn net.Conn) (string, error)
• func OwnFile(conn net.Conn, fp *os.File)
• func TCPConnToFile(tc *net.TCPConn) (*os.File, error)

Functions

func GetAndForgetFile

func GetAndForgetFile(conn net.Conn) *os.File

GetAndForgetFile returns the *os.File bound to |conn| that was previously saved with EnableAndRememberFile, or nil if no file was found. Note that you are given ownership of the returned file pointer. As the name implies, the *os.File is removed from the cache by this operation.

func GetUUID

func GetUUID(conn net.Conn) (string, error)

GetUUID returns the UUID for a passed-in connection.

func OwnFile

func OwnFile(conn net.Conn, fp *os.File)

OwnFile transfers ownership of |fp| to the fdcache module. Passing a nil |fp| to this function is a programming error that will cause a panic.

func TCPConnToFile

func TCPConnToFile(tc *net.TCPConn) (*os.File, error)

TCPConnToFile maps |tc| to the corresponding *os.File. Note that the returned *os.File is a dup() of the original, hence you now have ownership of two objects that you need to remember to defer Close() of.