remotedialer

README

Reverse Tunneling Dialer

remotedialer creates a two-way connection between a server and a client, so that a net.Dial can be performed on the server and actually connects to the client running services accessible on localhost or behind a NAT or firewall.

Architecture

Abstractions

remotedialer consists of structs that organize and abstract a TCP connection between a server and a client. Both client and server use Sessions, which provides a means to make a network connection to a remote endpoint and keep track of incoming connections. A server uses the Server object which contains a sessionManager which governs one or more Sessions, while a client creates a Session directly. A connection implements the io.Reader and io.WriteCloser interfaces, so it can be read from and written to directly. The connection's internal readBuffer monitors the size of the data it is carrying and uses backPressure to pause incoming data transfer until the amount of data is below a threshold.

Data flow

A client establishes a session with a server using the server's URL. The server upgrades the connection to a websocket connection which it uses to create a Session. The client then also creates a Session with the websocket connection.

The client sits in front of some kind of HTTP server, most often a kubernetes API server, and acts as a reverse proxy for that HTTP resource. When a user requests a resource from this remote resource, request first goes to the remotedialer server. The application containing the server is responsible for routing the request to the right client connection.

The request is sent through the websocket connection to the client and read into the client's connection buffer. A pipe is created between the client and the HTTP service which continually copies data between each socket. The request is forwarded through the pipe to the remote service, draining the buffer. The service's response is copied back to the client's buffer, and then forwarded back to the server and copied into the server connection's own buffer. As the user reads the response, the server's buffer is drained.

The pause/resume mechanism checks the size of the buffer for both the client and server. If it is greater than the threshold, a PAUSE message is sent back to the remote connection, as a suggestion not to send any more data. As the buffer is drained, either by the pipe to the remote HTTP service or the user's connection, the size is checked again. When it is lower than the threshold, a RESUME message is sent, and the data transfer may continue.

remotedialer in the Rancher ecosystem

remotedialer is used to connect Rancher to the downstream clusters it manages, enabling a user agent to access the cluster through an endpoint on the Rancher server. remotedialer is used in three main ways:

Agent config and tunnel server

When the agent starts, it initially makes a client connection to the endpoint /v3/connect/register, which runs an authorizer that sets some initial data about the node. The agent continues to connect to the endpoint /v3/connect on a loop. On each connection, it runs an OnConnect handler which pulls down node configuration data from /v3/connect/config.

Steve Aggregation

The steve aggregation server on the agent establishes a remotedialer Session with Rancher, making the steve API on the downstream cluster accessible from the Rancher server and facilitating resource watches.

Health Check

The clusterconnected controller in Rancher uses the established tunnel to check that clusters are still responsive and sets alert conditions on the cluster object if they are not.

HA operation (peering)

remotedialer supports a mode where multiple servers can be configured as peers. In that mode all servers maintain a mapping of all remotedialer client connections to all other servers, and can route incoming requests appropriately.

Therefore, http requests referring any of the remotedialer clients can be resolved by any of the peer servers. This is useful for high availability, and Rancher leverages that functionality to distribute downstream clusters (running agents acting as remotedialer clients) among replica pods (acting as remotedialer server peers). In case one Rancher replica pod breaks down, Rancher will reassign its downstream clusters to others.

Peers authenticate to one another via a shared token.

Running Locally

remotedialer provides an example client and server which can be run in standalone mode FOR TESTING ONLY. These are found in the server/ and client/ directories.`

Compile

Compile the server and client:

make server
make client

Run

Start the server first.

./server/server

The server has debug mode off by default. Enable it with --debug.

The client proxies requests from the remotedialer server to a web server, so it needs to be run somewhere where it can access the web server you want to target. The remotedialer server also needs to be reachable by the client.

For testing purposes, a basic HTTP file server is provided. Build the server with:

make dummy

Create a directory with files to serve, then run the web server from that directory:

mkdir www
cd www
echo 'hello' > bigfile
/path/to/dummy

Run the client with

./client/client

Both server and client can be run with even more verbose logging:

CATTLE_TUNNEL_DATA_DEBUG=true ./server/server --debug
CATTLE_TUNNEL_DATA_DEBUG=true ./client/client

Usage

If the remotedialer server is running on 192.168.0.42, and the web service that the client can access is running at address 127.0.0.1:8125, make proxied requests like this:

curl http://192.168.0.42:8123/client/foo/http/127.0.0.1:8125/bigfile

where foo is the hardcoded client ID for this test server.

This test server only supports GET requests.

HA Usage

To test remotedialer in HA mode, first start the dummy server from an appropriate directory, eg.:

cd /tmp
mkdir www
cd www
echo 'hello' > bigfile
/path/to/dummy -listen :8125

Then start two peer remotedialer servers with the -peers id:token:url flag:

./server/server -debug -id first -token aaa -listen :8123 -peers second:aaa:ws://localhost:8124/connect &
./server/server -debug -id second -token aaa -listen :8124 -peers first:aaa:ws://localhost:8123/connect

Then connect a client to the first server, eg:

./client/client -id foo -connect ws://localhost:8123/connect

Finally, use the second server to make a request to the client via the first server:

curl http://localhost:8124/client/foo/http/127.0.0.1:8125/

Documentation

Index

• Constants
• Variables
• func ClientConnect(ctx context.Context, wsURL string, headers http.Header, ...) error
• func ConnectToProxy(rootCtx context.Context, proxyURL string, headers http.Header, ...) error
• func ConnectToProxyWithDialer(rootCtx context.Context, proxyURL string, headers http.Header, ...) error
• func DefaultErrorWriter(rw http.ResponseWriter, req *http.Request, code int, err error)
• type Authorizer
• type ConnectAuthorizer
• type Dialer
• type ErrorWriter
• type Server
  • func New(auth Authorizer, errorWriter ErrorWriter) *Server
  • func (s *Server) AddPeer(url, id, token string)
  • func (s *Server) Dialer(clientKey string) Dialer
  • func (s *Server) HasSession(clientKey string) bool
  • func (s *Server) RemovePeer(id string)
  • func (s *Server) ServeHTTP(rw http.ResponseWriter, req *http.Request)
• type Session
  • func NewClientSession(auth ConnectAuthorizer, conn *websocket.Conn) *Session
  • func NewClientSessionWithDialer(auth ConnectAuthorizer, conn *websocket.Conn, dialer Dialer) *Session
  • func (s *Session) Close()
  • func (s *Session) Dial(ctx context.Context, proto, address string) (net.Conn, error)
  • func (s *Session) Serve(ctx context.Context) (int, error)

Constants

const (
	// Data is the main message type, used to transport application data
	Data messageType = iota + 1
	// Connect is a control message type, used to request opening a new connection
	Connect
	// Error is a message type used to send an error during the communication.
	// Any receiver of an Error message can assume the connection can be closed.
	// io.EOF is used for graceful termination of connections.
	Error
	// AddClient is a message type used to open a new client to the peering session
	AddClient
	// RemoveClient is a message type used to remove an existing client from a peering session
	RemoveClient
	// Pause is a message type used to temporarily stop a given connection
	Pause
	// Resume is a message type used to resume a paused connection
	Resume
	// SyncConnections is a message type used to communicate active connection IDs.
	// The receiver can consider any ID not present in this message as stale and free any associated resource.
	SyncConnections
)

const (
	PingWaitDuration  = 60 * time.Second
	PingWriteInterval = 5 * time.Second
	// SyncConnectionsInterval is the time after which the client will send the list of active connection IDs
	SyncConnectionsInterval = 60 * time.Second
	// SyncConnectionsTimeout sets the maximum duration for a SyncConnections operation
	SyncConnectionsTimeout = 60 * time.Second
	MaxRead                = 8192
	HandshakeTimeOut       = 10 * time.Second
	// SendErrorTimeout sets the maximum duration for sending an error message to close a single connection
	SendErrorTimeout = 5 * time.Second
)

const (
	MaxBuffer = 1 << 21
)

Variables

var (
	Token = "X-API-Tunnel-Token"
	ID    = "X-API-Tunnel-ID"
)

var PrintTunnelData bool

PrintTunnelData No tunnel logging by default

Functions

func ClientConnect

func ClientConnect(ctx context.Context, wsURL string, headers http.Header, dialer *websocket.Dialer,
	auth ConnectAuthorizer, onConnect func(context.Context, *Session) error) error

ClientConnect connect to WS and wait 5 seconds when error

func ConnectToProxy

func ConnectToProxy(rootCtx context.Context, proxyURL string, headers http.Header, auth ConnectAuthorizer, dialer *websocket.Dialer, onConnect func(context.Context, *Session) error) error

ConnectToProxy connects to the websocket server. Local connections on behalf of the remote host will be dialed using a default net.Dialer.

func ConnectToProxyWithDialer

func ConnectToProxyWithDialer(rootCtx context.Context, proxyURL string, headers http.Header, auth ConnectAuthorizer, dialer *websocket.Dialer, localDialer Dialer, onConnect func(context.Context, *Session) error) error

ConnectToProxyWithDialer connects to the websocket server. Local connections on behalf of the remote host will be dialed using the provided Dialer function.

func DefaultErrorWriter

func DefaultErrorWriter(rw http.ResponseWriter, req *http.Request, code int, err error)

Types

type Authorizer

type Authorizer func(req *http.Request) (clientKey string, authed bool, err error)

type ConnectAuthorizer

type ConnectAuthorizer func(proto, address string) bool

ConnectAuthorizer custom for authorization

type Dialer

type Dialer func(ctx context.Context, network, address string) (net.Conn, error)

type ErrorWriter

type ErrorWriter func(rw http.ResponseWriter, req *http.Request, code int, err error)

type Server

type Server struct {
	PeerID                  string
	PeerToken               string
	ClientConnectAuthorizer ConnectAuthorizer
	// contains filtered or unexported fields
}

func New

func New(auth Authorizer, errorWriter ErrorWriter) *Server

func (*Server) AddPeer

func (s *Server) AddPeer(url, id, token string)

func (*Server) Dialer

func (s *Server) Dialer(clientKey string) Dialer

func (*Server) HasSession

func (s *Server) HasSession(clientKey string) bool

func (*Server) RemovePeer

func (s *Server) RemovePeer(id string)

func (*Server) ServeHTTP

func (s *Server) ServeHTTP(rw http.ResponseWriter, req *http.Request)

type Session

type Session struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

func NewClientSession

func NewClientSession(auth ConnectAuthorizer, conn *websocket.Conn) *Session

func NewClientSessionWithDialer

func NewClientSessionWithDialer(auth ConnectAuthorizer, conn *websocket.Conn, dialer Dialer) *Session

func (*Session) Close

func (s *Session) Close()

func (*Session) Dial

func (s *Session) Dial(ctx context.Context, proto, address string) (net.Conn, error)

func (*Session) Serve

func (s *Session) Serve(ctx context.Context) (int, error)