README
¶
Go tracing and monitoring (Prometheus) for net.Conn
Prometheus monitoring and x/net/trace tracing wrappers net.Conn, both inbound (net.Listener) and outbound (net.Dialer).
Why?
Go standard library does a great job of doing "the right" things with your connections: http.Transport pools outbound ones, and http.Server sets good Keep Alive defaults.
However, it is still easy to get it wrong, see the excellent The complete guide to Go net/http timeouts.
That's why you should be able to monitor (using Prometheus) how many connections your Go frontend servers have inbound, and how big are the connection pools to your backends. You should also be able to inspect your connection without ssh and netstat.
How to use?
All of these examples can be found in example/server.go:
Conntrack Dialer for HTTP DefaultClient
Most often people use the default http.DefaultClient that uses http.DefaultTransport. The easiest way to make sure all your outbound connections monitored and trace is:
http.DefaultTransport.(*http.Transport).DialContext = conntrack.NewDialContextFunc(
conntrack.DialWithTracing(),
conntrack.DialWithDialer(&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}),
)
Dialer Name
Tracked outbound connections are organised by dialer name (with default being default). The dialer name is used for monitoring (dialer_name label) and tracing (net.ClientConn.<dialer_name> family).
You can pass conntrack.WithDialerName() to NewDialContextFunc to set the name for the dialer. Moreover, you can set the dialer name per invocation of the dialer, by passing it in the Context. For example using the ctxhttp lib:
callCtx := conntrack.DialNameToContext(parentCtx, "google")
ctxhttp.Get(callCtx, http.DefaultClient, "https://www.google.com")
Conntrack Listener for HTTP Server
Tracked inbound connections are organised by listener name (with default being default). The listener name is used for monitoring (listener_name label) and tracing (net.ServerConn.<listener_name> family). For example, a simple http.Server can be instrumented like this:
listener, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
listener = conntrack.NewListener(listener,
conntrack.TrackWithName("http"),
conntrack.TrackWithTracing(),
conntrack.TrackWithTcpKeepAlive(5 * time.Minutes))
httpServer.Serve(listener)
Note, the TrackWithTcpKeepAlive. The default http.ListenAndServe adds a tcp keep alive wrapper to inbound TCP connections. conntrack.NewListener allows you to do that without another layer of wrapping.
TLS server example
The standard lobrary http.ListenAndServerTLS does a lot to bootstrap TLS connections, including supporting HTTP2 negotiation. Unfortunately, that is hard to do if you want to provide your own net.Listener. That's why this repo comes with connhelpers package, which takes care of configuring tls.Config for that use case. Here's an example of use:
listener, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
listener = conntrack.NewListener(listener,
conntrack.TrackWithName("https"),
conntrack.TrackWithTracing(),
conntrack.TrackWithTcpKeepAlive(5 * time.Minutes))
tlsConfig, err := connhelpers.TlsConfigForServerCerts(*tlsCertFilePath, *tlsKeyFilePath)
tlsConfig, err = connhelpers.TlsConfigWithHttp2Enabled(tlsConfig)
tlsListener := tls.NewListener(listener, tlsConfig)
httpServer.Serve(listener)
Status
This code is used by Improbable's HTTP frontending and proxying stack for debuging and monitoring of established user connections.
Additional tooling will be added if needed, and contributions are welcome.
#License
go-conntrack is released under the Apache 2.0 license. See the LICENSE file for details.
Documentation
¶
Index ¶
- func DialNameFromContext(ctx context.Context) string
- func DialNameToContext(ctx context.Context, dialerName string) context.Context
- func DialWithDialContextFunc(parentDialerFunc dialerContextFunc) dialerOpt
- func DialWithDialer(parentDialer *net.Dialer) dialerOpt
- func DialWithName(name string) dialerOpt
- func DialWithTracing() dialerOpt
- func DialWithoutMonitoring() dialerOpt
- func NewDialContextFunc(optFuncs ...dialerOpt) func(context.Context, string, string) (net.Conn, error)
- func NewDialFunc(optFuncs ...dialerOpt) func(string, string) (net.Conn, error)
- func NewListener(inner net.Listener, optFuncs ...listenerOpt) net.Listener
- func PreRegisterDialerMetrics(dialerName string)
- func TrackWithName(name string) listenerOpt
- func TrackWithRetries(b backoff.Backoff) listenerOpt
- func TrackWithTcpKeepAlive(keepalive time.Duration) listenerOpt
- func TrackWithTracing() listenerOpt
- func TrackWithoutMonitoring() listenerOpt
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DialNameFromContext ¶
DialNameFromContext returns the name of the dialer from the context of the DialContext func, if any.
func DialNameToContext ¶
DialNameToContext returns a context that will contain a dialer name override.
func DialWithDialContextFunc ¶
func DialWithDialContextFunc(parentDialerFunc dialerContextFunc) dialerOpt
DialWithDialContextFunc allows you to override func gets used for the actual dialing. The default is `net.Dialer.DialContext`.
func DialWithDialer ¶
DialWithDialer allows you to override the `net.Dialer` instance used to actually conduct the dials.
func DialWithName ¶
func DialWithName(name string) dialerOpt
DialWithName sets the name of the dialer for tracking and monitoring. This is the name for the dialer (default is `default`), but for `NewDialContextFunc` can be overwritten from the Context using `DialNameToContext`.
func DialWithTracing ¶
func DialWithTracing() dialerOpt
DialWithTracing turns *on* the /debug/events tracing of the dial calls.
func DialWithoutMonitoring ¶
func DialWithoutMonitoring() dialerOpt
DialWithoutMonitoring turns *off* Prometheus monitoring for this dialer.
func NewDialContextFunc ¶
func NewDialContextFunc(optFuncs ...dialerOpt) func(context.Context, string, string) (net.Conn, error)
NewDialContextFunc returns a `DialContext` function that tracks outbound connections. The signature is compatible with `http.Tranport.DialContext` and is meant to be used there.
func NewDialFunc ¶
NewDialFunc returns a `Dial` function that tracks outbound connections. The signature is compatible with `http.Tranport.Dial` and is meant to be used there for Go < 1.7.
func NewListener ¶
NewListener returns the given listener wrapped in connection tracking listener.
func PreRegisterDialerMetrics ¶
func PreRegisterDialerMetrics(dialerName string)
preRegisterDialerMetrics pre-populates Prometheus labels for the given dialer name, to avoid Prometheus missing labels issue.
func TrackWithName ¶
func TrackWithName(name string) listenerOpt
TrackWithName sets the name of the Listener for use in tracking and monitoring.
func TrackWithRetries ¶
TrackWithRetries enables retrying of temporary Accept() errors, with the given backoff between attempts. Concurrent accept calls that receive temporary errors have independent backoff scaling.
func TrackWithTcpKeepAlive ¶
TrackWithTcpKeepAlive makes sure that any `net.TCPConn` that get accepted have a keep-alive. This is useful for HTTP servers in order for, for example laptops, to not use up resources on the server while they don't utilise their connection. A value of 0 disables it.
func TrackWithTracing ¶
func TrackWithTracing() listenerOpt
TrackWithTracing turns *on* the /debug/events tracing of the live listener connections.
func TrackWithoutMonitoring ¶
func TrackWithoutMonitoring() listenerOpt
TrackWithoutMonitoring turns *off* Prometheus monitoring for this listener.
Types ¶
This section is empty.
Source Files
Directories