httransform

README

httransform

httransform is the library/framework to build your own HTTP proxies. It relies on high-performant and memory-efficient fasthttp library as HTTP base layer and can give you the ability to build a proxy where you can control every aspect.

Main features of this framework:

1. Support of HTTP (plain HTTP) proxy protocol.
2. Support of HTTPS (with CONNECT method) protocol. This library does MITM and provides the possibility to generate TLS certificates for the hosts on-the-fly.
3. Keeps and maintains the order of headers and their case (no normalization).
4. Supports the concept of layers or middlewares which process HTTP requests and responses.
5. Supports custom executors: a function which converts HTTP requests to HTTP responses. Allowing your proxy to fetch the data from other services, which are not necessarily HTTP. The executor simply converts HTTP request structure to HTTP response.
6. Can support connection upgrades (this includes websockets) with an ability to look into them.

Please check the full documentation for more details.

Example

Just a small example to give you the feeling of how it all looks like:

package main

import (
	"context"
	"net"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/9seconds/httransform/v2"
	"github.com/9seconds/httransform/v2/auth"
	"github.com/9seconds/httransform/v2/layers"
)

// These are generates examples of self-signed certificates
// to simplify the example.
var caCert = []byte(`-----BEGIN CERTIFICATE-----
MIICWzCCAcSgAwIBAgIJAJ34yk7oiKv5MA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV
BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX
aWRnaXRzIFB0eSBMdGQwHhcNMTgxMjAyMTQyNTAyWhcNMjgxMTI5MTQyNTAyWjBF
MQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50
ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
gQDL7Hzfmx7xfFWTRm26t/lLsCZwOri6VIzp2dYM5Hp0dV4XUZ+q60nEbHwN3Usr
GKAK/Rsr9Caam3A18Upn2ly69Tyr29kVK+PlsOgSSCUnAYcqT166/j205n3CGNLL
OPtQKfAT/iH3dPBObd8N4FR9FlXiYIiAp1opCbyu2mlHiwIDAQABo1MwUTAdBgNV
HQ4EFgQUOJ+uGtIhHxXHPNESBNI4YbwAl+wwHwYDVR0jBBgwFoAUOJ+uGtIhHxXH
PNESBNI4YbwAl+wwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOBgQCW
s7P0wJ8ON8ieEJe4pAfACpL6IyhZ5YK/C/hip+czxdvZHc5zngVwHP2vsIcHKBTr
8qXoHgh2gaXqwn8kRVNnZzWrxgSe8IR3oJ2yTbLAxqDS42SPfRLAUpy9sK/tEEGM
rMk/LWMzH/S6bLcsAm0GfVIrUNfg0eF0ZVIjxINBVA==
-----END CERTIFICATE-----`)

var caPrivateKey = []byte(`-----BEGIN PRIVATE KEY-----
MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBAMvsfN+bHvF8VZNG
bbq3+UuwJnA6uLpUjOnZ1gzkenR1XhdRn6rrScRsfA3dSysYoAr9Gyv0JpqbcDXx
SmfaXLr1PKvb2RUr4+Ww6BJIJScBhypPXrr+PbTmfcIY0ss4+1Ap8BP+Ifd08E5t
3w3gVH0WVeJgiICnWikJvK7aaUeLAgMBAAECgYAk+/kR3OJZzcD/evB/wsoV7haq
mBvUv2znJLjrkayb3oV4GTeqGg5A76P4J8BwSoEMPSdma1ttAu/w+JgUCchzVPwU
34Sr80mYawOmGVGJsDnrrYA2w51Nj42e71pmRc9IqNLwFEhW5Uy7eASf3THJMWDl
F2M6xAVYr+X0eKLf4QJBAO8lVIIMnzIReSZukWBPp6GKmXOuEkWeBOfnYC2HOVZq
1M/E6naOP2MBk9CWG4o9ysjcZ1hosi3/txxrc8VmBAkCQQDaS651dpQ3TRE//raZ
s79ZBEdMCMlgXB6CPrZpvLz/3ZPcLih4MJ59oVkeFHCNct7ccQcQu4XHMGNBIRBh
kpvzAkEAlS/AjHC7T0y/O052upJ2jLweBqBtHaj6foFE6qIVDugOYp8BdXw/5s+x
GsrJ22+49Z0pi2mk3jVMUhpmWprNoQJBANdAT0v2XFpXfQ38bTQMYT82j9Myytdg
npjRm++Rs1AdvoIbZb52OqIoqoaVoxJnVchLD6t5LYXnecesAcok1e8CQEKB7ycJ
6yVwnBE3Ua9CHcGmrre6HmEWdPy1Zyb5DQC6duX46zEBzti9oWx0DJIQRZifeCvw
4J45NsSQjuuAAWs=
-----END PRIVATE KEY-----`)

func main() {
	// Root context is crucial here. When root context is closed, a
	// proxy is shutdown.
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// For demo purpose we are going to close by SIGINT and SIGTERM
	// signals.
	signals := make(chan os.Signal, 1)

	signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM)

	go func() {
		for range signals {
			cancel()
		}
	}()

	// Filter layer is required if you want to drop request
	// which are made to you internal networks.
	filterLayer, err := layers.NewFilterSubnetsLayer([]net.IPNet{
		{IP: net.ParseIP("127.0.0.0"), Mask: net.CIDRMask(8, 32)},
		{IP: net.ParseIP("172.16.0.0"), Mask: net.CIDRMask(12, 32)},
		{IP: net.ParseIP("10.0.0.0"), Mask: net.CIDRMask(8, 32)},
		{IP: net.ParseIP("192.168.0.0"), Mask: net.CIDRMask(16, 32)},
	})
	if err != nil {
		panic(err)
	}

	opts := httransform.ServerOpts{
		TLSCertCA:     caCert,
		TLSPrivateKey: caPrivateKey,
		// We are going to use basic proxy authorization with username
		// 'user' and password 'password'.
		Authenticator: auth.NewBasicAuth(map[string]string{
			"user": "password",
		}),
		Layers: []layers.Layer{
			filterLayer,
			// This guy will remove Proxy headers from the request.
			layers.ProxyHeadersLayer{},
			// This guy is going to limit request processing time to 3
			// minutes.
			layers.TimeoutLayer{
				Timeout: 3 * time.Minute,
			},
		},
	}

	proxy, err := httransform.NewServer(ctx, opts)
	if err != nil {
		panic(err)
	}

	// We bind our proxy to the port 3128 and all interfaces.
	listener, err := net.Listen("tcp", ":3128")
	if err != nil {
		panic(err)
	}

	if err := proxy.Serve(listener); err != nil {
		panic(err)
	}
}

This will create an HTTP proxy on 127.0.0.1:3128. It will also require authentication (user and password) and will remove the Proxy-Authorization header before sending the request further.

v1 version

Version 1 is not supported anymore. Please use version 2. Version 2 has a lot of breaking changes but which really help to maintain a package.

Unfortunately, I cannot enumerate all of them and I understand that it is going to make your life painful in some moments. Sorry for that. v2 was a complete rewrite and I assume that given a size of this library, it won't take a lot of time for you to migrate. But if you have any questions, feel free to open an issue. I'm happy to help.

Documentation

Overview

httransform is a framework to build a various MITM proxies.

This framework is built using fasthttp (https://github.com/valyala/fasthttp) library to provide fast processing of HTTP requests. Also, as fasthttp it actively uses object pooling to reduce the pressure on GC and possibly save a little bit of memory. Unlike its alternative, goproxy (https://github.com/elazarl/goproxy), httransform does not have disadvantages of using net/http: we can keep header case and order.

This framework provides features to build robust high-performant HTTP and HTTPS proxies (with the support of the CONNECT method and TLS certificate generation on-the-fly), it supports middleware layers and custom executors. It is also able to correctly process different http connection upgrades like websocket support.

Layers are middlewares which do preprocess of the requests or postprocessing of the response. The thing about layers is that you have to define both directions: to executor and from executor. We encourage to create layers which know as less as possible about each other. If, for example, you raise an error in one layer, this error has to be processed only there, other layers should ignore that. This framework does not restrict any deviations from this rule, but you'll get more simple and clean design if you treat layers as independent as possible.

An executor is some function which converts HTTP request to response. In the most simple case, an executor is HTTP client. But also, you can convert the request to JSON, send it somewhere, get ProtoBuf back and convert it to HTTP response. Or you can play with NATS. Or 0mq. Or RabbitMQ. You got the idea. Executor defines the function which converts HTTP request to HTTP response. If connection upgrade is required, this is a responsibility of the executor to make it.

If any of your layers creates an error, executor won't be called. HTTP response will be converted to 500 response and error would be propagated by the chain of layers back.

A Mental Scheme of the Request

Just take a look here:

        HTTP interface            Layer 1             Layer 2
     +----------------+      **************      **************
     |                |      *            *      *            *       ==============
---> |  HTTP request  | ===> *  OnRequest * ===> *  OnRequest * ===>  =            =
     |                |      *            *      *            *       =            =
     +----------------+      **************      **************       =  Executor  =
     |                |      *            *      *            *       =            =
<--- |  HTTP response | <=== * OnResponse * <=== * OnResponse * <===  =            =
     |                |      *            *      *            *       ==============
     +----------------+      **************      **************

As you see, the request goes through the all layers forward and backward. This is a contract of this package.

Features

1. Interface of HTTP proxy

2. Interface of HTTPS proxy (e.g CONNECT method)

3. CONNECT method can be used to establish plain TCP tunnels. Upgrade to TLS is not mandatory.

4. Both HTTP request and responses can be processed: headers are added/removed/changed and so on.

5. TCP connection upgrade is supported. Websockets are supported. But by default you can't interfere: you can just watch.

Index

• Constants
• type Server
  • func NewServer(ctx context.Context, opts ServerOpts) (*Server, error)
  • func (s *Server) Close() error
  • func (s *Server) Serve(ln net.Listener) error
• type ServerOpts
  • func (s *ServerOpts) GetAuthenticator() auth.Interface
  • func (s *ServerOpts) GetConcurrency() int
  • func (s *ServerOpts) GetEventProcessorFactory() events.ProcessorFactory
  • func (s *ServerOpts) GetExecutor() executor.Executor
  • func (s *ServerOpts) GetLayers() []layers.Layer
  • func (s *ServerOpts) GetMaxRequestBodySize() int
  • func (s *ServerOpts) GetReadBufferSize() int
  • func (s *ServerOpts) GetReadTimeout() time.Duration
  • func (s *ServerOpts) GetTCPKeepAlivePeriod() time.Duration
  • func (s *ServerOpts) GetTLSCertCA() []byte
  • func (s *ServerOpts) GetTLSPrivateKey() []byte
  • func (s *ServerOpts) GetTLSSkipVerify() bool
  • func (s *ServerOpts) GetWriteBufferSize() int
  • func (s *ServerOpts) GetWriteTimeout() time.Duration

Examples

• Server (ProxyExecutor)

Constants

const (
	// DefaultConcurrency defines a default number of concurrently
	// managed requests processed by a proxy.
	DefaultConcurrency = 65536

	// DefaultReadBufferSize defines a default size of the buffer to use
	// on reading client requests.
	DefaultReadBufferSize = 16 * 1024

	// DefaultWriteBufferSize defines a default size of the buffer to
	// use on writing to client socket.
	DefaultWriteBufferSize = 16 * 1024

	// DefaultReadTimeout defines a default timeout for reading of
	// client request.
	DefaultReadTimeout = time.Minute

	// DefaultWriteTimeout defines a default timeout for writing
	// to client.
	DefaultWriteTimeout = time.Minute

	// DefaultTCPKeepAlivePeriod defines a default time period for TCP
	// keepalive probes.
	DefaultTCPKeepAlivePeriod = 30 * time.Second

	// DefaultMaxRequestBodySize defines a max size of the request body.
	DefaultMaxRequestBodySize = 1024 * 1024 * 100
)

Types

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server defines a MITM proxy instance. Please pay attention that it has its own context. If this context is cancelled, Server starts to gracefully terminate.

Example (ProxyExecutor)

dialer, _ := dialers.DialerFromURL(dialers.Opts{}, "http://user:password@myproxy.host:3128")
opts := httransform.ServerOpts{
	Executor: executor.MakeDefaultExecutor(dialer),
}
proxy, _ := httransform.NewServer(context.Background(), opts)
listener, _ := net.Listen("tcp", ":3128")

proxy.Serve(listener)

func NewServer

func NewServer(ctx context.Context, opts ServerOpts) (*Server, error)

NewServer creates a new instance of the server based on a given options.

func (*Server) Close

func (s *Server) Close() error

Close stops server.

func (*Server) Serve

func (s *Server) Serve(ln net.Listener) error

Serve starts to serve on given net.Listener instance.

type ServerOpts

type ServerOpts struct {
	// Concurrency defines a number of concurrently managed requests.
	Concurrency uint

	// ReadBufferSize defines a size of the buffer allocated for reading
	// from client socket.
	ReadBufferSize uint

	// WriteBufferSize defines a size of the buffer allocated for
	// writing to a client socket.
	WriteBufferSize uint

	// MaxRequestBodySize defines a max size of the request body.
	MaxRequestBodySize uint

	// ReadTimeout defines a timeout for reading from client socket.
	ReadTimeout time.Duration

	// WriteTimeout defines a timeout for writing to client socket.
	WriteTimeout time.Duration

	// TCPKeepAlivePeriod defines a time period between 2 consecutive
	// TCP keepalive probes.
	TCPKeepAlivePeriod time.Duration

	// EventProcessorFactory defines a factory method which produces
	// event processors.
	EventProcessorFactory events.ProcessorFactory

	// TLSCertCA is a bytes which contains TLS CA certificate. This
	// certificate is required for generating fake TLS certifiates for
	// websites on TLS connection upgrades.
	TLSCertCA []byte

	// TLSPrivateKey is a bytes which contains TLS private key. This
	// certificate is required for generating fake TLS certifiates for
	// websites on TLS connection upgrades.
	TLSPrivateKey []byte

	// Layers defines a list of layers, middleware which should be used
	// by proxy.
	Layers []layers.Layer

	// Executor defines an executor function which should be used
	// to terminate HTTP request and fill HTTP response.
	Executor executor.Executor

	// Authenticator is an interface which is used to authenticate a
	// request.
	Authenticator auth.Interface

	// TLSSkipVerify defines if we need to verify TLS certifiates we have
	// to deal with.
	TLSSkipVerify bool
}

ServerOpts defines server options to use.

Please pay attention that each field is optional. We do provide sane defaults if you do not want to pass anything there.

func (*ServerOpts) GetAuthenticator

func (s *ServerOpts) GetAuthenticator() auth.Interface

GetLayers returns an authenticator instance to use paying attention to default value (no auth).

func (*ServerOpts) GetConcurrency

func (s *ServerOpts) GetConcurrency() int

GetConcurrency returns a concurrency paying attention to default value.

func (*ServerOpts) GetEventProcessorFactory

func (s *ServerOpts) GetEventProcessorFactory() events.ProcessorFactory

GetEventProcessorFactory returns an event factory paying attention to default value.

func (*ServerOpts) GetExecutor

func (s *ServerOpts) GetExecutor() executor.Executor

GetExecutor returns an instance of executor to use.

func (*ServerOpts) GetLayers

func (s *ServerOpts) GetLayers() []layers.Layer

GetLayers returns a set of server layers to use.

func (*ServerOpts) GetMaxRequestBodySize

func (s *ServerOpts) GetMaxRequestBodySize() int

GetMaxRequestBodySize returns max request body size paying attention to default value.

func (*ServerOpts) GetReadBufferSize

func (s *ServerOpts) GetReadBufferSize() int

GetReadBufferSize returns a read buffer size paying attention to default value.

func (*ServerOpts) GetReadTimeout

func (s *ServerOpts) GetReadTimeout() time.Duration

GetReadTimeout returns a read timeout paying attention to default value.

func (*ServerOpts) GetTCPKeepAlivePeriod

func (s *ServerOpts) GetTCPKeepAlivePeriod() time.Duration

GetTCPKeepAlivePeriod returns a period for TCP keepalive probes paying attention to default value.

func (*ServerOpts) GetTLSCertCA

func (s *ServerOpts) GetTLSCertCA() []byte

GetTLSCertCA returns a given TLS CA certificate.

func (*ServerOpts) GetTLSPrivateKey

func (s *ServerOpts) GetTLSPrivateKey() []byte

GetTLSPrivateKey returns a given TLS private key.

func (*ServerOpts) GetTLSSkipVerify

func (s *ServerOpts) GetTLSSkipVerify() bool

GetTLSSkipVerify returns a sign if we need to skip TLS verification.

func (*ServerOpts) GetWriteBufferSize

func (s *ServerOpts) GetWriteBufferSize() int

GetWriteBufferSize returns a write buffer size paying attention to default value.

func (*ServerOpts) GetWriteTimeout

func (s *ServerOpts) GetWriteTimeout() time.Duration

GetWriteTimeout returns a write timeout paying attention to default value.