multiproxy

README

MultiProxy Client for Go

Overview

MultiProxy Client is a robust Go library designed to manage multiple HTTP/HTTPS and SOCKS5 proxies efficiently. It provides a fault-tolerant and load-balanced approach to making HTTP requests through a pool of proxies, with features like automatic retries, backoff mechanisms, and proxy rotation.

Features

• Multiple proxy support
• Automatic proxy rotation
• Fault tolerance with retry mechanism
• Configurable timeouts and delays
• Cookie management
• Basic authentication support for proxies
• User-Agent rotation
• HTTPS and SOCKS5 proxy support
• Concurrent request handling using singleflight pattern
• Rate limiting for individual proxies
• Configurable proxy rotation
• Backoff mechanism for failed proxies

Installation

To use MultiProxy Client in your Go project, you can install it using go get:

go get github.com/presbrey/go-multiproxy

Replace yourusername with the actual GitHub username or organization where this project is hosted.

Usage

Here's a basic example of how to use the MultiProxy Client:

package main

import (
    "fmt"
    "net/http"
    "time"
    
    "github.com/presbrey/go-multiproxy"
)

func main() {
    config := multiproxy.Config{
        Proxies: []multiproxy.Proxy{
            {
                URL:  &url.URL{Scheme: "http", Host: "proxy1.example.com:8080"},
                Auth: &multiproxy.ProxyAuth{Username: "user1", Password: "pass1"},
            },
            {
                URL:  &url.URL{Scheme: "socks5", Host: "proxy2.example.com:1080"},
                Auth: &multiproxy.ProxyAuth{Username: "user2", Password: "pass2"},
            },
        },
        CookieTimeout:    10 * time.Minute,
        CookieOptions:    &cookiejar.Options{PublicSuffixList: publicsuffix.List},
        DialTimeout:      30 * time.Second,
        RequestTimeout:   1 * time.Minute,
        RetryAttempts:    3,
        RetryDelay:       5 * time.Second,
        ProxyRotateCount: 10,
    }

    client, err := multiproxy.NewClient(config)
    if err != nil {
        panic(err)
    }

    resp, err := client.Get("https://example.com")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    fmt.Printf("Response status: %s\n", resp.Status)
}

Configuration

The Config struct allows you to customize the behavior of the MultiProxy Client:

• Proxies: List of Proxy structs, each containing:

  • URL: The URL of the proxy
  • Auth: Pointer to ProxyAuth struct with Username and Password
  • UserAgent: User-Agent string for this specific proxy
  • RateLimit: Duration for rate limiting requests to this specific proxy

• ProxyRotateCount: Number of requests after which to rotate to the next proxy

• BackoffTime: Time to wait before retrying a failed proxy

• DialTimeout: Timeout for establishing a connection to a proxy

• RequestTimeout: Timeout for the entire request (including dialing, writing request, and reading response)

• RetryDelay: Delay between retry attempts

• CookieOptions: Options for configuring the cookie jar (see http.cookiejar.Options)

• CookieTimeout: Duration for which cookies are valid

• DefaultUserAgent: Default User-Agent string to use if not specified in ProxyUserAgents

• RetryAttempts: Number of times to retry a failed request

• RetryDelay: Delay between retry attempts

• BackoffTime: Time to wait before retrying a failed proxy

• InsecureSkipVerify: Whether to skip TLS certificate verification

Using the RoundTripper

The MultiProxy Client provides a RoundTripper() method that returns an http.RoundTripper. This allows you to use the multi-proxy functionality with any http.Client. Here's an example:

config := multiproxy.Config{
    // ... your config here ...
}

client, err := multiproxy.NewClient(config)
if err != nil {
    // handle error
}

httpClient := &http.Client{
    Transport: client.RoundTripper(),
}

// Now use httpClient for your requests
resp, err := httpClient.Get("https://example.com")

This is particularly useful when you need to use the multi-proxy functionality with libraries or APIs that accept an http.Client.

Testing

The project includes comprehensive test suites:

• multiproxy_test.go: Tests for the main MultiProxy Client functionality
• connect_test.go: Tests for HTTPS and CONNECT proxy functionality
• errors_test.go: Tests for error handling scenarios

To run the tests, use the following command:

go test ./...

Contributing

Contributions to the MultiProxy Client are welcome! Please feel free to submit issues, fork the repository and send pull requests.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Disclaimer

This software is provided as-is, and users should be aware of the legal and ethical considerations when using proxy servers. Always ensure you have the right to use the proxy servers you configure with this client.

Documentation

Index

• type Client
  • func NewClient(config Config) (*Client, error)
  • func (c *Client) Do(req *http.Request) (*http.Response, error)
  • func (c *Client) Get(url string) (*http.Response, error)
  • func (c *Client) Head(url string) (*http.Response, error)
  • func (c *Client) NewRequest(method, url string, body io.Reader) (*http.Request, error)
  • func (c *Client) Post(url, contentType string, body io.Reader) (*http.Response, error)
  • func (c *Client) RoundTripper() http.RoundTripper
• type Config
• type Proxy
• type ProxyAuth

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

func NewClient

func NewClient(config Config) (*Client, error)

NewClient creates a new Client with the given configuration. It returns an error if no proxies are provided in the configuration.

func (*Client) Do

func (c *Client) Do(req *http.Request) (*http.Response, error)

Do sends an HTTP request and returns an HTTP response, following policy (such as redirects, cookies, auth) as configured on the client.

func (*Client) Get

func (c *Client) Get(url string) (*http.Response, error)

Get issues a GET request to the specified URL.

func (*Client) Head

func (c *Client) Head(url string) (*http.Response, error)

Head issues a HEAD request to the specified URL.

func (*Client) NewRequest

func (c *Client) NewRequest(method, url string, body io.Reader) (*http.Request, error)

NewRequest creates a new http.Request with the provided method, URL, and optional body. It sets the default User-Agent if configured.

func (*Client) Post

func (c *Client) Post(url, contentType string, body io.Reader) (*http.Response, error)

Post issues a POST request to the specified URL with the given content type and body.

func (*Client) RoundTripper

func (c *Client) RoundTripper() http.RoundTripper

RoundTripper returns an http.RoundTripper that uses the Client's proxies.

type Config

type Config struct {
	// Proxy configuration
	Proxies          []Proxy
	ProxyRotateCount int

	// Timeouts and delays
	BackoffTime    time.Duration
	DialTimeout    time.Duration
	RequestTimeout time.Duration
	RetryDelay     time.Duration

	// Cookie handling
	CookieOptions *cookiejar.Options
	CookieTimeout time.Duration

	// User-Agent configuration
	DefaultUserAgent string

	// Retry configuration
	RetryAttempts int

	// TLS configuration
	InsecureSkipVerify bool
}

type Proxy

type Proxy struct {
	URL       *url.URL
	Auth      *ProxyAuth
	UserAgent string
	RateLimit time.Duration
}

type ProxyAuth

type ProxyAuth struct {
	Username string
	Password string
}