httpclient

Documentation

Overview

httpclient/client.go

The `http_client` package provides a configurable HTTP client tailored for interacting with specific APIs.

It supports different authentication methods, including "bearer" and "oauth". The client is designed with a focus on concurrency management, structured error handling, and flexible configuration options. The package offers a default timeout, custom backoff strategies, dynamic rate limiting, and detailed logging capabilities. The main `Client` structure encapsulates all necessary components, like the baseURL, authentication details, and an embedded standard HTTP client.

httpclient/client_configuration.go Description: This file contains functions to load and validate configuration values from a JSON file or environment variables.

httpclient/integration.go

httpmethod/httpmethod.go

httpclient/request.go

httpclient/utility.go

Index

• Constants
• type APIIntegration
• type Client
  • func (c *Client) CheckDeprecationHeader(resp *http.Response)
  • func (c *Client) DoMultiPartRequest(method, endpoint string, files map[string][]string, ...) (*http.Response, error)
  • func (c *Client) DoRequest(method, endpoint string, body, out interface{}) (*http.Response, error)
• type ClientConfig
  • func LoadConfigFromEnv() (*ClientConfig, error)
  • func LoadConfigFromFile(filepath string) (*ClientConfig, error)
  • func (c *ClientConfig) Build() (*Client, error)
  • func (c *ClientConfig) SetDefaultValuesClientConfig()
• type HTTPExecutor
• type MockExecutor
  • func (m *MockExecutor) CloseIdleConnections()
  • func (m *MockExecutor) Cookies(*url.URL) []*http.Cookie
  • func (m *MockExecutor) Do(req *http.Request) (*http.Response, error)
  • func (m *MockExecutor) Get(_ string) (*http.Response, error)
  • func (m *MockExecutor) Head(_ string) (*http.Response, error)
  • func (m *MockExecutor) Post(_ string, _ string, _ io.Reader) (*http.Response, error)
  • func (m *MockExecutor) PostForm(_ string, _ url.Values) (*http.Response, error)
  • func (m *MockExecutor) SetCookieJar(jar http.CookieJar)
  • func (m *MockExecutor) SetCookies(url *url.URL, cookies []*http.Cookie)
  • func (m *MockExecutor) SetCustomTimeout(time.Duration)
  • func (m *MockExecutor) SetRedirectPolicy(*func(req *http.Request, via []*http.Request) error)
• type ProdExecutor
  • func (c *ProdExecutor) Cookies(url *url.URL) []*http.Cookie
  • func (c *ProdExecutor) SetCookieJar(jar http.CookieJar)
  • func (c *ProdExecutor) SetCookies(url *url.URL, cookies []*http.Cookie)
  • func (c *ProdExecutor) SetCustomTimeout(timeout time.Duration)
  • func (c *ProdExecutor) SetRedirectPolicy(policy *func(req *http.Request, via []*http.Request) error)
• type UploadState

Constants

const (
	DefaultLogLevelString              = "LogLevelInfo"
	DefaultLogOutputFormatString       = "pretty"
	DefaultLogConsoleSeparator         = "	"
	DefaultLogExportPath               = "/defaultlogs"
	DefaultMaxRetryAttempts            = 3
	DefaultMaxConcurrentRequests       = 1
	DefaultExportLogs                  = false
	DefaultHideSensitiveData           = false
	DefaultEnableDynamicRateLimiting   = false
	DefaultCustomTimeout               = 5 * time.Second
	DefaultTokenRefreshBufferPeriod    = 2 * time.Minute
	DefaultTotalRetryDuration          = 5 * time.Minute
	DefaultEnableConcurrencyManagement = false
)

const ConfigFileExtension = ".json"

Types

type APIIntegration

type APIIntegration interface {
	GetFQDN() string
	ConstructURL(endpoint string) string
	GetAuthMethodDescriptor() string
	CheckRefreshToken() error
	PrepRequestParamsAndAuth(req *http.Request) error
	PrepRequestBody(body interface{}, method string, endpoint string) ([]byte, error)
	MarshalMultipartRequest(fields map[string]string, files map[string]string) ([]byte, string, error)
	GetSessionCookies() ([]*http.Cookie, error)
}

APIIntegration is an interface that defines the methods required for an API integration. These are obtained from go-api-http-client-integrations. The methods defined in this interface are used by the HTTP client to authenticate and prepare requests for the API.

type Client

type Client struct {
	Integration *APIIntegration

	Sugar       *zap.SugaredLogger
	Concurrency *concurrency.ConcurrencyHandler
	// contains filtered or unexported fields
}

Master struct/object

func (*Client) CheckDeprecationHeader

func (c *Client) CheckDeprecationHeader(resp *http.Response)

CheckDeprecationHeader checks the response headers for the Deprecation header and logs a warning if present.

func (*Client) DoMultiPartRequest

func (c *Client) DoMultiPartRequest(method, endpoint string, files map[string][]string, formDataFields map[string]string, fileContentTypes map[string]string, formDataPartHeaders map[string]http.Header, out interface{}) (*http.Response, error)

Parameters:

• method: A string representing the HTTP method to be used for the request. This method should be either POST or PUT as these are the only methods that support multipart/form-data requests.
• endpoint: The target API endpoint for the request. This should be a relative path that will be appended to the base URL configured for the HTTP client.
• files: A map where the key is the field name and the value is a slice of file paths to be included in the request.
• formDataFields: A map of additional form fields to be included in the multipart request, where the key is the field name and the value is the field value.
• fileContentTypes: A map specifying the content type for each file part. The key is the field name and the value is the content type (e.g., "image/jpeg").
• formDataPartHeaders: A map specifying custom headers for each part of the multipart form data. The key is the field name and the value is an http.Header containing the headers for that part.
• out: A pointer to an output variable where the response will be deserialized. This should be a pointer to a struct that matches the expected response schema.

Returns:

• *http.Response: The HTTP response received from the server. In case of successful execution, this response contains the status code, headers, and body of the response. In case of errors, this response may contain the last received HTTP response that led to the failure.
• error: An error object indicating failure during request execution. This could be due to network issues, server errors, or a failure in request serialization/deserialization.

Usage: This function is suitable for executing multipart/form-data HTTP requests, particularly for file uploads along with additional form fields. It ensures proper authentication, sets necessary headers, and logs the process for debugging and monitoring purposes. Example: var result MyResponseType resp, err := client.DoMultiPartRequest("POST", "/api/upload", files, formDataFields, fileContentTypes, formDataPartHeaders, &result)

if err != nil {
    // Handle error
}

// Use `result` or `resp` as needed

func (*Client) DoRequest

func (c *Client) DoRequest(method, endpoint string, body, out interface{}) (*http.Response, error)

DoRequest constructs and executes an HTTP request based on the provided method, endpoint, request body, and output variable. This function serves as a dispatcher, deciding whether to execute the request with or without retry logic based on the idempotency of the HTTP method. Idempotent methods (GET, PUT, DELETE) are executed with retries to handle transient errors and rate limits, while non-idempotent methods (POST, PATCH) are executed without retries to avoid potential side effects of duplicating non-idempotent operations. The function uses an instance of a logger implementing the logger.Logger interface, used to log informational messages, warnings, and errors encountered during the execution of the request. It also applies redirect handling to the client if configured, allowing the client to follow redirects up to a maximum number of times. Parameters:

• method: A string representing the HTTP method to be used for the request. This method determines the execution path and whether the request will be retried in case of failures.
• endpoint: The target API endpoint for the request. This should be a relative path that will be appended to the base URL configured for the HTTP client.
• body: The payload for the request, which will be serialized into the request body. The serialization format (e.g., JSON, XML) is determined by the content-type header and the specific implementation of the API handler used by the client.
• out: A pointer to an output variable where the response will be deserialized. The function expects this to be a pointer to a struct that matches the expected response schema.

Returns:

• *http.Response: The HTTP response received from the server. In case of successful execution, this response contains the status code, headers, and body of the response. In case of errors, particularly after exhausting retries for idempotent methods, this response may contain the last received HTTP response that led to the failure.
• error: An error object indicating failure during request execution. This could be due to network issues, server errors, or a failure in request serialization/deserialization. For idempotent methods, an error is returned if all retries are exhausted without success.

Usage: This function is the primary entry point for executing HTTP requests using the client. It abstracts away the details of request retries, serialization, and response handling, providing a simplified interface for making HTTP requests. It is suitable for a wide range of HTTP operations, from fetching data with GET requests to submitting data with POST requests. Example: var result MyResponseType resp, err := client.DoRequest("GET", "/api/resource", nil, &result, logger)

if err != nil {
    // Handle error
}

// Use `result` or `resp` as needed Note:

• The caller is responsible for closing the response body when not nil to avoid resource leaks.
• The function ensures concurrency control by managing concurrency tokens internally, providing safe concurrent operations within the client's concurrency model.
• The decision to retry requests is based on the idempotency of the HTTP method and the client's retry configuration, including maximum retry attempts and total retry duration.

type ClientConfig

type ClientConfig struct {
	// Interface which implements the APIIntegration patterns. Integration handles all server/endpoint specific configuration, auth and vars.
	Integration APIIntegration

	// Sugar is the logger from Zap.
	Sugar *zap.SugaredLogger

	// Wether or not empty values will be set or an error thrown for missing items.
	PopulateDefaultValues bool

	// HideSenitiveData controls if sensitive data will be visible in logs. Debug option which should be True in production use.
	HideSensitiveData bool `json:"hide_sensitive_data"`

	// CustomCookies allows implementation of persistent, session wide cookies.
	CustomCookies []*http.Cookie

	// MaxRetry Attempts limits the amount of retries the client will perform on requests which are deemd retriable.
	MaxRetryAttempts int `json:"max_retry_attempts"`

	// MaxConcurrentRequests limits the amount of Semaphore tokens available to the client and therefor limits concurrent requests.
	MaxConcurrentRequests int `json:"max_concurrent_requests"`

	// EnableDynamicRateLimiting // TODO because I don't know.
	EnableDynamicRateLimiting bool `json:"enable_dynamic_rate_limiting"`

	// CustomTimeout // TODO also because I don't know.
	CustomTimeout time.Duration

	// TokenRefreshBufferPeriod is the duration of time before the token expires in which it's deemed
	// more sensible to replace the token rather then carry on using it.
	TokenRefreshBufferPeriod time.Duration

	// TotalRetryDuration // TODO maybe this should be called context?
	TotalRetryDuration time.Duration

	// EnableCustomRedirectLogic allows the client to follow redirections when they're returned from a request.
	// Toggleable for debug reasons only
	CustomRedirectPolicy *func(req *http.Request, via []*http.Request) error

	// MaxRedirects is the maximum amount of redirects the client will follow before throwing an error.
	MaxRedirects int `json:"max_redirects"`

	// EnableConcurrencyManagement when false bypasses any concurrency management to allow for a simpler request flow.
	EnableConcurrencyManagement bool `json:"enable_concurrency_management"`

	// MandatoryRequestDelay is a short, usually sub 0.5 second, delay after every request as to not overwhelm an endpoint.
	// Can be set to nothing if you want to be lightning fast!
	MandatoryRequestDelay time.Duration

	// RetryEligiableRequests when false bypasses any retry logic for a simpler request flow.
	RetryEligiableRequests bool `json:"retry_eligiable_requests"`

	HTTPExecutor HTTPExecutor
}

Options/Variables for Client

func LoadConfigFromEnv

func LoadConfigFromEnv() (*ClientConfig, error)

LoadConfigFromEnv loads HTTP client configuration settings from environment variables. If any environment variables are not set, the default values defined in the constants are used instead.

func LoadConfigFromFile

func LoadConfigFromFile(filepath string) (*ClientConfig, error)

LoadConfigFromFile loads http client configuration settings from a JSON file.

func (*ClientConfig) Build

func (c *ClientConfig) Build() (*Client, error)

BuildClient creates a new HTTP client with the provided configuration.

func (*ClientConfig) SetDefaultValuesClientConfig

func (c *ClientConfig) SetDefaultValuesClientConfig()

SetDefaultValuesClientConfig sets default values for the client configuration. Ensuring that all fields have a valid or minimum value.

type HTTPExecutor

type HTTPExecutor interface {

	// Inherited
	CloseIdleConnections()
	Do(req *http.Request) (*http.Response, error)
	Get(url string) (resp *http.Response, err error)
	Head(url string) (resp *http.Response, err error)
	Post(url string, contentType string, body io.Reader) (resp *http.Response, err error)
	PostForm(url string, data url.Values) (resp *http.Response, err error)

	// Additional
	SetCookieJar(jar http.CookieJar)
	SetCookies(url *url.URL, cookies []*http.Cookie)
	SetCustomTimeout(time.Duration)
	Cookies(*url.URL) []*http.Cookie
	SetRedirectPolicy(*func(req *http.Request, via []*http.Request) error)
}

HTTPExecutor is an interface which wraps http.Client to allow mocking.

type MockExecutor

type MockExecutor struct {
	LockedResponseCode int
	ResponseBody       string
}

MockExecutor implements the same function pattern above but allows controllable responses for mocking/testing

func (*MockExecutor) CloseIdleConnections

func (m *MockExecutor) CloseIdleConnections()

CloseIdleConnections does nothing.

func (*MockExecutor) Cookies

func (m *MockExecutor) Cookies(*url.URL) []*http.Cookie

Cookies does nothing yet

func (*MockExecutor) Do

func (m *MockExecutor) Do(req *http.Request) (*http.Response, error)

Do returns a http.Response with controllable body and status code.

func (*MockExecutor) Get

func (m *MockExecutor) Get(_ string) (*http.Response, error)

Get returns Do

func (*MockExecutor) Head

func (m *MockExecutor) Head(_ string) (*http.Response, error)

Head returns Do

func (*MockExecutor) Post

func (m *MockExecutor) Post(_ string, _ string, _ io.Reader) (*http.Response, error)

Post returns Do

func (*MockExecutor) PostForm

func (m *MockExecutor) PostForm(_ string, _ url.Values) (*http.Response, error)

PostForm returns Do

func (*MockExecutor) SetCookieJar

func (m *MockExecutor) SetCookieJar(jar http.CookieJar)

SetCookieJar does nothing yet

func (*MockExecutor) SetCookies

func (m *MockExecutor) SetCookies(url *url.URL, cookies []*http.Cookie)

SetCookies does nothing yet

func (*MockExecutor) SetCustomTimeout

func (m *MockExecutor) SetCustomTimeout(time.Duration)

SetCustomTimeout does nothing yet

func (*MockExecutor) SetRedirectPolicy

func (m *MockExecutor) SetRedirectPolicy(*func(req *http.Request, via []*http.Request) error)

SetRedirectPolicy does nothing

type ProdExecutor

type ProdExecutor struct {
	*http.Client
}

ProdExecutor wraps http.Client and implements functions to adjust some of it's attrs

func (*ProdExecutor) Cookies

func (c *ProdExecutor) Cookies(url *url.URL) []*http.Cookie

Cookies wraps http.Client.Jar.Cookies()

func (*ProdExecutor) SetCookieJar

func (c *ProdExecutor) SetCookieJar(jar http.CookieJar)

SetCookieJar func to wrap c.Jar = jar

func (*ProdExecutor) SetCookies

func (c *ProdExecutor) SetCookies(url *url.URL, cookies []*http.Cookie)

SetCookies wraps http.Client.Jar.SetCookies

func (*ProdExecutor) SetCustomTimeout

func (c *ProdExecutor) SetCustomTimeout(timeout time.Duration)

SetCustomTimeout wraps http.Client.Timeout = timeout

func (*ProdExecutor) SetRedirectPolicy

func (c *ProdExecutor) SetRedirectPolicy(policy *func(req *http.Request, via []*http.Request) error)

SetRedirectPolicy wraps http.Client.Jar.CheckRedirect =

type UploadState

type UploadState struct {
	LastUploadedByte int64
	sync.Mutex
}

UploadState represents the state of an upload operation, including the last uploaded byte. This struct is used to track the progress of file uploads for resumable uploads and to resume uploads from the last uploaded byte.