go-bulkrequest

module
v0.0.0-...-725b82a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 30, 2022 License: MIT

README

go-bulkrequest


Description

go-bulkrequest is a simple library for fetching and parsing data from multiple URLs/endpoints.

Project is developed in go.

Every bulk request is highly configurable, for example:

  • You can use proxy connection instead of direct connection, for every proxy, requests will be parallelized.
  • You can set cookies/headers/user-agent for every request.
  • You can provide custom parser for every request.

Installation

go get github.com/jodua/go-bulkrequest/bulkrequest

Usage

Importing library
import "github.com/jodua/go-bulkrequest/bulkrequest"
Creating bulk request builder
bulkRequest := bulkrequest.NewBulkRequest()
Customizing bulk request

Builder methods

SetBaseUrl - sets base URL for all requests.

Params:

  • baseUrl string - baseUrl
bulkRequest.SetBaseUrl("https://example.com")

SetUrls - sets list of URL suffixes for all requests.

Params:

  • urls []string - list of URL suffixes

Following example will create requests for URLs:

  • Base URL + Urls[0]
  • Base URL + Urls[1]
  • ...
  • Base URL + Urls[len(Urls)-1]
urlList := []string{"1", "2", "3", "4", "5"}
bulkRequest.SetUrls(urlList)

SetTimeout - sets timeout after which request will be aborted.

Params:

  • timeout time.Duration - duration of timeout
bulkRequest.SetTimeout(time.Second * 10)

AddHeader - adds header to all requests.

Params:

  • key string - header key
  • value string - header value
bulkRequest.AddHeader("X-Header", "Value")

AddCookie - adds cookie to all requests.

Params:

  • name string - header key
  • value string - header value
bulkRequest.AddCookie("Cookie", "Value")

SetDelayConfig - sets delay configuration for all requests.

Params:

  • delayConfig *datatypes.DelayConfig - delay configuration

Delay config is a struct that contains delay configuration for all requests. It consists of:

  • DelayMin time.Duration - minimum delay between requests
  • DelayMax time.Duration - maximum delay between requests

When fetching data from multiple URLs, requests will be delayed between DelayMin and DelayMax time.

import "github.com/jodua/go-bulkrequest/datatypes"

delayConfig := datatypes.DelayConfig{
    DelayMin: time.Second * 1,
    DelayMax: time.Second * 2,
}

bulkRequest.SetDelayConfig(&delayConfig)

SetProxyConfig - sets proxy configuration for all requests. If proxy configuration is not set, requests will be made directly.

Params:

  • proxyConfig *datatypes.ProxyConfig - proxy configuration

Proxy config is a struct that contains proxy configuration for all requests. It consists of:

  • ProxyList []string - list of proxies
  • RequestsPerProxy - number of requests that will be sent through each proxy
import "github.com/jodua/go-bulkrequest/datatypes"

proxyList := []string{"http://proxy1:1231", "http://proxy2:1111"}

proxyConfig := datatypes.ProxyConfig{
    ProxyList:        proxyList,
    RequestsPerProxy: 5,
}

bulkRequest.SetProxyConfig(&delayConfig)

SetUserAgentConfig - sets user agent configuration for all requests.

Params:

  • userAgentConfig *datatypes.UserAgentConfig - user agent configuration

User agent config is a struct that contains user agent configuration for all requests. It consists of:

  • UserAgentList []string - list of user agents
import "github.com/jodua/go-bulkrequest/datatypes" 

userAgentList := []string{"UserAgent1", "UserAgent2"}

userAgentConfig := datatypes.UserAgentConfig{
    UserAgentList: userAgentList,
}

bulkRequest.SetUserAgentConfig(&userAgentConfig)

SetParser - sets parser for all requests.

Params:

  • parser *jsonparser.JSONParser - pointer to parser object

Example parser can be found in github.com/jodua/go-bulkrequest/JSONParser/schemas package.

var parser = schemas.JSONPlaceholderTodoParser

bulkRequest.SetParser(&parser)
Building and executing bulk request
br := bulkRequest.Build()

fetch, err := br.Fetch()
if err != nil {
    // handle error
}
log.Println(fetch)

Full example can be found in main.go file.

Building JSON parser

JSONParser struct consists of:

  • JSONSchema any - pointer to JSON schema object
  • ConvertFunction func(any,string) any - function that converts JSON data to desired format, second parameter is request URL
  • ValidatorFunction func(any) error - function that validates JSON data
  • Output any - pointer to output struct that will be filled with data
  • Name string - name of parser

Example parser can be found in github.com/jodua/go-bulkrequest/JSONParser/schemas package.

Issues

File issues through Issues tab.

License

MIT License

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL