gidari

README

Gidari

Gidari is a library for batch querying data and persisting the results to local storage.

Installation

go get github.com/alpstable/gidari@latest

For information on using the CLI, see here.

Usage

At the moment, Gidari only supports HTTP services. There are two ways to use an HTTP service:

1. Iterate over http.Response data, for pre-defined http.Requests.
2. Define a writer to concurrently "write" response data for pre-defined http.Requests.

See the Go Docs or Web-to-Storage Examples section for usage.

Authenticating HTTP Requests

Protocol	Parameters	Description
Basic	Requires a username/email and password	The server will authorize the request only if it can validate the user-ID and password for the protection space of the Request-URI
Coinbase	Requires a key, passphrase and secrete generated in the Coinbase Exchange GUI	The Coinbase API requires a CB-ACCESS-SIGN header that is generated by creating a sha256 HMAC using the base64-decoded secret key on the prehash string from a timestamp + method + requestPath + body combination

Web-to-Storage Examples

Data Type	Writer	Example	Description
CSV	csvpb	examples/csvp	Use the HTTPService to write web API data to stdout as CSV
MongoDB Document	mongopb	exmaples/mongopb	Use the HTTPService to write web API data to a MongoDB Collection

Contributing

Follow this guide for information on contributing.

Resources

• Public REST APIs from Postman Documenter
• Go Gopher artwork by Victoria Trum
• The original Go gopher was designed by the awesome Renee French

Documentation

Overview

Package gidari provides a "web-to-storage" API for batch querying web APIs and persisting the resulting data.

Index

• Constants
• Variables
• type Client
• type Current
• type DecodeFunc
• type DecodeType
• type HTTPIteratorService
  • func NewHTTPIteratorService(svc *HTTPService) *HTTPIteratorService
  • func (iter *HTTPIteratorService) Close() error
  • func (iter *HTTPIteratorService) Err() error
  • func (iter *HTTPIteratorService) Next(ctx context.Context) bool
• type HTTPService
  • func NewHTTPService(svc *Service) *HTTPService
  • func (svc *HTTPService) Client(client Client) *HTTPService
  • func (svc *HTTPService) RateLimiter(rlimiter *rate.Limiter) *HTTPService
  • func (svc *HTTPService) Requests(reqs ...*Request) *HTTPService
  • func (svc *HTTPService) Store(ctx context.Context) error
• type ListWriter
• type Request
  • func NewHTTPRequest(req *http.Request, opts ...RequestOption) *Request
• type RequestOption
  • func WithAuth(auth func(*http.Request) (*http.Response, error)) RequestOption
  • func WithWriters(w ...ListWriter) RequestOption
• type Service
  • func NewService(ctx context.Context, opts ...ServiceOption) (*Service, error)
• type ServiceOption

Examples

• HTTPIteratorService.Next
• HTTPService.Store
• WithAuth

Constants

const Version = "0.3.0"

Version is the version of the Gidari library.

Variables

var ErrUnsupportedDecodeType = fmt.Errorf("unsupported decode type")

ErrUnsupportedDecodeType is returned when the provided decode type is not supported.

var ErrUnsupportedProtobufType = fmt.Errorf("unsupported proto type")

ErrUnsupportedProtobufType is returned when the provided proto type is not supported.

Types

type Client

type Client interface {
	Do(*http.Request) (*http.Response, error)
}

Client is an interface that wraps the "Do" method of the "net/http" package's "client" type.

type Current

type Current struct {
	Response *http.Response // HTTP response from the request.
	// contains filtered or unexported fields
}

Current is a struct that represents the most recent response by calling the "Next" method on the HTTPIteratorService.

type DecodeFunc

type DecodeFunc func(list *structpb.ListValue) error

DecodeFunc is a function that will decode the results of a request into a the target.

type DecodeType

type DecodeType int32

DecodeType is an enum that represents the type of data that is being decoded.

const (
	// DecodeTypeUnknown is the default value for the DecodeType enum.
	DecodeTypeUnknown DecodeType = iota

	// DecodeTypeJSON is used to decode JSON data.
	DecodeTypeJSON
)

type HTTPIteratorService

type HTTPIteratorService struct {

	// Current is the most recent response from the iterator. This value is
	// set and blocked by the "Next" method, updating with each iteration.
	Current *Current
	// contains filtered or unexported fields
}

HTTPIteratorService is a service that will iterate over the requests defined for the HTTPService and return the response from each request.

func NewHTTPIteratorService

func NewHTTPIteratorService(svc *HTTPService) *HTTPIteratorService

NewHTTPIteratorService will return a new HTTPIteratorService.

func (*HTTPIteratorService) Close

func (iter *HTTPIteratorService) Close() error

Close closes the iterator.

func (*HTTPIteratorService) Err

func (iter *HTTPIteratorService) Err() error

Err returns any error encountered by the iterator.

func (*HTTPIteratorService) Next

func (iter *HTTPIteratorService) Next(ctx context.Context) bool

Next will push the next response as a byte slice onto the Iterator. If there are no more responses, the returned boolean will be false. The user is responsible for decoding the response.

The HTTP requests used to define the configuration will be fetched concurrently once the "Next" method is called for the first time.

Example

package main

import (
	"context"
	"fmt"
	"io"
	"log"
	"net/http"
	"time"

	"github.com/alpstable/gidari"
	"golang.org/x/time/rate"
)

func main() {
	ctx := context.TODO()

	const api = "https://anapioficeandfire.com/api"

	// First we create a service that can be used to make bulk HTTP
	// requests to the API.
	svc, err := gidari.NewService(ctx)
	if err != nil {
		log.Fatalf("failed to create service: %v", err)
	}

	// Create some requests and add them to the service.
	charReq, _ := http.NewRequestWithContext(ctx, http.MethodGet, api+"/characters", nil)
	housReq, _ := http.NewRequestWithContext(ctx, http.MethodGet, api+"/houses", nil)

	// Wrap the HTTP Requests in the gidalri.HTTPRequest type.
	charReqWrapper := gidari.NewHTTPRequest(charReq)
	housReqWrapper := gidari.NewHTTPRequest(housReq)

	// Add the wrapped HTTP requests to the HTTP Service.
	svc.HTTP.Requests(charReqWrapper, housReqWrapper)

	// Add a rate limiter to the service, 5 requests per second. This can
	// help avoid "429" errors.
	svc.HTTP.RateLimiter(rate.NewLimiter(rate.Every(1*time.Second), 5))

	// byteSize will keep track of the sum of bytes for each HTTP Response's
	// body.
	var byteSize int

	for svc.HTTP.Iterator.Next(ctx) {
		current := svc.HTTP.Iterator.Current

		rsp := current.Response
		if rsp == nil {
			break
		}

		// Get the byte slice from the response body.
		body, err := io.ReadAll(current.Response.Body)
		if err != nil {
			log.Fatalf("failed to read response body: %v", err)
		}

		// Add the number of bytes to the sum.
		byteSize += len(body)
	}

	// Check to see if an error occurred.
	if err := svc.HTTP.Iterator.Err(); err != nil {
		log.Fatalf("failed to iterate over HTTP responses: %v", err)
	}

	fmt.Println("Total number of bytes:", byteSize)
}

Output:

Total number of bytes: 10455

type HTTPService

type HTTPService struct {

	// Iterator is a service that provides the functionality to
	// asynchronously iterate over a set of requests, handling them with a
	// custom handler. Each response in the request is achieved by calling
	// the Iterator's "Next" method, returning the "http.Response" object
	// defined by the "net/http" package.
	Iterator *HTTPIteratorService
	// contains filtered or unexported fields
}

HTTPService is used process response data from requests sent to an HTTP client. "Processing" includes upserting data into a database, or concurrently iterating over the response data using a "Next" pattern.

func NewHTTPService

func NewHTTPService(svc *Service) *HTTPService

NewHTTPService will create a new HTTPService.

func (*HTTPService) Client

func (svc *HTTPService) Client(client Client) *HTTPService

Client sets the optional client to be used by the service. If no client is set, the default "http.DefaultClient" defined by the "net/http" package will be used.

func (*HTTPService) RateLimiter

func (svc *HTTPService) RateLimiter(rlimiter *rate.Limiter) *HTTPService

RateLimiter sets the optional rate limiter for the service. A rate limiter will limit the request to a set of bursts per period, avoiding 429 errors.

func (*HTTPService) Requests

func (svc *HTTPService) Requests(reqs ...*Request) *HTTPService

Requests sets the option requests to be made by the service to the client. If no client has been set for the service, the default "http.DefaultClient" defined by the "net/http" package will be used.

func (*HTTPService) Store

func (svc *HTTPService) Store(ctx context.Context) error

Store will concurrently make the requests to the client and store the data from the responses in the provided storage. If no storage is provided, then the data will be discarded.

Example

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/alpstable/gidari"
	"golang.org/x/time/rate"
	"google.golang.org/protobuf/types/known/structpb"
)

type ExampleWriter struct {
	lists []*structpb.ListValue
}

func (w *ExampleWriter) Write(ctx context.Context, list *structpb.ListValue) error {
	w.lists = append(w.lists, list)

	return nil
}

func main() {
	ctx := context.TODO()

	const api = "https://anapioficeandfire.com/api"

	// First we create a service that can be used to make bulk HTTP
	// requests to the API.
	svc, err := gidari.NewService(ctx)
	if err != nil {
		log.Fatalf("failed to create service: %v", err)
	}

	// Create some HTTP Requests.
	charReq, _ := http.NewRequestWithContext(ctx, http.MethodGet, api+"/characters", nil)
	housReq, _ := http.NewRequestWithContext(ctx, http.MethodGet, api+"/houses", nil)

	// Create a writer to write the data.
	w := &ExampleWriter{}

	// Wrap the HTTP Requests in the gidalri.HTTPRequest type.
	charReqWrapper := gidari.NewHTTPRequest(charReq, gidari.WithWriters(w))
	housReqWrapper := gidari.NewHTTPRequest(housReq, gidari.WithWriters(w))

	// Add the wrapped HTTP requests to the HTTP Service.
	svc.HTTP.Requests(charReqWrapper, housReqWrapper)

	// Add a rate limiter to the service, 5 requests per second. This can
	// help avoid "429" errors.
	svc.HTTP.RateLimiter(rate.NewLimiter(rate.Every(1*time.Second), 5))

	// Upsert the responses to the database.
	if err := svc.HTTP.Store(ctx); err != nil {
		log.Fatalf("failed to upsert HTTP responses: %v", err)
	}

	// Print the result of the mock writer.
	for _, list := range w.lists {
		fmt.Println("list size: ", len(list.Values))
	}

}

Output:

list size:  10
list size:  10

type ListWriter

type ListWriter interface {
	Write(cxt context.Context, list *structpb.ListValue) error
}

ListWriter is use to write data to io, storage, whatever, from a list of structpb.Values.

type Request

type Request struct {
	// contains filtered or unexported fields
}

Request represents a request to be made by the service to the client. This object wraps the "net/http" package request object.

func NewHTTPRequest

func NewHTTPRequest(req *http.Request, opts ...RequestOption) *Request

NewHTTPRequest will create a new HTTP request.

type RequestOption

type RequestOption func(*Request)

RequestOption is used to set an option on a request.

func WithAuth

func WithAuth(auth func(*http.Request) (*http.Response, error)) RequestOption

WithAuth will set a round tripper to be used by the service to authenticate the request during the http transport.

Example

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"os"

	"github.com/alpstable/gidari"
	"github.com/alpstable/gidari/auth"
)

func main() {
	ctx := context.TODO()

	const api = "https://api-public.sandbox.exchange.coinbase.com"

	key := os.Getenv("COINBASE_API_KEY")
	secret := os.Getenv("COINBASE_API_SECRET")
	passphrase := os.Getenv("COINBASE_API_PASSPHRASE")

	// First we create a service that can be used to make bulk HTTP
	// requests to the API.
	svc, err := gidari.NewService(ctx)
	if err != nil {
		log.Fatalf("failed to create service: %v", err)
	}

	// Set a round tripper that will sign the requests.
	roundTripper := auth.NewCoinbaseRoundTrip(key, secret, passphrase)
	withAuth := gidari.WithAuth(roundTripper)

	// Create some requests and add them to the service.
	accounts, _ := http.NewRequestWithContext(ctx, http.MethodGet, api+"/accounts", nil)
	currencies, _ := http.NewRequestWithContext(ctx, http.MethodGet, api+"/currencies", nil)

	// Wrap the HTTP Requests in the gidalri.HTTPRequest type.
	accountsW := gidari.NewHTTPRequest(accounts, withAuth)
	currenciesW := gidari.NewHTTPRequest(currencies, withAuth)

	// Add the wrapped HTTP requests to the HTTP Service.
	svc.HTTP.Requests(accountsW, currenciesW)

	// Get the status code for the responses.
	for svc.HTTP.Iterator.Next(ctx) {
		current := svc.HTTP.Iterator.Current

		rsp := current.Response
		if rsp == nil {
			break
		}

		fmt.Println("status code:", rsp.Request.URL.Path, rsp.StatusCode)
	}
}

func WithWriters

func WithWriters(w ...ListWriter) RequestOption

WithWriters sets optional writers to be used by the HTTP Service upsert method to write the data from the response.

type Service

type Service struct {
	// HTTP is used for transporting and processing HTTP requests and
	// responses.
	HTTP *HTTPService
}

Service is the main service for Gidari. It is responsible for providing the services for transporting and processing data.

func NewService

func NewService(ctx context.Context, opts ...ServiceOption) (*Service, error)

NewService will create a new Service.

type ServiceOption

type ServiceOption func(*Service)

ServiceOption is a function for configuring a Service.