download

Documentation

Overview

Package download contains a library for downloading data from logs.

Package download provides a fetcher to get data via http(s).

Index

• func Bulk(ctx context.Context, first, treeSize uint64, batchFetch BatchFetch, ...)
• type BatchFetch
• type BulkResult
• type HTTPFetcher
  • func NewHTTPFetcher(prefix *url.URL) HTTPFetcher
  • func (f HTTPFetcher) GetData(path string) ([]byte, error)

Functions

func Bulk

func Bulk(ctx context.Context, first, treeSize uint64, batchFetch BatchFetch, workers, batchSize uint, rc chan<- BulkResult)

Bulk keeps downloading leaves starting from `first`, using the given leaf fetcher. The number of workers and the batch size to use for each of the fetch requests are also specified. The resulting leaves or terminal errors are returned in order over `rc`. Bulk takes ownership of `rc` and will close it when no more values will be written. Internally this uses exponential backoff on the workers to download as fast as possible, but no faster.

Types

type BatchFetch

type BatchFetch func(start uint64, leaves [][]byte) (uint64, error)

BatchFetch should be implemented to provide a mechanism to fetch a range of leaves. It should return the number of leaves fetched, or an error if the fetch failed.

type BulkResult

type BulkResult struct {
	Leaf []byte
	Err  error
}

BulkResult combines a downloaded leaf, or the error found when trying to obtain the leaf.

type HTTPFetcher

type HTTPFetcher struct {
	// contains filtered or unexported fields
}

HTTPFetcher gets the data over HTTP(S). This is thread safe.

func NewHTTPFetcher

func NewHTTPFetcher(prefix *url.URL) HTTPFetcher

NewHTTPFetcher returns an HTTPFetcher that gets paths appended to the given prefix.

func (HTTPFetcher) GetData

func (f HTTPFetcher) GetData(path string) ([]byte, error)

GetData gets the data at the given path.