downloader

Documentation

Index

• func DownloadResults(ctx context.Context, publishedResults []model.PublishedResult, ...) error
• type Downloader
• type DownloaderProvider

Functions

func DownloadResults

func DownloadResults(
	ctx context.Context,
	publishedResults []model.PublishedResult,
	downloadProvider DownloaderProvider,
	settings *model.DownloaderSettings,
) error

DownloadResult downloads published results from a storage source and saves them to the specific download path. It supports downloading multiple results from different jobs and will append the logs to the global log file. This behavior is left from when we supported sharded jobs, and multiple results per job. It is not currently being user, and we can evaluate removing it in the future if we don't expose merging results from multiple jobs.

* make a temp dir * download all cids into temp dir * ensure top level output dir exists * iterate over each published result * copy stdout, stderr, exitCode * append stdout, stderr to global log * iterate over each output volume * make new folder for output volume * iterate over each result and merge files in output folder to results dir

Types

type Downloader

type Downloader interface {
	provider.Providable

	// DescribeResult provides information on the contents of the result,
	// providing a mapping between the 'path' of the contents and the
	// identifier used to fetch it (by this Downloader).
	DescribeResult(ctx context.Context, result model.PublishedResult) (map[string]string, error)

	// FetchResult fetches item and saves to disk (as per item's Target)
	FetchResult(ctx context.Context, item model.DownloadItem) error
}

type DownloaderProvider

type DownloaderProvider interface {
	provider.Provider[Downloader]
}