downloader

Documentation

Index

• func DownloadResults(ctx context.Context, publishedResults []model.PublishedResult, ...) error
• type Downloader
• type DownloaderProvider

Functions

func DownloadResults

func DownloadResults(
	ctx context.Context,
	publishedResults []model.PublishedResult,
	downloadProvider DownloaderProvider,
	settings *model.DownloaderSettings,
) error

DownloadResult downloads published results from a storage source and saves them to the specific download path. It supports downloading multiple results from different jobs and will append the logs to the global log file. This behavior is left from when we supported sharded jobs, and multiple results per job. It is not currently being user, and we can evaluate removing it in the future if we don't expose merging results from multiple jobs. * make a temp dir * download all cids into temp dir * ensure top level output dir exists * iterate over each published result * copy stdout, stderr, exitCode * append stdout, stderr to global log * iterate over each output volume * make new folder for output volume * iterate over each result and merge files in output folder to results dir

Types

type Downloader

type Downloader interface {
	model.Providable

	// FetchResult fetches published result and saves it to downloadPath
	FetchResult(ctx context.Context, result model.PublishedResult, downloadPath string) error
}

type DownloaderProvider

type DownloaderProvider interface {
	model.Provider[model.StorageSourceType, Downloader]
}