fetcher

Documentation

Overview

Package fetcher provides means for plugins and engines to fetch resources with generic references. Hence, the format for reference is consistent across plugins and engines, and we can re-use the download logic.

Index

• Variables
• func FetchAsStream(context Context, reference Reference, target StreamHandler) error
• func IsBrokenReferenceError(err error) bool
• type BrokenReferenceError
  • func (e BrokenReferenceError) Error() string
• type Context
  • func WithCancel(ctx Context) (Context, func())
• type Fetcher
  • func Combine(fetchers ...Fetcher) Fetcher
• type File
• type FileReseter
  • func (f *FileReseter) Reset() error
• type Reference
• type StreamHandler
• type WriteReseter

Variables

var ErrStreamReset = errors.New("stream was reset by fetcher")

ErrStreamReset returned from io.Reader when the fetch process is reset

Functions

func FetchAsStream

func FetchAsStream(context Context, reference Reference, target StreamHandler) error

FetchAsStream gets a reference as a stream.

Notice that target StreamHandler may be invoked multiple times, if the connection breaks while fetching it might be retried. In which case the Context passed to the target StreamHandler will be canceled.

func IsBrokenReferenceError

func IsBrokenReferenceError(err error) bool

IsBrokenReferenceError returns true, if err is a BrokenReferenceError error.

This auxiliary function helps ensure that we type cast correctly.

Types

type BrokenReferenceError

type BrokenReferenceError struct {
	// contains filtered or unexported fields
}

BrokenReferenceError is used to communicate references that are broken. Error message shall be some human-readable explanation of why fetching failed and is expected to fail consistently.

func (BrokenReferenceError) Error

func (e BrokenReferenceError) Error() string

type Context

type Context interface {
	context.Context      // Context for aborting the fetch operation
	Queue() client.Queue // Client with credentials covering Fetcher.Scopes()
	// Print a progress report that looks somewhat like this:
	//     "Fetching <description> - <percent> %"
	// The <percent> is given as a float between 0 and 1, when formatting
	// consumers may wish to round to one decimal using "%.0f" formatting.
	// Progress reports won't be sent more than once every 5 seconds.
	Progress(description string, percent float64)
}

Context for fetching resource from a reference.

func WithCancel

func WithCancel(ctx Context) (Context, func())

WithCancel returns a Context and a cancel function similar to context.WithCancel

type Fetcher

type Fetcher interface {
	// Schema for references, should **only** match this type
	Schema() schematypes.Schema
	// NewReference returns a reference for options matching Schema.
	//
	// This method may fully or partially resolve the reference, in-order to be
	// able to return a consistent HashKey. Hence, this method may also return
	// a human-readable error message.
	// If the referenced resource doesn't exist it returns a BrokenReferenceError.
	NewReference(context Context, options interface{}) (Reference, error)
}

A Fetcher specifies a schema for references that it knows how to fetch. It also provides a method to generate a HashKey for each valid reference, as well as a list of scopes required for a task to use a reference.

var Artifact Fetcher = artifactFetcher{}

Artifact is a Fetcher for downloading from an (taskId, artifact) tuple

var Index Fetcher = indexFetcher{}

Index is a Fetcher for downloading from an (index, artifact) tuple

var URL Fetcher = urlFetcher{}

URL is Fetcher for downloading files from a URL.

var URLHash Fetcher = urlHashFetcher{}

URLHash is Fetcher for downloading files from a URL with a given hash

func Combine

func Combine(fetchers ...Fetcher) Fetcher

Combine a list of Fetchers into a single Fetcher implementation.

For this to work well, the reference schemas should all be distinct such that no reference matches more than one fetcher. If there is ambiguity the first Fetcher whose schema matches will be used.

type File

type File interface {
	Truncate(size int64) error
	io.Seeker
	io.Writer
}

File interface as implemented by *os.File

type FileReseter

type FileReseter struct {
	File
}

FileReseter implements WriteReseter for an *os.File instance

func (*FileReseter) Reset

func (f *FileReseter) Reset() error

Reset will truncate the file and seek to the beginning.

type Reference

type Reference interface {
	// Unique key for the reference.
	//
	// Useful for caching resources.
	HashKey() string

	// List of scope-sets that could grant access to the reference.
	// Tasks using a cached instance of this resources should satisfy at-least
	// one of these scope-sets.
	//
	// Returns [][]string{[]string{}} if no scopes are required.
	Scopes() [][]string

	// Fetch a reference to a target, sending progress to Context as well
	// as returning a human readable error message, if fetching fails.
	// If the referenced resource doesn't exist it returns a BrokenReferenceError.
	Fetch(context Context, target WriteReseter) error
}

A Reference to a blob that can be fetched.

type StreamHandler

type StreamHandler func(context.Context, io.Reader) error

A StreamHandler is a function that handles a stream, the stream maybe a aborted in which case io.Reader will return ErrStreamReset and the Context will be canceled.

type WriteReseter

type WriteReseter interface {
	io.Writer
	Reset() error
}

WriteReseter is a io.Writer with Reset() method that discards everything written and starts over from scratch.

This is easily implemented by wrapping os.File with FileReseter.