job

Documentation

Index

• Constants
• type Aggregation
  • func NewAggregation(id string, limit int, proxy string) (*Aggregation, error)
  • func (a *Aggregation) UnmarshalJSON(b []byte) error
• type Callback
  • func (cb *Callback) Bytes() ([]byte, error)
• type Job
  • func (j *Job) CallbackInfo(downloadURL url.URL) (Callback, error)
  • func (j *Job) Path() string
  • func (j Job) String() string
  • func (j *Job) UnmarshalJSON(b []byte) error
• type State
  • func (s State) MarshalBinary() (data []byte, err error)

Constants

const (
	StatePending    = "Pending"
	StateFailed     = "Failed"
	StateSuccess    = "Success"
	StateInProgress = "InProgress"
)

The available states of a job's DownloadState/CallbackState.

Types

type Aggregation

type Aggregation struct {
	ID string `json:"aggr_id"`

	// Maximum numbers of concurrent download requests
	Limit int `json:"aggr_limit"`

	// Proxy url for the client to use, optional
	Proxy string `json:"aggr_proxy"`
}

Aggregation is the concept through which the rate limit rules are defined and enforced.

func NewAggregation

func NewAggregation(id string, limit int, proxy string) (*Aggregation, error)

NewAggregation creates an aggregation with the provided ID and limit. If any of the prerequisites fail, an error is returned.

func (*Aggregation) UnmarshalJSON

func (a *Aggregation) UnmarshalJSON(b []byte) error

UnmarshalJSON populates the aggregation with the values in the provided JSON.

type Callback

type Callback struct {
	// Success refers to whether a job download was successful or not
	Success bool `json:"success"`

	// Error contains errors that occured during a job download
	Error string `json:"error"`

	// Extra are opaque/pass through data
	Extra string `json:"extra"`

	// ResourceURL is the url of the requested for download resource
	ResourceURL string `json:"resource_url"`

	// DownlaodURL the url where the downloaded resource resides
	DownloadURL string `json:"download_url"`

	// JobID is the unique id of a Job
	JobID string `json:"job_id"`

	// ResponseCode is the http response for the downloaded resource e.g 200, 404
	ResponseCode int `json:"response_code"`

	// Delivered signifies where the callback has been delivered or not
	Delivered bool `json:"delivered"`

	// DeliveryError contains the error occured while delivering a callback
	DeliveryError string `json:"delivery_error"`
}

Callback holds info to be posted back to the provided callback destination.

func (*Callback) Bytes

func (cb *Callback) Bytes() ([]byte, error)

Bytes returns a byte slice for a callback info encoded as JSON

type Job

type Job struct {
	// Auto-generated
	ID string `json:"-"`

	// The URL pointing to the resource to be downloaded
	URL string `json:"url"`

	// AggrID is the ID of the aggregation the job belongs to.
	//
	// TODO: should this be a pointer to an Aggregation? If so, then
	// NewJob should be a function on Aggregation
	AggrID string `json:"aggr_id"`

	DownloadState State `json:"-"`

	// How many times the download request was attempted
	DownloadCount int `json:"-"`

	// Auxiliary ad-hoc information. Typically used for communicating
	// download errors back to the user.
	DownloadMeta string `json:"-"`

	CallbackState State  `json:"-"`
	CallbackType  string `json:"callback_type"`
	CallbackDst   string `json:"callback_dst"`
	// TODO: Remove CallbackURL, in favor of CallbackType and CallbackDst, after
	// all users of Downloader have upgraded their request scheme.
	CallbackURL string `json:"callback_url"`

	// Auxiliary ad-hoc information used for debugging.
	CallbackMeta string `json:"-"`

	// How many times the callback request was attempted
	CallbackCount int `json:"-"`

	// Arbitrary info provided by the user that are posted
	// back during the callback
	Extra string `json:"extra"`

	// Response code of the download request
	ResponseCode int `json:"response_code"`

	// Mime type pattern provided by the client
	MimeType string `json:"mime_type"`

	// Http client timeout for download in seconds
	DownloadTimeout int `json:"download_timeout"`

	// The HTTP request headers provided by the user will be used for
	// downloading files.
	// This attribute is optional.
	RequestHeaders map[string]string `json:"request_headers,omitempty"`
}

Job represents a user request for downloading a resource.

It is the core entity of the downloader and holds all info and state of the download.

TODO: should this be valid with an empty aggregation id?

func (*Job) CallbackInfo

func (j *Job) CallbackInfo(downloadURL url.URL) (Callback, error)

CallbackInfo validates the state of a job and returns a callback info along with an error if appropriate. The expected argument downloadURL is the base path of a downloaded resource in the downloader.

func (*Job) Path

func (j *Job) Path() string

Path returns the relative job path

func (Job) String

func (j Job) String() string

func (*Job) UnmarshalJSON

func (j *Job) UnmarshalJSON(b []byte) error

UnmarshalJSON is used to populate a job from the values in the provided JSON message.

type State

type State string

State represents the download & callback states. For valid values see constants below.

func (State) MarshalBinary

func (s State) MarshalBinary() (data []byte, err error)

MarshalBinary is used by redis driver to marshall custom type State