Documentation
¶
Overview ¶
Package job provides an http handler to serve up jobs to ETL parsers.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidDateConfig = errors.New("invalid source date configuration")
ErrInvalidDateConfig is returned if both DailyOnly and FullHistory are true for a source. Only one of these options can be true.
var ErrInvalidStartDate = errors.New("invalid start date")
ErrInvalidStartDate is returned if startDate is time.Time{}
var ErrMoreJSON = errors.New("JSON body not completely consumed")
ErrMoreJSON is returned when response from gardener has unknown fields.
var ErrNoConfiguredJobs = errors.New("no configured jobs")
ErrNoConfiguredJobs is returned when a new Job Service cannot be created because there are no configured jobs.
var ( // ErrNoDateAvailable is returned if a DateIterator does not have a date available. ErrNoDateAvailable = errors.New("no date is available") )
Functions ¶
This section is empty.
Types ¶
type DailyIterator ¶
type DailyIterator struct {
Date time.Time // The next date to process.
// contains filtered or unexported fields
}
DailyIterator tracks state and dates for "daily" processing events.
func NewDailyIterator ¶
func NewDailyIterator(delay time.Duration, saver namedSaver) *DailyIterator
NewDailyIterator creates a new DailyIterator. The delay is an additional offset after UTC midnight before the next Date is processed. The starting Date will be yesterday or the date successfully loaded from the given saver.
type DateIterator ¶
DateIterator defines the interface common to the Daily and Historical iterators.
DateIterator implementations should prioritize process continuity and reliability after restart. For example, the sequence of operations should roughly follow: filter, save, read, update, return. By saving the current date before updating, we ensure that a mid-process restart would pickup where it left off, possibly at the expense of some reprocessing, but with assurance that no date data was skipped.
type HistoricalIterator ¶
type HistoricalIterator struct {
Date time.Time // The next date to process.
// contains filtered or unexported fields
}
HistoricalIterator tracks state and dates for "historical" processing events.
func NewHistoricalIterator ¶
func NewHistoricalIterator(start time.Time, saver namedSaver) *HistoricalIterator
NewHistoricalIterator creates a new HistoricalIterator. The iterator will begin from the given start unless a different date is successfully loaded from the given saver. The returned HistoricalIterator.Date is guaranteed to be later or equal to the start date.
type JobIterator ¶
type JobIterator struct {
// contains filtered or unexported fields
}
JobIterator iterates over Job configurations for an underlying DateIterator sequence.
func NewJobIterator ¶
func NewJobIterator(date DateIterator, jobSpecs []tracker.JobWithTarget) *JobIterator
NewJobIterator creates a new JobIterator. For every date returned by the DateIterator, the JobIterator will enumerate every jobSpec for that date before advancing to the next date.
func (*JobIterator) Next ¶
func (j *JobIterator) Next() (*tracker.JobWithTarget, error)
Next returns the next JobWithTarget for the current date. Next may return an error when the underlying DateIterator returns an error, e.g. ErrNoDateAvailable.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service contains all information needed to provide a job service. It iterates through successive dates, processing that date from all TypeSources in the source bucket.
func NewJobService ¶
func NewJobService(startDate time.Time, sources []config.SourceConfig, statsClient stiface.Client, dailySaver namedSaver, histSaver namedSaver, ) (*Service, error)
NewJobService creates the default job service.
Source Files