Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func HandleMapRequest ¶
HandleMapRequest handles a new Lambda Invoke with the "map" action This action will map as many files as it can off the SQS queue into groups of "merge" invokes for groups of files by hour
Types ¶
type FileCollectorForMerge ¶
type FileCollectorForMerge struct {
// contains filtered or unexported fields
}
FileCollectorForMerge is a worker go routine that will collect files for a particular hour.
func MakeFileCollectorForMerge ¶
func MakeFileCollectorForMerge(forHour int, forMinute int, queueURL string, channel chan *data.FileToProcessInfo, wg *sync.WaitGroup, collectorIndex int32) *FileCollectorForMerge
MakeFileCollectorForMerge creates a new Collector for Merge worker and returns it.
func (*FileCollectorForMerge) Run ¶
func (fp *FileCollectorForMerge) Run()
Run starts the FileProcessor loop, reading files off the channel
type FileTimeRangeProcessor ¶
type FileTimeRangeProcessor struct {
// contains filtered or unexported fields
}
FileTimeRangeProcessor is a worker go routine that will process files out of the "filesToProcess" channel.
func MakeFileTimeRangeProcessor ¶
func MakeFileTimeRangeProcessor(filesToProcess chan *data.FileToProcessInfo, hourlyChannels *[]chan *data.FileToProcessInfo, wg *sync.WaitGroup) *FileTimeRangeProcessor
MakeFileTimeRangeProcessor creates a new FileTimeRangeProcessor and returns it.
func (*FileTimeRangeProcessor) Run ¶
func (fp *FileTimeRangeProcessor) Run()
Run starts the FileProcessor loop, reading files off the channel
type MergeInvokeEvent ¶
type MergeInvokeEvent struct {
Action string `json:"action"`
Records []data.FileToProcessInfo `json:"records"`
ForHour int `json:"forHour"`
ForMinute int `json:"forMinute"`
}
MergeInvokeEvent describes a file to be processed by a merger
Source Files