README
¶
batch103
Simple SpringBatch like Batch Framework for Go
batch103 consists of the following components:
- JobLauncher
- Reader
- Processor
- Writer
Job Launcher
Job Launcher is main skeleton of the batch framework. Job Launcher offers one method Run(r ReaderType, p []ProcessorType, w []WriterType); which take in Reader, one/multiple Processor(s) and one/multiple Writer(s).
Reader
Reader is the component that responsible to receive data from any sources (e.g. File, Database) that you can implement. You can implement your custom reader by conforming to the Reader interface.
Below is a sample implementation of Reader (sampleReader.go):
type SampleReader struct {}
func (r *DelimitedFileReader) Read() ([]batch103.BatchData, error) {
var result []batch103.BatchData
input := []string{"Field1", "Field2", "Field3", "Field4"}
var batchData = &batch103.BatchData{}
batchData = batchData.Create(input)
result = append(result, *batchData)
return result, nil
}
Alternatively, you can also use standard file reader which has already been implemented. Below are the standard readers avaiable:
- FixedWidthFileReader
- DelimitedFileReader
Below is a sample implementation of using standard file reader in Job Launcher:
func main() {
var myJobLauncher batch103.JobLauncher
var myReader = &batch103.DelimitedFileReader{}
myReader.FileReader.SetFileName("output.csv")
myJobLauncher.Run(myReader, myProcessorList, myWriterList)
}
Processor
Processor is the component that responsible to process data received from Reader. Slice of BatchData return from Reader will be passed as a parameter to Processor. You may pass an empty slice to Job Launcher if you don't need a Processor. If Processor is required, You need to implement your custom processor by conforming to the Processor interface.
Below is a sample implementation of Reader (sampleProcessor.go):
type SampleProcessor struct {
batch103.BaseProcessor
}
func (r *SampleProcessor) Process(data []batch103.BatchData) ([]batch103.BatchData, error) {
var processRecords []batch103.BatchData
for i := 0; i < len(data); i++ {
if data[i].IsActive() {
var record = data[i].GenericData
if record[1] == "TREASURE" {
log.Println("TREASURE FOUND : ", record[0])
} else {
data[i].Reject(batch103.StgProcess)
}
processRecords = append(processRecords, data[i])
}
}
return processRecords, nil
}
Writer
Writer is the component that responsible to write the processed data received from Processor to database or files. Slice of BatchData return from Reader will be passed as a parameter to Writer. You may pass an empty slice to Job Launcher if you don't need a Writer. If Writer is required, You need to implement your custom writer by conforming to the Writer interface.
Below is a sample implementation of Writer (sampleWriter.go):
type SampleWriter struct {
batch103.BaseWriter
}
func (w *SampleWriter) Write(data []batch103.BatchData) error {
var concatenatedContent string
for i := 0; i < len(data); i++ {
if data[i].IsActive() {
var record = data[i].GenericData
concatenatedContent = concatenatedContent + record[0] + "," + record[1] + "," + record[2] + "\n"
}
}
fileContent := []byte(concatenatedContent)
err := os.WriteFile("output.csv", fileContent, 0644)
if err != nil {
return err
}
return nil
}
Documentation
¶
Index ¶
- Constants
- type AWSUtils
- type BaseProcessor
- type BaseReader
- type BaseWriter
- type BatchData
- type ConfigError
- type DBWriter
- type DelimitedFileReader
- type Field
- type Fields
- type FileReader
- type FileUtils
- type FixedWidthFileReader
- type InputError
- type JobLauncher
- type ProcessorType
- type ReaderType
- type WriterType
Constants ¶
const ( StgRead = "READ" StgProcess = "PROCESS" StgWrite = "WRITE" StatActive = "A" StatReject = "R" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AWSUtils ¶
type AWSUtils struct {
// contains filtered or unexported fields
}
func (*AWSUtils) DownloadFileFromS3Bucket ¶
type BaseProcessor ¶
type BaseProcessor struct {
// contains filtered or unexported fields
}
func (BaseProcessor) Parameters ¶
func (b BaseProcessor) Parameters() map[string]string
Parameters Getter
func (*BaseProcessor) SetParameters ¶
func (b *BaseProcessor) SetParameters(values map[string]string)
Parameters Setter
type BaseReader ¶
type BaseReader struct {
// contains filtered or unexported fields
}
func (*BaseReader) SetParameters ¶
func (b *BaseReader) SetParameters(values map[string]string)
Method under Base Reader
type BaseWriter ¶
type BaseWriter struct {
// contains filtered or unexported fields
}
func (*BaseWriter) SetParameters ¶
func (b *BaseWriter) SetParameters(values map[string]string)
type BatchData ¶
type BatchData struct {
GenericData []string
// contains filtered or unexported fields
}
type ConfigError ¶
type ConfigError struct{}
func (*ConfigError) Error ¶
func (m *ConfigError) Error() string
type DBWriter ¶
type DBWriter struct {
BaseWriter
ConnectionString string
DbType string
}
type DelimitedFileReader ¶
type DelimitedFileReader struct {
FileReader
}
func (*DelimitedFileReader) Read ¶
func (r *DelimitedFileReader) Read() ([]BatchData, error)
type FileReader ¶
type FileReader struct {
BaseReader
// contains filtered or unexported fields
}
File Reader inherits from Base Reader
func (*FileReader) GetFileName ¶
func (f *FileReader) GetFileName() string
func (*FileReader) OpenCSVFile ¶
func (f *FileReader) OpenCSVFile() (csv.Reader, error)
func (*FileReader) SetFileName ¶
func (f *FileReader) SetFileName(value string)
Method under File Reader
type FixedWidthFileReader ¶
type FixedWidthFileReader struct {
FileReader
Configuration string
}
func (*FixedWidthFileReader) Read ¶
func (r *FixedWidthFileReader) Read() ([]BatchData, error)
type InputError ¶
type InputError struct{}
func (*InputError) Error ¶
func (m *InputError) Error() string
type JobLauncher ¶
type JobLauncher struct {
// Different between lower and upper case
ParameterData map[string]string
// contains filtered or unexported fields
}
func (*JobLauncher) Run ¶
func (j *JobLauncher) Run(r ReaderType, p []ProcessorType, w []WriterType)
func (*JobLauncher) SetChunkSize ¶
func (j *JobLauncher) SetChunkSize(Size int) *JobLauncher
Chunk size property
Source Files