Documentation
¶
Overview ¶
Package parallel implements helpers for fast processing of line oriented inputs. Basic usage example:
r := strings.NewReader("1\n2\n3\n")
f := func(ln int, b []byte) ([]byte, error) {
result := fmt.Sprintf("#%d %s", ln, string(b))
return []byte(result), nil
}
p := parallel.NewProcessor(r, os.Stdout, f)
if err := p.Run(); err != nil {
log.Fatal(err)
}
This would print out:
#0 1 #1 2 #2 3
Note that the order of the input is not guaranteed to be preserved. If you care about the exact position, utilize the originating line number passed into the transforming function.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BytesBatch ¶
type BytesBatch struct {
// contains filtered or unexported fields
}
BytesBatch is a slice of byte slices.
func NewBytesBatch ¶
func NewBytesBatch() *BytesBatch
NewBytesBatch creates a new BytesBatch with a given capacity.
func NewBytesBatchCapacity ¶
func NewBytesBatchCapacity(cap int) *BytesBatch
NewBytesBatchCapacity creates a new BytesBatch with a given capacity.
func (*BytesBatch) Size ¶
func (bb *BytesBatch) Size() int
Size returns the number of elements in the batch.
func (*BytesBatch) Slice ¶
func (bb *BytesBatch) Slice() []Record
Slice returns a slice of byte slices.
type Processor ¶
type Processor struct {
BatchSize int
RecordSeparator byte
NumWorkers int
SkipEmptyLines bool
BatchMemoryLimit int64
// contains filtered or unexported fields
}
Processor can process lines in parallel.
func NewProcessor ¶
NewProcessor creates a new line processor, which reads lines from a reader, applies a function and writes results back to a writer.
func (*Processor) RunWorkers ¶
RunWorkers allows to quickly set the number of workers.
Source Files