collector

Documentation

Index

• Constants
• Variables
• func Collect(ctx context.Context, emitter Emitter, handleErr ErrHandler) error
• func Publish(ctx context.Context, d *processor.Document) error
• func RegisterDocumentCollector(c Collector, collectorType string) error
• type Collector
• type Emitter
• type ErrHandler

Constants

const (
	BufferChannelSize int = 1000
)

Variables

var (
	ErrCollectorOverwrite = fmt.Errorf("the document collector is being overwritten")
)

Functions

func Collect

func Collect(ctx context.Context, emitter Emitter, handleErr ErrHandler) error

Collect takes all the collectors and starts collecting artifacts after Collect is called, no calls to RegisterDocumentCollector should happen.

func Publish

func Publish(ctx context.Context, d *processor.Document) error

Publish is used by NATS JetStream to stream the documents and send them to the processor

func RegisterDocumentCollector

func RegisterDocumentCollector(c Collector, collectorType string) error

Types

type Collector

type Collector interface {
	// RetrieveArtifacts collects the documents from the collector. It emits each collected
	// document through the channel to be collected and processed by the upstream processor.
	// The function should block until all the artifacts are collected and return a nil error
	// or return an error from the collector crashing. This function can keep running and check
	// for new artifacts as they are being uploaded by polling on an interval or run once and
	// grab all the artifacts and end.
	RetrieveArtifacts(ctx context.Context, docChannel chan<- *processor.Document) error
	// Type returns the collector type
	Type() string
}

type Emitter

type Emitter func(*processor.Document) error

Emitter processes a document

type ErrHandler

type ErrHandler func(error) bool

ErrHandler processes an error and returns a boolean representing if the error was able to be gracefully handled