emitter

Documentation

Overview

Package emitter provides an interface for crawling data sources and "emitting" records.

Index

• Constants
• func ReaderWithPath(ctx context.Context, abs_path string) (io.ReadSeekCloser, error)
• func RegisterEmitter(ctx context.Context, scheme string, f EmitterInitializeFunc) error
• func Schemes() []string
• type CwdEmitter
  • func (idx *CwdEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error
• type DirectoryEmitter
  • func (idx *DirectoryEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error
• type Emitter
  • func NewCwdEmitter(ctx context.Context, uri string) (Emitter, error)
  • func NewDirectoryEmitter(ctx context.Context, uri string) (Emitter, error)
  • func NewEmitter(ctx context.Context, uri string) (Emitter, error)
  • func NewFeatureCollectionEmitter(ctx context.Context, uri string) (Emitter, error)
  • func NewFileEmitter(ctx context.Context, uri string) (Emitter, error)
  • func NewFileListEmitter(ctx context.Context, uri string) (Emitter, error)
  • func NewGeoJSONLEmitter(ctx context.Context, uri string) (Emitter, error)
  • func NewNullEmitter(ctx context.Context, uri string) (Emitter, error)
  • func NewRepoEmitter(ctx context.Context, uri string) (Emitter, error)
• type EmitterCallbackFunc
• type EmitterInitializeFunc
• type FeatureCollectionEmitter
  • func (idx *FeatureCollectionEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error
• type FileEmitter
  • func (idx *FileEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error
• type FileListEmitter
  • func (idx *FileListEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error
• type GeojsonLEmitter
  • func (idx *GeojsonLEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error
• type NullEmitter
  • func (idx *NullEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error
• type RepoEmitter
  • func (idx *RepoEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

Constants

const STDIN string = "STDIN"

STDIN is a constant value signaling that a record was read from `STDIN` and has no URI (path).

Functions

func ReaderWithPath

func ReaderWithPath(ctx context.Context, abs_path string) (io.ReadSeekCloser, error)

ReaderWithPath returns a new `io.ReadSeekCloser` instance derived from 'abs_path'.

func RegisterEmitter

func RegisterEmitter(ctx context.Context, scheme string, f EmitterInitializeFunc) error

RegisterEmitter() associates 'scheme' with 'init_func' in an internal list of avilable `Emitter` implementations.

func Schemes

func Schemes() []string

Schemes() returns the list of schemes that have been "registered".

Types

type CwdEmitter

type CwdEmitter struct {
	Emitter
	// contains filtered or unexported fields
}

CwdEmitter implements the `Emitter` interface for crawling records in the current (working) directory.

func (*CwdEmitter) WalkURI

func (idx *CwdEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

WalkURI() walks (crawls) the current directory each file (not excluded by any filters specified when `idx` was created) invokes 'index_cb'.

type DirectoryEmitter

type DirectoryEmitter struct {
	Emitter
	// contains filtered or unexported fields
}

DirectoryEmitter implements the `Emitter` interface for crawling records in a directory.

func (*DirectoryEmitter) WalkURI

func (idx *DirectoryEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

WalkURI() walks (crawls) the directory named 'uri' and for each file (not excluded by any filters specified when `idx` was created) invokes 'index_cb'.

type Emitter

type Emitter interface {
	WalkURI(context.Context, EmitterCallbackFunc, string) error
}

type Emitter is an interface for crawling data sources and "emitting" records. Data sources are assumed to be Who's On First records.

func NewCwdEmitter

func NewCwdEmitter(ctx context.Context, uri string) (Emitter, error)

NewCwdEmitter() returns a new `CwdEmitter` instance configured by 'uri' in the form of:

cwd://?{PARAMETERS}

Where {PARAMETERS} may be: * `?include=` Zero or more `aaronland/go-json-query` query strings containing rules that must match for a document to be considered for further processing. * `?exclude=` Zero or more `aaronland/go-json-query` query strings containing rules that if matched will prevent a document from being considered for further processing. * `?include_mode=` A valid `aaronland/go-json-query` query mode string for testing inclusion rules. * `?exclude_mode=` A valid `aaronland/go-json-query` query mode string for testing exclusion rules.

func NewDirectoryEmitter

func NewDirectoryEmitter(ctx context.Context, uri string) (Emitter, error)

NewDirectoryEmitter() returns a new `DirectoryEmitter` instance configured by 'uri' in the form of:

directory://?{PARAMETERS}

Where {PARAMETERS} may be: * `?include=` Zero or more `aaronland/go-json-query` query strings containing rules that must match for a document to be considered for further processing. * `?exclude=` Zero or more `aaronland/go-json-query` query strings containing rules that if matched will prevent a document from being considered for further processing. * `?include_mode=` A valid `aaronland/go-json-query` query mode string for testing inclusion rules. * `?exclude_mode=` A valid `aaronland/go-json-query` query mode string for testing exclusion rules.

func NewEmitter

func NewEmitter(ctx context.Context, uri string) (Emitter, error)

NewEmitter() returns a new `Emitter` instance derived from 'uri'. The semantics of and requirements for 'uri' as specific to the package implementing the interface.

func NewFeatureCollectionEmitter

func NewFeatureCollectionEmitter(ctx context.Context, uri string) (Emitter, error)

NewFeatureCollectionEmitter() returns a new `FeatureCollectionEmitter` instance configured by 'uri' in the form of:

featurecollection://?{PARAMETERS}

Where {PARAMETERS} may be: * `?include=` Zero or more `aaronland/go-json-query` query strings containing rules that must match for a document to be considered for further processing. * `?exclude=` Zero or more `aaronland/go-json-query` query strings containing rules that if matched will prevent a document from being considered for further processing. * `?include_mode=` A valid `aaronland/go-json-query` query mode string for testing inclusion rules. * `?exclude_mode=` A valid `aaronland/go-json-query` query mode string for testing exclusion rules.

func NewFileEmitter

func NewFileEmitter(ctx context.Context, uri string) (Emitter, error)

NewFileEmitter() returns a new `FileEmitter` instance configured by 'uri' in the form of:

file://?{PARAMETERS}

Where {PARAMETERS} may be: * `?include=` Zero or more `aaronland/go-json-query` query strings containing rules that must match for a document to be considered for further processing. * `?exclude=` Zero or more `aaronland/go-json-query` query strings containing rules that if matched will prevent a document from being considered for further processing. * `?include_mode=` A valid `aaronland/go-json-query` query mode string for testing inclusion rules. * `?exclude_mode=` A valid `aaronland/go-json-query` query mode string for testing exclusion rules.

func NewFileListEmitter

func NewFileListEmitter(ctx context.Context, uri string) (Emitter, error)

NewFileListEmitter() returns a new `FileListEmitter` instance configured by 'uri' in the form of:

file://?{PARAMETERS}

Where {PARAMETERS} may be: * `?include=` Zero or more `aaronland/go-json-query` query strings containing rules that must match for a document to be considered for further processing. * `?exclude=` Zero or more `aaronland/go-json-query` query strings containing rules that if matched will prevent a document from being considered for further processing. * `?include_mode=` A valid `aaronland/go-json-query` query mode string for testing inclusion rules. * `?exclude_mode=` A valid `aaronland/go-json-query` query mode string for testing exclusion rules.

func NewGeoJSONLEmitter

func NewGeoJSONLEmitter(ctx context.Context, uri string) (Emitter, error)

NewGeojsonLEmitter() returns a new `GeojsonLEmitter` instance configured by 'uri' in the form of:

geojsonl://?{PARAMETERS}

Where {PARAMETERS} may be: * `?include=` Zero or more `aaronland/go-json-query` query strings containing rules that must match for a document to be considered for further processing. * `?exclude=` Zero or more `aaronland/go-json-query` query strings containing rules that if matched will prevent a document from being considered for further processing. * `?include_mode=` A valid `aaronland/go-json-query` query mode string for testing inclusion rules. * `?exclude_mode=` A valid `aaronland/go-json-query` query mode string for testing exclusion rules.

func NewNullEmitter

func NewNullEmitter(ctx context.Context, uri string) (Emitter, error)

NewNullEmitter() returns a new `NullEmitter` instance configured by 'uri' in the form of:

null://

func NewRepoEmitter

func NewRepoEmitter(ctx context.Context, uri string) (Emitter, error)

NewDirectoryEmitter() returns a new `RepoEmitter` instance configured by 'uri' in the form of:

repo://?{PARAMETERS}

Where {PARAMETERS} may be: * `?include=` Zero or more `aaronland/go-json-query` query strings containing rules that must match for a document to be considered for further processing. * `?exclude=` Zero or more `aaronland/go-json-query` query strings containing rules that if matched will prevent a document from being considered for further processing. * `?include_mode=` A valid `aaronland/go-json-query` query mode string for testing inclusion rules. * `?exclude_mode=` A valid `aaronland/go-json-query` query mode string for testing exclusion rules.

type EmitterCallbackFunc

type EmitterCallbackFunc func(context.Context, string, io.ReadSeeker, ...interface{}) error

EmitterCallbackFunc is a custom function used to process individual records as they are crawled by an instance of the `Emitter` interface.

type EmitterInitializeFunc

type EmitterInitializeFunc func(context.Context, string) (Emitter, error)

type EmitterInitializeFunc is a function used to initialize an implementation of the `Emitter` interface.

type FeatureCollectionEmitter

type FeatureCollectionEmitter struct {
	Emitter
	// contains filtered or unexported fields
}

FeatureCollectionEmitter implements the `Emitter` interface for crawling features in a GeoJSON FeatureCollection record.

func (*FeatureCollectionEmitter) WalkURI

func (idx *FeatureCollectionEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

WalkURI() walks (crawls) each feature in the GeoJSON FeatureCollection found in the file identified by 'uri' and for each file (not excluded by any filters specified when `idx` was created) invokes 'index_cb'.

type FileEmitter

type FileEmitter struct {
	Emitter
	// contains filtered or unexported fields
}

FileEmitter implements the `Emitter` interface for crawling individual file records.

func (*FileEmitter) WalkURI

func (idx *FileEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

WalkURI() applies 'index_cb' to the file named 'uri'.

type FileListEmitter

type FileListEmitter struct {
	Emitter
	// contains filtered or unexported fields
}

FileListEmitter implements the `Emitter` interface for crawling records listed in a "file list" (a plain text newline-delimted list of files).

func (*FileListEmitter) WalkURI

func (idx *FileListEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

WalkURI() walks (crawls) the list of files found in 'uri' and for each file (not excluded by any filters specified when `idx` was created) invokes 'index_cb'.

type GeojsonLEmitter

type GeojsonLEmitter struct {
	Emitter
	// contains filtered or unexported fields
}

GeojsonLEmitter implements the `Emitter` interface for crawling features in a line-separated GeoJSON record.

func (*GeojsonLEmitter) WalkURI

func (idx *GeojsonLEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

WalkURI() walks (crawls) each GeoJSON feature found in the file identified by 'uri' and for each file (not excluded by any filters specified when `idx` was created) invokes 'index_cb'.

type NullEmitter

type NullEmitter struct {
	Emitter
}

NullEmitter implements the `Emitter` interface for appearing to crawl records but not doing anything.

func (*NullEmitter) WalkURI

func (idx *NullEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

WalkURI() does nothing.

type RepoEmitter

type RepoEmitter struct {
	Emitter
	// contains filtered or unexported fields
}

RepoEmitter implements the `Emitter` interface for crawling records in a Who's On First style data directory.

func (*RepoEmitter) WalkURI

func (idx *RepoEmitter) WalkURI(ctx context.Context, index_cb EmitterCallbackFunc, uri string) error

WalkURI() appends 'uri' with "data" and then walks that directory and for each file (not excluded by any filters specified when `idx` was created) invokes 'index_cb'.