omniparser

README

omniparser

Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, txt, fixed length/width, XML, EDI/X12/EDIFACT, JSON, and custom formats) in streaming fashion and transforms data into desired JSON output based on a schema written in JSON.

Min Golang Version: 1.14

Documentation

Docs:

• Getting Started: a tutorial for writing your first omniparser schema.
• IDR: in-memory data representation of ingested data for omniparser.
• XPath Based Record Filtering and Data Extraction: xpath queries are essential to omniparser schema writing. Learn the concept and tricks in depth.
• All About Transforms: everything about transform_declarations.
• Use of custom_func, Specially javascript: An in depth look of how custom_func is used, specially the all mighty javascript (and javascript_with_context).
• CSV Schema in Depth: everything about schemas for CSV input.
• Fixed-Length Schema in Depth: everything about schemas for fixed-length (e.g. TXT) input
• JSON/XML Schema in Depth: everything about schemas for JSON or XML input.
• EDI Schema in Depth: everything about schemas for EDI input.
• Programmability: Advanced techniques for using omniparser (or some of its components) in your code.

References:

• Custom Functions: a complete reference of all built-in custom functions.

Examples:

• CSV Examples
• Fixed-Length Examples
• JSON Examples
• XML Examples.
• EDI Examples.
• Custom File Format
• Custom Funcs

In the example folders above you will find pairs of input files and their schema files. Then in the .snapshots sub directory, you'll find their corresponding output files.

Online Playground

Use https://omniparser.herokuapp.com/ (may need to wait for a few seconds for heroku instance to wake up) for trying out schemas and inputs, yours or existing samples, to see how ingestion and transform work.

Why

• No good ETL transform/parser library exists in Golang.
• Even looking into Java and other languages, choices aren't many and all have limitations:
  • Smooks is dead, plus its EDI parsing/transform is too heavyweight, needing code-gen.
  • BeanIO can't deal with EDI input.
  • Jolt can't deal with anything other than JSON input.
  • JSONata still only JSON -> JSON transform.
• Many of the parsers/transforms don't support streaming read, loading entire input into memory - not acceptable in some situations.

Requirements

• Golang 1.14 or later.

Recent Major Feature Additions/Changes

• 1.0.0 Released!
• Added Transform.RawRecord() for caller of omniparser to access the raw ingested record.
• Deprecated custom_parse in favor of custom_func (custom_parse is still usable for back-compatibility, it is just removed from all public docs and samples).
• Added NonValidatingReader EDI segment reader.
• Added fixed-length file format support in omniv21 handler.
• Added EDI file format support in omniv21 handler.
• Major restructure/refactoring
  • Upgrade omni schema version to omni.2.1 due a number of incompatible schema changes:
    • 'result_type' -> 'type'
    • 'ignore_error_and_return_empty_str -> 'ignore_error'
    • 'keep_leading_trailing_space' -> 'no_trim'
  • Changed how we handle custom functions: previously we always use strings as in param type as well as result param type. Not anymore, all types are supported for custom function in and out params.
  • Changed the way how we package custom functions for extensions: previously we collect custom functions from all extensions and then pass all of them to the extension that is used; This feels weird, now changed to only the custom functions included in a particular extension are used in that extension.
  • Deprecated/removed most of the custom functions in favor of using 'javascript'.
  • A number of package renaming.
• Added CSV file format support in omniv2 handler.
• Introduced IDR node cache for allocation recycling.
• Introduced IDR for in-memory data representation.
• Added trie based high performance times.SmartParse.
• Command line interface (one-off transform cmd or long-running http server mode).
• javascript engine integration as a custom_func.
• JSON stream parser.
• Extensibility:
  • Ability to provide custom functions.
  • Ability to provide custom schema handler.
  • Ability to customize the built-in omniv2 schema handler's parsing code.
  • Ability to provide a new file format support to built-in omniv2 schema handler.

Footnotes

• omniparser is a collaboration effort of jf-tech,Simon and Steven.

Documentation

Index

• type Extension
• type Schema
  • func NewSchema(name string, schemaReader io.Reader, exts ...Extension) (Schema, error)
• type Transform

Types

type Extension

type Extension struct {
	CreateSchemaHandler       schemahandler.CreateFunc
	CreateSchemaHandlerParams interface{}
	CustomFuncs               customfuncs.CustomFuncs
}

Extension allows user of omniparser to add new schema handlers, and/or new custom functions in addition to the builtin handlers and functions.

type Schema

type Schema interface {
	NewTransform(name string, input io.Reader, ctx *transformctx.Ctx) (Transform, error)
	Header() header.Header
	Content() []byte
}

Schema is an interface that represents an schema used by omniparser. One instance of Schema is associated with one and only one schema. The instance of Schema can be reused for ingesting and transforming multiple input files/streams, as long as they are all intended for the same schema. Each ingestion/transform, however, needs a separate instance of Transform. A Transform must not be shared and reused across different input files/streams. While the same instance of Schema can be shared across multiple threads, Transform is not multi-thread safe. All operations on it must be done within the same go routine.

func NewSchema

func NewSchema(name string, schemaReader io.Reader, exts ...Extension) (Schema, error)

NewSchema creates a new instance of Schema. Caller can use the optional Extensions for customization. NewSchema will scan through exts left to right to find the first extension with a schema handler (specified by CreateSchemaHandler field) that supports the input schema. If no ext provided or no ext with a handler that supports the schema, then NewSchema will fall back to builtin extension (currently for schema version 'omni.2.1'. If the input schema is still not supported by builtin extension, NewSchema will fail with ErrSchemaNotSupported. Each extension much be fully self-contained meaning all the custom functions it intends to use in the schemas supported by it must be included in the same extension.

type Transform

type Transform interface {
	// Read returns a JSON byte slice representing one ingested and transformed record.
	// io.EOF should be returned when input stream is completely consumed and future calls
	// to Read should always return io.EOF.
	// errs.ErrTransformFailed should be returned when a record ingestion and transformation
	// failed and such failure isn't considered fatal. Future calls to Read will attempt
	// new record ingestion and transformations.
	// Any other error returned is considered fatal and future calls to Read will always
	// return the same error.
	// Note if returned error isn't nil, then returned []byte will be nil.
	Read() ([]byte, error)
	// RawRecord returns the current raw record ingested from the input stream. If the last
	// Read call failed, or Read hasn't been called yet, it will return an error.
	RawRecord() (schemahandler.RawRecord, error)
}

Transform is an interface that represents one input stream ingestion and transform operation. An instance of a Transform must not be shared and reused among different input streams. An instance of a Transform must not be used across multiple goroutines.