exception

Documentation

Overview

package exception implements a Tokenizer which extracts pieces matched by a regular expression from the input data, delegates the rest to another tokenizer, then insert back extracted parts in the token stream. Use it to preserve sequences which a regular tokenizer would alter or remove.

Its constructor takes the following arguments:

"exceptions" ([]string): one or more Go regular expressions matching the sequence to preserve. Multiple expressions are combined with "|".

"tokenizer" (string): the name of the tokenizer processing the data not matched by "exceptions".

Index

• Constants
• func ExceptionsTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error)
• type ExceptionsTokenizer
  • func NewExceptionsTokenizer(exception *regexp.Regexp, remaining analysis.Tokenizer) *ExceptionsTokenizer
  • func (t *ExceptionsTokenizer) Tokenize(input []byte) analysis.TokenStream

Constants

const Name = "exception"

Functions

func ExceptionsTokenizerConstructor

func ExceptionsTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error)

Types

type ExceptionsTokenizer

type ExceptionsTokenizer struct {
	// contains filtered or unexported fields
}

func NewExceptionsTokenizer

func NewExceptionsTokenizer(exception *regexp.Regexp, remaining analysis.Tokenizer) *ExceptionsTokenizer

func (*ExceptionsTokenizer) Tokenize

func (t *ExceptionsTokenizer) Tokenize(input []byte) analysis.TokenStream