scanner

package

v0.0.0-...-65fd79d Latest Latest Go to latest Published: Jun 13, 2022 License: MIT Imports: 4 Imported by: 0

Details

Package scanner implements a tokenizer that creates tokens of the SQLite grammar.

View Source

const (
	ErrUnexpectedToken   = Error("unexpected token")
	ErrInvalidUTF8String = Error("input is not a valid utf8 encoded string")
)

Constant errors

This section is empty.

This section is empty.

type Error string

Error allows constant errors.

func (s Error) Error() string

type Scanner interface {
	Next() token.Token
	Peek() token.Token
}

Scanner is the interface that describes a scanner.

func NewRuleBased(input string, ruleset ruleset.Ruleset) (Scanner, error)

NewRuleBased creates a new, ready to use rule based scanner with the given ruleset, that will process the given input rune slice.

Path	Synopsis
matcher Package matcher implements a matcher component, that can determine whether or not a rune is part of a custom character class.	Package matcher implements a matcher component, that can determine whether or not a rune is part of a custom character class.
ruleset Package ruleset implements rules that can be used in combination with a rule based scanner.	Package ruleset implements rules that can be used in combination with a rule based scanner.
token Package token implements tokens that are produced by the scanner.	Package token implements tokens that are produced by the scanner.