stateful

package
v0.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 26, 2020 License: MIT Imports: 14 Imported by: 9

Documentation

Overview

Package stateful defines a nested stateful lexer.

This lexer is based heavily on the approach used by Chroma (and Pygments).

The lexer is a state machine defined by a map of rules keyed by state. Each rule is a named regex and optional operation to apply when the rule matches.

As a convenience, any Rule starting with a lowercase letter will be elided from output.

Lexing starts in the "Root" group. Each rule is matched in order, with the first successful match producing a lexeme. If the matching rule has an associated Action it will be executed. The name of each non-root rule is prefixed with the name of its group to yield the token identifier used during matching.

A state change can be introduced with the Action `Push(state)`. `Pop()` will return to the previous state.

To reuse rules from another state, use `Include(state)`.

As a special case, regexes containing backrefs in the form \N (where N is a digit) will match the corresponding capture group from the immediate parent group. This can be used to parse, among other things, heredocs.

See the example and tests in this package for details.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Action added in v0.7.0

type Action interface {
	// contains filtered or unexported methods
}

A Action is applied when a rule matches.

func Pop

func Pop() Action

Pop to the previous state.

func Push

func Push(state string) Action

Push to the given state.

The target state will then be the set of rules used for matching until another Push or Pop is encountered.

type ActionFunc added in v0.7.0

type ActionFunc func(*Lexer, []string) error

ActionFunc is a function that is also a Action.

type Definition

type Definition struct {
	// contains filtered or unexported fields
}

Definition is the lexer.Definition.

func New

func New(rules Rules) (*Definition, error)

New constructs a new stateful lexer from rules.

Example

An example of parsing nested expressions within strings.

type Terminal struct {
	String *String `  @@`
	Ident  string  `| @Ident`
}

type Expr struct {
	Left  *Terminal `@@`
	Op    string    `( @Oper`
	Right *Terminal `  @@)?`
}

type Fragment struct {
	Escaped string `(  @Escaped`
	Expr    *Expr  ` | "${" @@ "}"`
	Text    string ` | @Char)`
}

type String struct {
	Fragments []*Fragment `"\"" @@* "\""`
}

def, err := New(interpolatedRules)
if err != nil {
	log.Fatal(err)
}
parser, err := participle.Build(&String{}, participle.Lexer(def))
if err != nil {
	log.Fatal(err)
}

actual := &String{}
err = parser.ParseString(`"hello ${user + "??"}"`, actual)
if err != nil {
	log.Fatal(err)
}
repr.Println(actual)
Output:

func NewSimple added in v0.7.0

func NewSimple(rules []Rule) (*Definition, error)

NewSimple creates a new stateful lexer with a single "Root" state.

func (*Definition) Lex

func (d *Definition) Lex(r io.Reader) (lexer.Lexer, error)

func (*Definition) Symbols

func (d *Definition) Symbols() map[string]rune

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer implementation.

func (*Lexer) Next

func (l *Lexer) Next() (lexer.Token, error)

type Rule

type Rule struct {
	Name    string
	Pattern string
	Action  Action
}

A Rule matching input and possibly changing state.

func Include

func Include(state string) Rule

Include rules from another state in this one.

func Return added in v0.7.0

func Return() Rule

Return to the parent state.

Useful as the last rule in a sub-state.

type Rules

type Rules map[string][]Rule

Rules grouped by name.

type RulesAction added in v0.7.0

type RulesAction interface {
	// contains filtered or unexported methods
}

RulesAction is an optional interface that Actions can implement.

It is applied during rule construction to mutate the rule map.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL