parser

Documentation

Overview

Package parser defines generic LL(*) parser.

Index

• Constants
• type Hooks
• type NodeHook
• type NodeHookInstance
• type NodeHooks
• type ParseContext
  • func (pc *ParseContext) EmitToken(t *Token) error
• type Parser
  • func New(g *grammar.Grammar) (*Parser, error)
  • func (p *Parser) Parse(q *source.Queue, hs *Hooks) (result any, e error)
  • func (p *Parser) ParseString(name, content string, hs *Hooks) (result any, e error)
• type Token
• type TokenHook
• type TokenHooks

Constants

const (
	// expecting token, got end of input
	UnexpectedEoiError = llx.SyntaxErrors + iota
	// cannot match grammar rule for incoming token
	UnexpectedTokenError
)

Syntax error codes used by parser:

const (
	// trying to emit token of unknown type, a literal, or an error token
	EmitWrongTokenError = llx.ParserErrors + iota
	// token hook for unknown token type name
	UnknownTokenTypeError
	// literal hook for unknown literal
	UnknownTokenLiteralError
	// node hook for unknown node
	UnknownNodeError
)

Other error codes used by parser:

const (
	AnyToken = ""                 // any token type
	EofToken = lexer.EofTokenName // end-of-file token
	EoiToken = lexer.EoiTokenName // end-of-input token
)

Special token type names used by token hooks.

const AnyNode = ""

AnyNode denotes any node, used by node hooks.

Types

type Hooks

type Hooks struct {
	// Tokens contains hooks for different token types. Key is either token type name or AnyToken constant.
	// AnyToken hook is a fallback.
	Tokens TokenHooks

	// Literals contains hooks for tokens with specific content. Key is token content.
	// These hooks have top priority (if token type allows matching against literals).
	Literals TokenHooks

	// Nodes contains hooks for nodes. Key is either node name or AnyNode constant.
	// AnyNode hook is a fallback.
	Nodes NodeHooks
}

Hooks contains all token and node hooks used in parsing process. Default action when no suitable token hook found is to drop aside token and use non-aside token as is.

type NodeHook

type NodeHook = func(node string, token *Token, pc *ParseContext) (NodeHookInstance, error)

NodeHook allows to perform actions on nodes emitted by parser. Receives node name and initial token.

type NodeHookInstance

type NodeHookInstance interface {
	// NewNode is called before a child node is pushed on stack.
	// Receives child node name and its initial token.
	NewNode(node string, token *Token) error

	// HandleNode is called when nested node is finalized and dropped.
	// Receives child node name.
	// Receives result of closest nested hook EndNode() call or nil if none of nested nodes was hooked.
	HandleNode(node string, result any) error

	// HandleToken is called when a token belonging to current node is received.
	HandleToken(token *Token) error

	// EndNode is called when current node is finalized.
	// result is passed to parent node hook or returned as a parse result if current node is the root.
	EndNode() (result any, e error)
}

NodeHookInstance receives notifications for node being processed by parser.

type NodeHooks

type NodeHooks map[string]NodeHook

type ParseContext

type ParseContext struct {
	// contains filtered or unexported fields
}

ParseContext contains all context used in parsing process.

func (*ParseContext) EmitToken

func (pc *ParseContext) EmitToken(t *Token) error

EmitToken adds new element to the end of token queue. Token's type must be defined in grammar, and it must not be a literal or an error token.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser holds prepared data for some grammar. Parser is immutable and reusable.

func New

func New(g *grammar.Grammar) (*Parser, error)

New constructs new parser for specific grammar. Grammar must not be changed after this function is called.

func (*Parser) Parse

func (p *Parser) Parse(q *source.Queue, hs *Hooks) (result any, e error)

Parse launches new parsing process with new ParseContext. result is the value returned by root node hook or nil if no node hooks used.

func (*Parser) ParseString

func (p *Parser) ParseString(name, content string, hs *Hooks) (result any, e error)

ParseString is same as Parse, except it creates source queue containing single source having provided content with provided name (name may be empty).

type Token

type Token = lexer.Token

type TokenHook

type TokenHook = func(token *Token, pc *ParseContext) (emit bool, e error)

TokenHook allows to perform additional actions when token is fetched from lexer, but before it is queued, e.g. emit external $indent/$dedent tokens when text indentation changes, fetch complex lexemes (e.g. heredoc strings). emit flag set to true means that incoming token (even aside one) must be put to the beginning of token queue, false means that it must be skipped.

type TokenHooks

type TokenHooks map[string]TokenHook