lex

Documentation

Index

• func HasLessPrecedence(current Token, next Token) bool
• func IsTerminal(tok Token) bool
• type Lexer
  • func Lex(input string) *Lexer
  • func (l *Lexer) Next() Token
  • func (l Lexer) Peek() Token
• type TokType
  • func (tt TokType) String() string
• type Token
  • func (i Token) String() string

Functions

func HasLessPrecedence

func HasLessPrecedence(current Token, next Token) bool

HasLessPrecedence checks if a current token has lower precedence than the next. There is a specific ordering in the iota (lower numbers = higher precedence) indicating whether the operator has more precedence or not.

func IsTerminal

func IsTerminal(tok Token) bool

IsTerminal checks wether a specific token is a terminal token meaning it can't be matched in the grammar.

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer is a lexer that will parse an input string into tokens for consumption by a grammar parser.

func Lex

func Lex(input string) *Lexer

Lex creates a lexer for an input string

func (*Lexer) Next

func (l *Lexer) Next() Token

Next parses and returns just the next token in the input.

func (Lexer) Peek

func (l Lexer) Peek() Token

Peek looks at the the next token but does not impact the lexer state note this is intentionally not a pointer because we don't want any changes to take affect here.

type TokType

type TokType int

TokType is an enum of token types that can be parsed by the lexer. Order matters here for non terminals with a lower number meaning a higher precedence.

const (
	// terminal characters
	TErr TokType = iota
	TLiteral
	TQuoted
	TRegexp

	// precedence of operators. Order matters here. This might need to be abstracted
	// to a grammar specific precedence but for now it is fine here.
	TEqual
	TGreater
	TLess
	TColon
	TPlus
	TMinus
	TTilde
	TCarrot
	TNot
	TAnd
	TOr
	TRParen
	TLParen

	// operators that do not have a set precedence because we specifically handle them
	// due to ambiguities in the grammar
	TLCurly
	TRCurly
	TTO
	TLSquare
	TRSquare

	// start and end operators
	TEOF
	TStart
)

types of tokens that can be parsed

func (TokType) String

func (tt TokType) String() string

type Token

type Token struct {
	Typ TokType // the type of the item

	Val string // the value of the item
	// contains filtered or unexported fields
}

Token is a parsed token from the input buffer sent to the lexer

func (Token) String

func (i Token) String() string

String is a string representation of a lex item