lexer

Documentation

Index

• Constants
• type Lexer
  • func New(src string) *Lexer
  • func (l *Lexer) NextDynamicToken() (t token.Token)
  • func (l *Lexer) NextToken() (t token.Token)
  • func (l *Lexer) PeekNextTokenType() token.Type

Constants

const (
	// Begin is the symbol which lexer should scan forward to.
	Begin = '{' // token.LBRACE
	// End is the symbol which lexer should stop scanning.
	End = '}' // token.RBRACE
)

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer helps us to read/scan characters of a source and resolve their token types.

func New

func New(src string) *Lexer

New takes a source, series of chars, and returns a new, ready to read from the first letter, lexer.

func (*Lexer) NextDynamicToken

func (l *Lexer) NextDynamicToken() (t token.Token)

NextDynamicToken doesn't cares about the grammar. It reads numbers or any unknown symbol, it's being used by parser to skip all characters between parameter function's arguments inside parenthesis, in order to allow custom regexp on the end-language too.

It moves the cursor forward.

func (*Lexer) NextToken

func (l *Lexer) NextToken() (t token.Token)

NextToken returns the next token in the series of characters. It can be a single symbol, a token type or a literal. It's able to return an EOF token too.

It moves the cursor forward.

func (*Lexer) PeekNextTokenType

func (l *Lexer) PeekNextTokenType() token.Type

PeekNextTokenType returns only the token type of the next character and it does not move forward the cursor. It's being used by parser to recognise empty functions, i.e `even()` as valid functions with zero input arguments.