lexer

Documentation

Index

• type Lexer
  • func New(source string) *Lexer
  • func (l *Lexer) Next() bool
  • func (l *Lexer) Peek() (token *Token)
  • func (l *Lexer) Scan() *Token
• type Token
• type TokenType

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer is a tokenizer that returns individual runes along with their associated types.

func New

func New(source string) *Lexer

New returns a new Lexer that wraps the given source string.

func (*Lexer) Next

func (l *Lexer) Next() bool

Next advances the lexer to the next token, and discards the current one. It must be called before any calls to Scan.

func (*Lexer) Peek

func (l *Lexer) Peek() (token *Token)

Peek returns the next token without consuming the current one. If the current token is the last token, Peek returns nil. It can be called before the first call to Next.

func (*Lexer) Scan

func (l *Lexer) Scan() *Token

Scan returns the current token.

type Token

type Token struct {
	Value string
	Type  TokenType
}

type TokenType

type TokenType int

TokenType enumerates the possible token types returned by the Lexer. Any unexported types never exit the package.

const (
	Asterisk TokenType
	Text
	Bracket
	Backslash
	Dash
	Caret
	Plus
)