baselexer

Documentation

Index

• Constants
• type Config
• type Lexer
  • func New(cfg Config) *Lexer
  • func (l *Lexer) Peek() rune
  • func (l *Lexer) Pos() gotoken.Pos
  • func (l *Lexer) ReadToken() (Token, int)
• type StateMachine
• type Token

Constants

const EOF = 0

const ERROR = 1

Types

type Config

type Config struct {
	// StateMachine is the Lox-generated state machine.
	StateMachine StateMachine

	// File is a go/token.File which is used to efficiently tag tokens with file
	// positions.
	File *gotoken.File

	// Input is input to be parsed by the Lexer.
	Input []byte
}

Config configures a Lexer.

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer is a simple lexer implementation to be used with lexer state machines produced by Lox. It produces tokens containing byte slices of the original input which may not be appropriate for very large or stream inputs.

func New

func New(cfg Config) *Lexer

New creates a Lexer using a Lox-generated state machine.

func (*Lexer) Peek

func (l *Lexer) Peek() rune

Peek returns the next input rune to be consumed by the Lexer.

func (*Lexer) Pos

func (l *Lexer) Pos() gotoken.Pos

Pos returns the position of the next rune to be consumed by the Lexer.

func (*Lexer) ReadToken

func (l *Lexer) ReadToken() (Token, int)

ReadToken parses a token from the input. It returns both the token value and the token id as required by the generated Lox _Lexer interface. Return token id EOF after the end of the input has been reached.

type StateMachine

type StateMachine interface {
	// PushRune pushes a rune to the state machine. It can return one of the
	// following:
	// 0 (consume): consume the rune and include it in the token.
	// 1 (accept): accept the current token. The token id is specified by Token().
	// 2 (discard): discard current token.
	// 3 (try-again): Call PushRune again with the same rune.
	// 4 (EOF): We are done.
	// -1 (error): That's an error.
	PushRune(r rune) int

	// Token returns the token-id of the recognized token when PushRune
	// returns 1 (accept).
	Token() int

	// Reset resets the state machine so it can be used again.
	Reset()
}

StateMachine is an interface that is satisfied by a state machine implementation generated by Lox.

type Token

type Token struct {
	Type int
	Str  []byte
	Pos  gotoken.Pos
}

Token is the type of tokens produced by Lexer.