lexer

Documentation

Overview

Package lexer implements the lexical analysis that is used to transform the source code input into a stream of tokens for parsing by the parser.

Index

• type Lexer
  • func New(input string) *Lexer
  • func (l *Lexer) NextToken() token.Token

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Represents the Lexer and contains the source input and internal state

func New

func New(input string) *Lexer

Returns a new Lexer

func (*Lexer) NextToken

func (l *Lexer) NextToken() token.Token

Returns the next token read from the input stream