lexer

Documentation

Index

• Variables
• type Lexer
  • func (s *Lexer) GetTokenPosition() int
  • func (s *Lexer) JumpToToken(i int)
  • func (s *Lexer) NextToken() token.Token
  • func (s *Lexer) PeekToken() token.Token
  • func (s *Lexer) Scan(source string) []token.Token
  • func (s *Lexer) SetTokenPosition(position int)

Variables

var Builtins = map[string]string{
	"LEN":    "LEN",
	"ABS":    "ABS",
	"ATN":    "ATN",
	"COS":    "COS",
	"EXP":    "EXP",
	"INT":    "INT",
	"LN":     "LN",
	"LOG":    "LOG",
	"RND":    "RND",
	"SGN":    "SGN",
	"SIN":    "SIN",
	"SQR":    "SQR",
	"TAN":    "TAN",
	"GET":    "GET",
	"LOOKUP": "LOOKUP",
	"PATH$":  "PATH$",
	"STR$":   "STR$",
	"CHR$":   "CHR$",
	"PITCH":  "PITCH",
}

This is a bit hacky:

Types

type Lexer

type Lexer struct {
	Source          string        // source code string
	Tokens          []token.Token // buffer of tokens created by Scan()
	CurrentPosition int           // position in the string
	// contains filtered or unexported fields
}

Lexer describes a type with methods for lexing a line of source code and generating tokens. To tokenize a line of source code all we do is this:

s := &Scanner{}
tokens := s.ScanTokens(source)

func (*Lexer) GetTokenPosition

func (s *Lexer) GetTokenPosition() int

GetTokenPosition returns the current token position

func (*Lexer) JumpToToken

func (s *Lexer) JumpToToken(i int)

JumpToToken sets the currentTokenPosition

func (*Lexer) NextToken

func (s *Lexer) NextToken() token.Token

NextToken returns the token at the current tokenPosition in the buffer and consumes it, moving to the next token

func (*Lexer) PeekToken

func (s *Lexer) PeekToken() token.Token

PeekToken returns the token at the current tokenPosition in the buffer but does not consume it.

func (*Lexer) Scan

func (s *Lexer) Scan(source string) []token.Token

Scan scans the source code and returns a slice of tokens

func (*Lexer) SetTokenPosition

func (s *Lexer) SetTokenPosition(position int)

SetTokenPosition sets the current token position