token

Documentation

Overview

There are 2 types of Token, constant ones (with no "value") and the ones with attached value that is variable (e.g. IDENT, INT, FLOAT, STRING, *COMMENT). We might have used the upcoming unique https://tip.golang.org/doc/go1.23#new-unique-package but we want this to run on 1.22 and earlier and rolled our own, not multi threaded.

Index

• Variables
• func Init()
• func ResetInterning()
• type GrolInfo
  • func Info() GrolInfo
• type Token
  • func ByType(t Type) *Token
  • func ConstantTokenChar(literal byte) *Token
  • func ConstantTokenChar2(c1, c2 byte) *Token
  • func Intern(t Type, literal string) *Token
  • func InternToken(t *Token) *Token
  • func LookupIdent(ident string) *Token
  • func (t *Token) DebugString() string
  • func (t *Token) Literal() string
  • func (t *Token) Type() Type
• type Type
  • func (i Type) String() string

Variables

var (
	EOLT   = &Token{tokenType: EOL}
	EOFT   = &Token{tokenType: EOF}
	TRUET  *Token
	FALSET *Token
)

Functions

func Init

func Init()

func ResetInterning

func ResetInterning()

Types

type GrolInfo

type GrolInfo struct {
	// Keywords is a map of all known keywords.
	Keywords sets.Set[string]
	Builtins sets.Set[string]
	Tokens   sets.Set[string]
}

Info enables introspection of known keywords and tokens and operators.

func Info

func Info() GrolInfo

type Token

type Token struct {
	// contains filtered or unexported fields
}

func ByType

func ByType(t Type) *Token

ByType is the cheapest lookup for all the tokens whose type only have one possible instance/value (ie all the tokens except for the first 4 value tokens). TODO: codegen all the token constants to avoid needing this function. (even though that's better than string comparisons).

func ConstantTokenChar

func ConstantTokenChar(literal byte) *Token

func ConstantTokenChar2

func ConstantTokenChar2(c1, c2 byte) *Token

func Intern

func Intern(t Type, literal string) *Token

func InternToken

func InternToken(t *Token) *Token

Lookup a unique pointer to a token of same values, if it exists, otherwise store the passed in one for future lookups.

func LookupIdent

func LookupIdent(ident string) *Token

func (*Token) DebugString

func (t *Token) DebugString() string

func (*Token) Literal

func (t *Token) Literal() string

func (*Token) Type

func (t *Token) Type() Type

type Type

type Type uint8

const (
	ILLEGAL Type = iota
	EOL

	// Identifiers + literals. with attached value.
	IDENT  // add, foobar, x, y, ...
	INT    // 1343456
	FLOAT  // .5, 3.14159,...
	STRING // "foo bar" or `foo bar`
	LINECOMMENT
	BLOCKCOMMENT
	REGISTER // not used for parsing, only to tag object.Register as ast node of unique type.

	// Single character operators.
	ASSIGN
	PLUS
	MINUS
	BANG
	ASTERISK
	SLASH
	PERCENT
	LT
	GT
	BITAND
	BITOR
	BITXOR
	BITNOT

	// Delimiters.
	COMMA
	SEMICOLON

	LPAREN
	RPAREN
	LBRACE
	RBRACE
	LBRACKET
	RBRACKET
	COLON
	DOT

	// LT, GT or equal variants.
	LTEQ
	GTEQ
	EQ
	NOTEQ
	INCR
	DECR
	DOTDOT
	OR
	AND
	LEFTSHIFT
	RIGHTSHIFT
	LAMBDA // => lambda short cut: `a,b => a+b` alias for `func(a,b) {a+b}`
	DEFINE // := (force create new variable instead of possible ref to upper scope)

	// Keywords.
	FUNC
	TRUE
	FALSE
	IF
	ELSE
	RETURN
	FOR
	BREAK
	CONTINUE
	// Macro magic.
	MACRO
	QUOTE
	UNQUOTE
	// Built-in functions.
	LEN
	FIRST
	REST
	PRINT
	PRINTLN
	LOG
	ERROR
	CATCH

	EOF
)

func (Type) String

func (i Type) String() string