token

README

Token package

Token.go

Token data structure ( Tokens are identified based on the specific rules of the lexer. ... Special characters, including punctuation characters, are commonly used by lexers to identify tokens because of their natural use in written and programming languages. )

Documentation

Index

• Constants
• type Token
• type TokenType
  • func LookupIdent(ident string) TokenType

Constants

const (
	// Control
	ILLEGAL = "ILLEGAL" // When we find token/character we don't know
	EOF     = "EOF"     // End of file
	// Identifiers + literals
	IDENT = "IDENT" // add, foobar, x, y, ...
	INT   = "INT"   // 1343456
	// Operators
	ASSIGN   = "="
	PLUS     = "+"
	MINUS    = "-"
	BANG     = "!"
	ASTERISK = "*"
	SLASH    = "/"
	LT       = "<"
	GT       = ">"
	EQ       = "=="
	NOT_EQ   = "!="
	// Delimiters
	COMMA     = ","
	SEMICOLON = ";"
	LPAREN    = "("
	RPAREN    = ")"
	LBRACE    = "{"
	RBRACE    = "}"
	// Keywords
	FUNCTION = "FUNCTION"
	LET      = "LET"
	TRUE     = "TRUE"
	FALSE    = "FALSE"
	IF       = "IF"
	ELSE     = "ELSE"
	RETURN   = "RETURN"
)

Types

type Token

type Token struct {
	Type    TokenType
	Literal string
}

type TokenType

type TokenType string

func LookupIdent

func LookupIdent(ident string) TokenType

LookupIdent checks the keywords table to see whether the given identifier is in fact a keyword