Documentation ¶
Overview ¶
Package lexer converts source code into tokens. Tokens are indivisible units that are used by the parser package to create the AST structures that represent the source code.
Index ¶
Constants ¶
View Source
const ( TokenEOF = "end of file" // Dynamic TokenBoolLiteral = "bool literal" // boolean literal, eg. true TokenCharLiteral = "char literal" // char literal, eg. 'a' TokenComment = "comment" // eg. "//..." TokenDataLiteral = "data literal" // data literal, eg. `foo` TokenIdentifier = "identifier" // any non-keyword TokenNumberLiteral = "number literal" // number literal, eg. 12.3 TokenStringLiteral = "string literal" // string literal, eg. "hello" // Keywords TokenAnd = "and" TokenAny = "any" TokenAssert = "assert" TokenBool = "bool" TokenBreak = "break" TokenCase = "case" TokenChar = "char" TokenContinue = "continue" TokenData = "data" TokenElse = "else" TokenFor = "for" TokenFunc = "func" TokenIf = "if" TokenImport = "import" TokenIn = "in" TokenNot = "not" TokenNumber = "number" TokenOr = "or" TokenReturn = "return" TokenString = "string" TokenSwitch = "switch" TokenTest = "test" // Operators TokenAssign = "=" TokenColon = ":" TokenComma = "," TokenCurlyClose = "}" TokenCurlyOpen = "{" TokenDecrement = "--" TokenDivide = "/" TokenDivideAssign = "/=" TokenDot = "." TokenEqual = "==" TokenGreaterThan = ">" TokenGreaterThanEqual = ">=" TokenIncrement = "++" TokenLessThan = "<" TokenLessThanEqual = "<=" TokenMinus = "-" TokenMinusAssign = "-=" TokenNotEqual = "!=" TokenParenClose = ")" TokenParenOpen = "(" TokenPlus = "+" TokenPlusAssign = "+=" TokenRemainder = "%" TokenRemainderAssign = "%=" TokenSemiColon = ";" TokenSquareClose = "]" TokenSquareOpen = "[" TokenTimes = "*" TokenTimesAssign = "*=" )
Tokens defined here have a human-readable name used in error messages. You should not rely on these values staying the same, only that their value will be unique amongst the defined tokens.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Options ¶
type Options struct { // IncludeComments will include TokenComment in the returned tokens. IncludeComments bool }
Options allows configuration of the lexer.
type Token ¶
type Token struct { // Kind will be one of the Token* constants. Kind string // Value is captured from the original source code. It is only useful for // dynamic tokens such as TokenString or TokenIdentifier. Value string // IsEndOfLine will be true if there is at least one new line character // after this token (ignoring other whitespace). This is needed by some // grammars to determine the end of the line, but newlines have no effect in // between most tokens. // // One exception to this is comments. Comment always have to be terminated // by a new line, however, IsEndOfLine will only be true if the comment is // followed by an empty line. This is because IsEndOfLine is also used by // the lexer to determine if comments are part of the same block as their // previous lines and/or if they might be attached to functions are // documentation. IsEndOfLine bool }
Token represents a single token.
func TokenizeString ¶
TokenizeString returns a slice of tokens from the provided str.
Click to show internal directories.
Click to hide internal directories.