token

Documentation

Overview

This package defines constants representing the lexical tokens of the Go programming language and basic operations on tokens (printing, predicates).

Index

• Constants
• type Position
  • func (pos *Position) IsValid() bool
  • func (pos *Position) Pos() Position
  • func (pos Position) String() string
• type Token
  • func Lookup(ident []byte) Token
  • func (tok Token) IsKeyword() bool
  • func (tok Token) IsLiteral() bool
  • func (tok Token) IsOperator() bool
  • func (op Token) Precedence() int
  • func (tok Token) String() string

Constants

const (
	LowestPrec  = 0 // non-operators
	UnaryPrec   = 7
	HighestPrec = 8
)

A set of constants for precedence-based expression parsing. Non-operators have lowest precedence, followed by operators starting with precedence 1 up to unary operators. The highest precedence corresponds serves as "catch-all" precedence for selector, indexing, and other operator and delimiter tokens.

Types

type Position

type Position struct {
	Filename string // filename, if any
	Offset   int    // byte offset, starting at 0
	Line     int    // line number, starting at 1
	Column   int    // column number, starting at 1 (character count)
}

Token source positions are represented by a Position value. A Position is valid if the line number is > 0.

func (*Position) IsValid

func (pos *Position) IsValid() bool

IsValid returns true if the position is valid.

func (*Position) Pos

func (pos *Position) Pos() Position

Pos is an accessor method for anonymous Position fields. It returns its receiver.

func (Position) String

func (pos Position) String() string

type Token

type Token int

Token is the set of lexical tokens of the Go programming language.

const (
	// Special tokens
	ILLEGAL Token = iota
	EOF
	COMMENT

	// Identifiers and basic type literals
	// (these tokens stand for classes of literals)
	IDENT  // main
	INT    // 12345
	FLOAT  // 123.45
	IMAG   // 123.45i
	CHAR   // 'a'
	STRING // "abc"

	// Operators and delimiters
	ADD // +
	SUB // -
	MUL // *
	QUO // /
	REM // %

	AND     // &
	OR      // |
	XOR     // ^
	SHL     // <<
	SHR     // >>
	AND_NOT // &^

	ADD_ASSIGN // +=
	SUB_ASSIGN // -=
	MUL_ASSIGN // *=
	QUO_ASSIGN // /=
	REM_ASSIGN // %=

	AND_ASSIGN     // &=
	OR_ASSIGN      // |=
	XOR_ASSIGN     // ^=
	SHL_ASSIGN     // <<=
	SHR_ASSIGN     // >>=
	AND_NOT_ASSIGN // &^=

	LAND  // &&
	LOR   // ||
	ARROW // <-
	INC   // ++
	DEC   // --

	EQL    // ==
	LSS    // <
	GTR    // >
	ASSIGN // =
	NOT    // !

	NEQ      // !=
	LEQ      // <=
	GEQ      // >=
	DEFINE   // :=
	ELLIPSIS // ...

	LPAREN // (
	LBRACK // [
	LBRACE // {
	COMMA  // ,
	PERIOD // .

	RPAREN    // )
	RBRACK    // ]
	RBRACE    // }
	SEMICOLON // ;
	COLON     // :

	// Keywords
	BREAK
	CASE
	CHAN
	CONST
	CONTINUE

	DEFAULT
	DEFER
	ELSE
	FALLTHROUGH
	FOR

	FUNC
	GO
	GOTO
	IF
	IMPORT

	INTERFACE
	MAP
	PACKAGE
	RANGE
	RETURN

	SELECT
	STRUCT
	SWITCH
	TYPE
	VAR
)

The list of tokens.

func Lookup

func Lookup(ident []byte) Token

Lookup maps an identifier to its keyword token or IDENT (if not a keyword).

func (Token) IsKeyword

func (tok Token) IsKeyword() bool

IsKeyword returns true for tokens corresponding to keywords; returns false otherwise.

func (Token) IsLiteral

func (tok Token) IsLiteral() bool

IsLiteral returns true for tokens corresponding to identifiers and basic type literals; returns false otherwise.

func (Token) IsOperator

func (tok Token) IsOperator() bool

IsOperator returns true for tokens corresponding to operators and delimiters; returns false otherwise.

func (Token) Precedence

func (op Token) Precedence() int

Precedence returns the operator precedence of the binary operator op. If op is not a binary operator, the result is LowestPrecedence.

func (Token) String

func (tok Token) String() string

String returns the string corresponding to the token tok. For operators, delimiters, and keywords the string is the actual token character sequence (e.g., for the token ADD, the string is "+"). For all other tokens the string corresponds to the token constant name (e.g. for the token IDENT, the string is "IDENT").