scanner

Documentation

Overview

Package scanner implements a scanner for yacc[0] source text with actions written in Go. It takes a []byte as source which can then be tokenized through repeated calls to the Scan method.

Changelog

2014-12-18: Support %precedence for bison compatibility[1].

2014-11-17: Allow dashes in symbol names for bison compatibility[1].

Index

• type Scanner
  • func New(fset *token.FileSet, fname string, src []byte) (s *Scanner)
  • func (s *Scanner) Error(msg string)
  • func (s *Scanner) Mode(yacc bool)
  • func (s *Scanner) Pos() int
  • func (s *Scanner) Scan() (tok Token, lval interface{}, num int)
  • func (s *Scanner) ScanRaw() (tok Token, lval interface{}, num int)
• type Token
  • func (i Token) String() string

Types

type Scanner

type Scanner struct {
	Col    int     // Starting column of the last scanned token.
	Errors []error // List of accumulated errors.
	FName  string  // File name (reported) of the scanned source.
	Line   int     // Starting line of the last scanned token.
	NCol   int     // Starting column (reported) for the next scanned token.
	NLine  int     // Starting line (reported) for the next scanned token.
	// contains filtered or unexported fields
}

A Scanner holds the scanner's internal state while processing a given text.

func New

func New(fset *token.FileSet, fname string, src []byte) (s *Scanner)

New returns a newly created Scanner and set its FName to fname

func (*Scanner) Error

func (s *Scanner) Error(msg string)

Error implements yyLexer.

func (*Scanner) Mode

func (s *Scanner) Mode(yacc bool)

Mode allows to switch the scanner mode from scanning yacc tokens to scanning rule action tokens (Go tokens). Initially the yacc mode is active.

func (*Scanner) Pos

func (s *Scanner) Pos() int

Pos returns the starting offset of the last scanned token.

func (*Scanner) Scan

func (s *Scanner) Scan() (tok Token, lval interface{}, num int)

Scan works like ScanRaw but recognizes also yacc's C_IDENTIFIER token (in yacc mode).

func (*Scanner) ScanRaw

func (s *Scanner) ScanRaw() (tok Token, lval interface{}, num int)

ScanRaw scans the next token and returns the token and its value if applicable. The source end is indicated by EOF.

If the returned token is IDENT, INT, FLOAT, IMAG, CHAR, STRING or COMMENT, lval has has the corresponding value - not the string representation of the value. However, numeric literals which overflow the corresponding Go predeclared types are returned as string.

If the returned token is DLR_NUM or DLR_TAG_DLR, num has the corresponding number (int) or lval has the corresponding tag (string).

If the returned token is DLR_TAG_NUM, lval has the corresponding tag (string) and num has the corresponding number.

If the returned token is ILLEGAL, the literal string is the offending character or number/string/char literal.

type Token

type Token int

const (

	// Special tokens
	ILLEGAL Token
	EOF
	COMMENT

	// Identifiers and basic type literals
	// (these tokens stand for classes of literals)
	IDENT  // main
	INT    // 12345
	FLOAT  // 123.45
	IMAG   // 123.45i
	CHAR   // 'a'
	STRING // "abc"

	// Operators and delimiters
	ADD // +
	SUB // -
	MUL // *
	QUO // /
	REM // %

	AND     // &
	OR      // |
	XOR     // ^
	SHL     // <<
	SHR     // >>
	AND_NOT // &^

	ADD_ASSIGN // +=
	SUB_ASSIGN // -=
	MUL_ASSIGN // *=
	QUO_ASSIGN // /=
	REM_ASSIGN // %=

	AND_ASSIGN     // &=
	OR_ASSIGN      // |=
	XOR_ASSIGN     // ^=
	SHL_ASSIGN     // <<=
	SHR_ASSIGN     // >>=
	AND_NOT_ASSIGN // &^=

	LAND  // &&
	LOR   // ||
	ARROW // <-
	INC   // ++
	DEC   // --

	EQL    // ==
	LSS    // <
	GTR    // >
	ASSIGN // =
	NOT    // !

	NEQ      // !=
	LEQ      // <=
	GEQ      // >=
	DEFINE   // :=
	ELLIPSIS // ...

	LPAREN // (
	LBRACK // [
	LBRACE // {
	COMMA  // ,
	PERIOD // .

	RPAREN    // )
	RBRACK    // ]
	RBRACE    // }
	SEMICOLON // ;
	COLON     // :

	// Keywords
	BREAK
	CASE
	CHAN
	CONST
	CONTINUE

	DEFAULT
	DEFER
	ELSE
	FALLTHROUGH
	FOR

	FUNC
	GO
	GOTO
	IF
	IMPORT

	INTERFACE
	MAP
	PACKAGE
	RANGE
	RETURN

	SELECT
	STRUCT
	SWITCH
	GO_TYPE
	VAR

	DLR_DLR     // $$
	DLR_NUM     // $1
	DLR_TAG_DLR // $<tag>$
	DLR_TAG_NUM // $<tag>2

	// yacc mode tokens
	C_IDENTIFIER // IDENTIFIER ':'
	ERR_VERBOSE  // %error-verbose
	IDENTIFIER   // [a-zA-Z_][a-zA-Z0-9_.-]*
	LCURL        // %{
	LEFT         // %left
	MARK         // %%
	NONASSOC     // %nonassoc
	PREC         // %prec
	RCURL        // %}
	RIGHT        // %right
	START        // %start
	TOKEN        // %token
	TYPE         // %type
	UNION        // %union
	PRECEDENCE   // %precedence
)

func (Token) String

func (i Token) String() string

String implements fmt.Stringer