scanner

Documentation

Overview

A scanner for Go source text. Takes a []byte as source which can then be tokenized through repeated calls to the Scan function. For a sample use of a scanner, see the implementation of Tokenize.

Index

• Constants
• func PrintError(w io.Writer, err os.Error)
• func Tokenize(filename string, src []byte, err ErrorHandler, mode uint, ...) int
• type Error
  • func (e *Error) String() string
• type ErrorHandler
• type ErrorList
  • func (p ErrorList) Len() int
  • func (p ErrorList) Less(i, j int) bool
  • func (p ErrorList) String() string
  • func (p ErrorList) Swap(i, j int)
• type ErrorVector
  • func (h *ErrorVector) Error(pos token.Position, msg string)
  • func (h *ErrorVector) ErrorCount() int
  • func (h *ErrorVector) GetError(mode int) os.Error
  • func (h *ErrorVector) GetErrorList(mode int) ErrorList
  • func (h *ErrorVector) Reset()
• type Scanner
  • func (S *Scanner) Init(filename string, src []byte, err ErrorHandler, mode uint)
  • func (S *Scanner) Scan() (pos token.Position, tok token.Token, lit []byte)

Constants

const (
	Raw         = iota // leave error list unchanged
	Sorted             // sort error list by file, line, and column number
	NoMultiples        // sort error list and leave only the first error per line
)

These constants control the construction of the ErrorList returned by GetErrors.

const (
	ScanComments      = 1 << iota // return comments as COMMENT tokens
	AllowIllegalChars             // do not report an error for illegal chars
	InsertSemis                   // automatically insert semicolons
)

The mode parameter to the Init function is a set of flags (or 0). They control scanner behavior.

Functions

func PrintError

func PrintError(w io.Writer, err os.Error)

PrintError is a utility function that prints a list of errors to w, one error per line, if the err parameter is an ErrorList. Otherwise it prints the err string.

func Tokenize

func Tokenize(filename string, src []byte, err ErrorHandler, mode uint, f func(pos token.Position, tok token.Token, lit []byte) bool) int

Tokenize calls a function f with the token position, token value, and token text for each token in the source src. The other parameters have the same meaning as for the Init function. Tokenize keeps scanning until f returns false (usually when the token value is token.EOF). The result is the number of errors encountered.

Types

type Error

type Error struct {
	Pos token.Position
	Msg string
}

Within ErrorVector, an error is represented by an Error node. The position Pos, if valid, points to the beginning of the offending token, and the error condition is described by Msg.

func (*Error) String

func (e *Error) String() string

type ErrorHandler

type ErrorHandler interface {
	Error(pos token.Position, msg string)
}

An implementation of an ErrorHandler may be provided to the Scanner. If a syntax error is encountered and a handler was installed, Error is called with a position and an error message. The position points to the beginning of the offending token.

type ErrorList

type ErrorList []*Error

An ErrorList is a (possibly sorted) list of Errors.

func (ErrorList) Len

func (p ErrorList) Len() int

ErrorList implements the sort Interface.

func (ErrorList) Less

func (p ErrorList) Less(i, j int) bool

func (ErrorList) String

func (p ErrorList) String() string

func (ErrorList) Swap

func (p ErrorList) Swap(i, j int)

type ErrorVector

type ErrorVector struct {
	// contains filtered or unexported fields
}

ErrorVector implements the ErrorHandler interface. It maintains a list of errors which can be retrieved with GetErrorList and GetError. The zero value for an ErrorVector is an empty ErrorVector ready to use.

A common usage pattern is to embed an ErrorVector alongside a scanner in a data structure that uses the scanner. By passing a reference to an ErrorVector to the scanner's Init call, default error handling is obtained.

func (*ErrorVector) Error

func (h *ErrorVector) Error(pos token.Position, msg string)

ErrorVector implements the ErrorHandler interface.

func (*ErrorVector) ErrorCount

func (h *ErrorVector) ErrorCount() int

ErrorCount returns the number of errors collected.

func (*ErrorVector) GetError

func (h *ErrorVector) GetError(mode int) os.Error

GetError is like GetErrorList, but it returns an os.Error instead so that a nil result can be assigned to an os.Error variable and remains nil.

func (*ErrorVector) GetErrorList

func (h *ErrorVector) GetErrorList(mode int) ErrorList

GetErrorList returns the list of errors collected by an ErrorVector. The construction of the ErrorList returned is controlled by the mode parameter. If there are no errors, the result is nil.

func (*ErrorVector) Reset

func (h *ErrorVector) Reset()

Reset resets an ErrorVector to no errors.

type Scanner

type Scanner struct {

	// public state - ok to modify
	ErrorCount int // number of errors encountered
	// contains filtered or unexported fields
}

A Scanner holds the scanner's internal state while processing a given text. It can be allocated as part of another data structure but must be initialized via Init before use. For a sample use, see the implementation of Tokenize.

func (*Scanner) Init

func (S *Scanner) Init(filename string, src []byte, err ErrorHandler, mode uint)

Init prepares the scanner S to tokenize the text src. Calls to Scan will use the error handler err if they encounter a syntax error and err is not nil. Also, for each error encountered, the Scanner field ErrorCount is incremented by one. The filename parameter is used as filename in the token.Position returned by Scan for each token. The mode parameter determines how comments and illegal characters are handled.

func (*Scanner) Scan

func (S *Scanner) Scan() (pos token.Position, tok token.Token, lit []byte)

Scan scans the next token and returns the token position pos, the token tok, and the literal text lit corresponding to the token. The source end is indicated by token.EOF.

For more tolerant parsing, Scan will return a valid token if possible even if a syntax error was encountered. Thus, even if the resulting token sequence contains no illegal tokens, a client may not assume that no error occurred. Instead it must check the scanner's ErrorCount or the number of calls of the error handler, if there was one installed.