scanner

Documentation

Overview

Package scanner implements a lexical scanner for River source files.

Index

• type ErrorHandler
• type Mode
• type Scanner
  • func New(file *token.File, input []byte, eh ErrorHandler, mode Mode) *Scanner
  • func (s *Scanner) NumErrors() int
  • func (s *Scanner) Scan() (pos token.Pos, tok token.Token, lit string)

Types

type ErrorHandler

type ErrorHandler func(pos token.Pos, msg string)

ErrorHandler is invoked whenever there is an error.

type Mode

type Mode uint

Mode is a set of bitwise flags which control scanner behavior.

const (
	// IncludeComments will cause comments to be returned as comment tokens.
	// Otherwise, comments are ignored.
	IncludeComments Mode = 1 << iota
)

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner holds the internal state for the tokenizer while processing configs.

func New

func New(file *token.File, input []byte, eh ErrorHandler, mode Mode) *Scanner

New creates a new scanner to tokenize the provided input config. The scanner uses the provided file for adding line information for each token. The mode parameter customizes scanner behavior.

Calls to Scan will invoke the error handler eh when a lexical error is found if eh is not nil.

func (*Scanner) NumErrors

func (s *Scanner) NumErrors() int

NumErrors returns the current number of errors encountered during scanning. This is useful as a fallback to detect errors when no ErrorHandler was provided to the scanner.

func (*Scanner) Scan

func (s *Scanner) Scan() (pos token.Pos, tok token.Token, lit string)

Scan scans the next token and returns the token's position, the token itself, and the token's literal string (when applicable). The end of the input is indicated by token.EOF.

If the returned token is a literal (such as token.STRING), then lit contains the corresponding literal text (including surrounding quotes).

If the returned token is a keyword, lit is the keyword text that was scanned.

If the returned token is token.TERMINATOR, lit will contain "\n".

If the returned token is token.ILLEGAL, lit contains the offending character.

In all other cases, lit will be an empty string.

For more tolerant parsing, Scan returns a valid token character whenever possible when a syntax error was encountered. Callers must check NumErrors or the number of times the provided ErrorHandler was invoked to ensure there were no errors found during scanning.

Scan will inject line information to the file provided by NewScanner. Returned token positions are relative to that file.