scanner

Documentation

Index

• type Scanner
  • func New(f *token.File, data []byte) *Scanner
  • func (s *Scanner) File() *token.File
  • func (s *Scanner) Init(f *token.File, data []byte)
  • func (s *Scanner) Scan() (pos token.Pos, tok token.Token, lit string)
  • func (s *Scanner) ScanStringExpr() (pos token.Pos, tok token.Token, lit string)
  • func (s *Scanner) ScanWithRegex() (pos token.Pos, tok token.Token, lit string)
  • func (s *Scanner) Unread()

Types

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner is used to tokenize Flux source.

func New

func New(f *token.File, data []byte) *Scanner

New will construct and initialize a new Scanner.

func (*Scanner) File

func (s *Scanner) File() *token.File

File returns the file being processed by the Scanner.

func (*Scanner) Init

func (s *Scanner) Init(f *token.File, data []byte)

Init initializes the Scanner to scan the data in the byte array.

func (*Scanner) Scan

func (s *Scanner) Scan() (pos token.Pos, tok token.Token, lit string)

Scan will scan the next token while excluding regex literal tokens. This is useful in situations where the parser is not expecting a regular expression such as when reading a binary operator in an expression. This method is needed because the start token for division and the token for a regular expression are the same so this ensures the expression "a / b / c" will not be parsed as a regular expression.

func (*Scanner) ScanStringExpr

func (s *Scanner) ScanStringExpr() (pos token.Pos, tok token.Token, lit string)

ScanStringExpr will scan the next token in a string expression context.

func (*Scanner) ScanWithRegex

func (s *Scanner) ScanWithRegex() (pos token.Pos, tok token.Token, lit string)

ScanWithRegex will scan the next token for the entire grammar.

func (*Scanner) Unread

func (s *Scanner) Unread()

Unread will reset the Scanner to go back to the Scanner's location before the last ScanWithRegex or Scan call. If either of the ScanWithRegex methods returned an EOF token, a call to Unread will not unread the discarded whitespace. This method is a no-op if called multiple times.