scanner

Documentation

Index

• type Mode
• type Option
  • func WithFilename(filename string) Option
• type Position
  • func NewPosition() *Position
  • func (pos *Position) Advance(r rune)
  • func (pos *Position) Revert(r rune)
  • func (pos Position) String() string
• type Scanner
  • func NewScanner(r io.Reader, opts ...Option) *Scanner
  • func (s *Scanner) LastScanRaw() []rune
  • func (s *Scanner) Scan() (Token, string, Position, error)
  • func (s *Scanner) SetLastScanRaw(raw []rune)
  • func (s *Scanner) UnScan()
• type Token

Types

type Mode

type Mode uint

Mode is an enum type to control recognition of tokens.

const (
	ScanIdent Mode = 1 << iota
	ScanNumberLit
	ScanStrLit
	ScanBoolLit
	ScanKeyword
	ScanComment
	ScanLit = ScanNumberLit | ScanStrLit | ScanBoolLit
)

Predefined mode bits to control recognition of tokens.

type Option

type Option func(*Scanner)

Option is an option for scanner.NewScanner.

func WithFilename

func WithFilename(filename string) Option

WithFilename is an option to set filename to the pos.

type Position

type Position struct {
	// Filename is a name of file, if any
	Filename string
	// Offset is a byte offset, starting at 0
	Offset int
	// Line is a line number, starting at 1
	Line int
	// Column is a column number, starting at 1 (character count per line)
	Column int
	// contains filtered or unexported fields
}

Position represents a source position.

func NewPosition

func NewPosition() *Position

NewPosition creates a new Position.

func (*Position) Advance

func (pos *Position) Advance(r rune)

Advance advances the position value.

func (*Position) Revert

func (pos *Position) Revert(r rune)

Revert reverts the position value.

func (Position) String

func (pos Position) String() string

String stringify the position.

type Scanner

type Scanner struct {

	// The Mode field controls which tokens are recognized.
	Mode Mode
	// contains filtered or unexported fields
}

Scanner represents a lexical scanner.

func NewScanner

func NewScanner(r io.Reader, opts ...Option) *Scanner

NewScanner returns a new instance of Scanner.

func (*Scanner) LastScanRaw

func (s *Scanner) LastScanRaw() []rune

LastScanRaw returns the deep-copied lastScanRaw.

func (*Scanner) Scan

func (s *Scanner) Scan() (Token, string, Position, error)

Scan returns the next token and text value.

func (*Scanner) SetLastScanRaw

func (s *Scanner) SetLastScanRaw(raw []rune)

SetLastScanRaw sets lastScanRaw to the given raw.

func (*Scanner) UnScan

func (s *Scanner) UnScan()

UnScan put the last scanned text back to the read buffer.

type Token

type Token int

Token represents a lexical token.

const (
	// Special tokens
	TILLEGAL Token = iota
	TEOF

	// Identifiers
	TIDENT

	// Literals
	TINTLIT
	TFLOATLIT
	TBOOLLIT
	TSTRLIT

	// Comment
	TCOMMENT

	// Misc characters
	TSEMICOLON   // ;
	TCOLON       // :
	TEQUALS      // =
	TQUOTE       // " or '
	TLEFTPAREN   // (
	TRIGHTPAREN  // )
	TLEFTCURLY   // {
	TRIGHTCURLY  // }
	TLEFTSQUARE  // [
	TRIGHTSQUARE // ]
	TLESS        // <
	TGREATER     // >
	TCOMMA       // ,
	TDOT         // .

	// Keywords
	TSYNTAX
	TSERVICE
	TRPC
	TRETURNS
	TMESSAGE
	TEXTEND
	TIMPORT
	TPACKAGE
	TOPTION
	TREPEATED
	TREQUIRED
	TOPTIONAL
	TWEAK
	TPUBLIC
	TONEOF
	TMAP
	TRESERVED
	TEXTENSIONS
	TENUM
	TSTREAM
	TGROUP
)

The result of Scan is one of these tokens.