scan

Documentation

Overview

Package scan wraps itself around text/scanner to provide a more compiler friendly interface. The intent is to provide a smaller snapshot of the scanning process to make the learning process more approachable at an early stage

Index

• type Item
• type Scanner
  • func (s *Scanner) Init(fileName string, r io.Reader)
  • func (s *Scanner) Scan()
• type Token
  • func Tokenize(s string) Token
  • func (t Token) String() string

Types

type Item

type Item struct {
	Lit string           // the literal, textual value of the lexem
	Pos scanner.Position // start position of the literal
	Tok Token            // the token representing the lexem
}

Item contains the literal-token pairs of each lexical element lexed by the scanner

type Scanner

type Scanner struct {
	scanner.Scanner
	Items chan *Item
}

Scanner embeds text/scanner.Scanner and provides facilities to send scanized lexems via the Items channel. Any errors are sent via the Errors channel

func (*Scanner) Init

func (s *Scanner) Init(fileName string, r io.Reader)

Init initializes the embedded scanner and initializes the Items and Errors channels. Init must be called prior to using any other functions else undefined behaviour or panics may occur

func (*Scanner) Scan

func (s *Scanner) Scan()

Scan is intended to be run in a goroutine but this isn't strictly necessary. Scan will loop until EOF is returned, sending results or errors via Items and Errors channels respectively

type Token

type Token int

Token represents valid tokens the scanner may emit

const (
	Invalid Token = iota // Invalid/Unknown character
	EOF                  // End-of-File
	Func                 // Function
	Ident                // Identifier
	LBrace               // "{"
	RBrace               // "}"
	LParen               // "("
	RParen               // ")"
)

List of valid token constants

func Tokenize

func Tokenize(s string) Token

func (Token) String

func (t Token) String() string

String returns a textual represented of a valid token