scanner

Documentation

Index

• func Scan(s string, startPos ast.Pos) <-chan *Token
• type Peeker
  • func NewPeeker(ch <-chan *Token) *Peeker
  • func (p *Peeker) Close()
  • func (p *Peeker) Peek() *Token
  • func (p *Peeker) Read() *Token
• type Token
  • func (t *Token) String() string
• type TokenType
  • func (i TokenType) String() string

Functions

func Scan

func Scan(s string, startPos ast.Pos) <-chan *Token

Scan returns a channel that recieves Tokens from the given input string.

The scanner's job is just to partition the string into meaningful parts. It doesn't do any transformation of the raw input string, so the caller must deal with any further interpretation required, such as parsing INTEGER tokens into real ints, or dealing with escape sequences in LITERAL or STRING tokens.

Strings in the returned tokens are slices from the original string.

startPos should be set to ast.InitPos unless the caller knows that this interpolation string is part of a larger file and knows the position of the first character in that larger file.

Types

type Peeker

type Peeker struct {
	// contains filtered or unexported fields
}

Peeker is a utility that wraps a token channel returned by Scan and provides an interface that allows a caller (e.g. the parser) to work with the token stream in a mode that allows one token of lookahead, and provides utilities for more convenient processing of the stream.

func NewPeeker

func NewPeeker(ch <-chan *Token) *Peeker

func (*Peeker) Close

func (p *Peeker) Close()

Close ensures that the token stream has been exhausted, to prevent the goroutine in the underlying scanner from leaking.

It's not necessary to call this if the caller reads the token stream to EOF, since that implicitly closes the scanner.

func (*Peeker) Peek

func (p *Peeker) Peek() *Token

Peek returns the next token in the stream without consuming it. A subsequent call to Read will return the same token.

func (*Peeker) Read

func (p *Peeker) Read() *Token

Read consumes the next token in the stream and returns it.

type Token

type Token struct {
	Type    TokenType
	Content string
	Pos     ast.Pos
}

func (*Token) String

func (t *Token) String() string

type TokenType

type TokenType rune

const (
	// Raw string data outside of ${ .. } sequences
	LITERAL TokenType = 'o'

	// STRING is like a LITERAL but it's inside a quoted string
	// within a ${ ... } sequence, and so it can contain backslash
	// escaping.
	STRING TokenType = 'S'

	// Other Literals
	INTEGER TokenType = 'I'
	FLOAT   TokenType = 'F'
	BOOL    TokenType = 'B'

	BEGIN    TokenType = '$' // actually "${"
	END      TokenType = '}'
	OQUOTE   TokenType = '“' // Opening quote of a nested quoted sequence
	CQUOTE   TokenType = '”' // Closing quote of a nested quoted sequence
	OPAREN   TokenType = '('
	CPAREN   TokenType = ')'
	OBRACKET TokenType = '['
	CBRACKET TokenType = ']'
	COMMA    TokenType = ','

	IDENTIFIER TokenType = 'i'

	PERIOD  TokenType = '.'
	PLUS    TokenType = '+'
	MINUS   TokenType = '-'
	STAR    TokenType = '*'
	SLASH   TokenType = '/'
	PERCENT TokenType = '%'

	AND  TokenType = '∧'
	OR   TokenType = '∨'
	BANG TokenType = '!'

	EQUAL    TokenType = '='
	NOTEQUAL TokenType = '≠'
	GT       TokenType = '>'
	LT       TokenType = '<'
	GTE      TokenType = '≥'
	LTE      TokenType = '≤'

	QUESTION TokenType = '?'
	COLON    TokenType = ':'

	EOF TokenType = '␄'

	// Produced for sequences that cannot be understood as valid tokens
	// e.g. due to use of unrecognized punctuation.
	INVALID TokenType = '�'
)

func (TokenType) String

func (i TokenType) String() string