scan

Documentation

Overview

Package scan implements a scanner for Goal source text.

Index

• func GetLineCol(s string, pos int) (string, int, int)
• func IsAlphaNum(r rune) bool
• func IsLetter(r rune) bool
• type IdentType
• type Scanner
  • func New(src string, names map[string]IdentType, shebang bool) *Scanner
  • func (s *Scanner) AfterNewline() bool
  • func (s *Scanner) AfterNoun() bool
  • func (s *Scanner) AfterOpen() bool
  • func (s *Scanner) AtExprStart() bool
  • func (s *Scanner) Next() Token
  • func (s *Scanner) Source() string
• type Token
  • func (t Token) String() string
• type Type
  • func (i Type) String() string

Functions

func GetLineCol

func GetLineCol(s string, pos int) (string, int, int)

GetLineCol returns the line text, line number and column number of a given position in a source string. This can be used to retrieve human-readable position information given a token Pos field.

func IsAlphaNum

func IsAlphaNum(r rune) bool

IsAlphaNum returns true if the rune is a letter or a digit (any non-first character of an identifier).

func IsLetter

func IsLetter(r rune) bool

IsLetter returns true if the rune is a letter (possible first character of an identifier).

Types

type IdentType

type IdentType int

identType represents the different kinds of special roles for alphanumeric identifiers that act as keywords.

const (
	IdentVar   IdentType = iota // a variable identifier (default zero value)
	IdentMonad                  // a builtin monad (cannot have left argument)
	IdentDyad                   // a builtin dyad (can have left argument)
)

These constants represent the different kinds of identifiers.

type Scanner

type Scanner struct {
	Comments bool // output Comment tokens too (default: false)
	// contains filtered or unexported fields
}

Scanner represents the state of the scanner.

func New

func New(src string, names map[string]IdentType, shebang bool) *Scanner

New returns a scanner for the given source string, with the given monadic and dyadic keyword mappings. If shebang is true, the scanner ignores the starting line if it starts with #!.

func (*Scanner) AfterNewline

func (s *Scanner) AfterNewline() bool

AfterNewline returns true if last read non-comment token was a newline.

func (*Scanner) AfterNoun

func (s *Scanner) AfterNoun() bool

AfterNoun returns true if last read non-comment token ended a noun.

func (*Scanner) AfterOpen

func (s *Scanner) AfterOpen() bool

AfterOpen returns true if last read non-comment token was an opening brace, bracket or paren.

func (*Scanner) AtExprStart

func (s *Scanner) AtExprStart() bool

AtExprStart returns true if the scanner expects next a new (sub)expression. This means last non-comment token was an opening brace, bracket or paren, a newline, semicolon or special token.

func (*Scanner) Next

func (s *Scanner) Next() Token

Next produces the next token from the input source.

func (*Scanner) Source

func (s *Scanner) Source() string

Source returns the text being scanned, as provided to New.

type Token

type Token struct {
	Type Type // token type
	Pos  int  // offset of token's first character
	End  int  // offset of first character after token

	// Content text. It may differ from the exact token text in the source:
	// delimiter token types have an empty Text field, and Ident, String
	// and Regexp types can go through some source processing (like
	// handling some escapes or changing/removing delimiters) before the
	// content is stored in the Text field.
	Text string
}

Token represents a token information returned from the scanner.

func (Token) String

func (t Token) String() string

String returns a string representation of the token for debugging purposes.

type Type

type Type int

tokenType represents the different kinds of tokens.

const (
	EOF   Type = iota
	Error      // syntax error: Text contains message

	Adverb         // ' / \ (alone or following tightly expression)
	Comment        // comments /.* or multi-line /.*\ (without final newline)
	Dyad           // : + * - or dyadic keyword
	DyadAssign     // +: *: -: (only for builtin dyadic symbols)
	Var            // alphanumeric variable var (also from $var ${var})
	VarField       // var..field
	LeftBrace      // {
	LeftBracket    // [ (sequences and lambda arguments)
	LeftBracketIdx // [ (indexing)
	LeftParen      // (
	Monad          // monadic keyword
	Newline        // \n (except duplicates and after opening delimiter)
	Number         // literal number (integer or float)
	QqEnd          // " or / (or other delimiters)
	QqStart        // " or qq/ (or other delimiters)
	Regexp         // escaped PATTERN in rx/PATTERN/ (or other delimiters)
	RightBrace     // }
	RightBracket   // ]
	RightParen     // )
	Semicolon      // ;
	Special        // non-adverbial ' \ (return early on error & rt.log)
	String         // `string` or properly escaped "string" (from rq// too)
)

These constants describe the possible kinds of tokens.

func (Type) String

func (i Type) String() string