lexer

Documentation

Index

• Variables
• func Lex(filename string, r io.Reader) lexer.Lexer
• func LexBytes(filename string, b []byte) lexer.Lexer
• func LexString(filename, s string) lexer.Lexer
• func LexWithScanner(filename string, scan *scanner.Scanner) lexer.Lexer
• func NewCustomTextScannerLexer() lexer.Definition

Variables

var (
	TextScannerLexer lexer.Definition = &textScannerLexerDefinition{}

	// DefaultDefinition defines properties for the default lexer.
	DefaultDefinition = TextScannerLexer
)

TextScannerLexer is a lexer that uses the text/scanner module.

Functions

func Lex

func Lex(filename string, r io.Reader) lexer.Lexer

Lex an io.Reader with text/scanner.Scanner.

This provides very fast lexing of source code compatible with Go tokens.

Note that this differs from text/scanner.Scanner in that string tokens will be unquoted.

func LexBytes

func LexBytes(filename string, b []byte) lexer.Lexer

LexBytes returns a new default lexer over bytes.

func LexString

func LexString(filename, s string) lexer.Lexer

LexString returns a new default lexer over a string.

func LexWithScanner

func LexWithScanner(filename string, scan *scanner.Scanner) lexer.Lexer

LexWithScanner creates a Lexer from a user-provided scanner.Scanner.

Useful if you need to customise the Scanner.

func NewCustomTextScannerLexer

func NewCustomTextScannerLexer() lexer.Definition

NewCustomTextScannerLexer constructs a Definition that uses an underlying scanner.Scanner

It's custom because: - string token can have different sets of quotes: single, double, backtick - special token type - JSON