lexer

Documentation

Overview

Package lexer is based on Rop Pike's talk: https://www.youtube.com/watch?v=HxaD_trXwRE

Index

• type Lexer
  • func Lex(input string) Lexer
• type Pos
• type Token
  • func (t Token) IsDelimiter() bool
  • func (t Token) IsKeyword() bool
  • func (t Token) IsLiteral() bool
• type TokenType
  • func (tt TokenType) String() string

Types

type Lexer

type Lexer interface {
	// Next returns the next available Token from the lexer.
	// If no more Tokens are available or the lexer was closed,
	// the Token has type EOF.
	Next() Token
}

The Lexer interface represents a lexer that lexes its input into Tokens.

func Lex

func Lex(input string) Lexer

Lex concurrently starts lexing the given input and returns the associated Lexer instance.

type Pos

type Pos struct {
	Line   int
	Column int
}

Pos describes a position in a text file.

type Token

type Token struct {
	Type  TokenType
	Value string
	Pos   Pos
}

Token is an item returned from the scanner.

func (Token) IsDelimiter

func (t Token) IsDelimiter() bool

IsDelimiter returns true, if the token has type delimiter.

func (Token) IsKeyword

func (t Token) IsKeyword() bool

IsKeyword returns true, if the token has type keyword.

func (Token) IsLiteral

func (t Token) IsLiteral() bool

IsLiteral returns true, if the token has type literal.

type TokenType

type TokenType int

TokenType identifies the type of lex tokens.

const (
	ILLEGAL TokenType = iota
	EOF

	IDENT     // sendData
	INT       // 123
	RAWSTRING // `rawstring`
	BYTESIZE  // 1MB
	DURATION  // 80ns

	VERSION
	ERRORS
	ENUM
	TYPE
	SERVICE
	CALL
	STREAM
	ASYNC
	ARG
	RET
	MAXARGSIZE
	MAXRETSIZE
	TIMEOUT
	MAP

	LPAREN // (
	RPAREN // )
	LBRACE // {
	RBRACE // }
	LBRACK // [
	RBRACK // ]
	COLON  // :
	EQUAL  // =
	COMMA  // ,

)

func (TokenType) String

func (tt TokenType) String() string