token

Documentation

Index

• func NewStream() *stream
• type Stream
• type Token
  • func New(t Type, v string, pos int) Token
  • func (t Token) String() string
• type Type

Functions

func NewStream

func NewStream() *stream

NewStream creates a new, ready to use token stream with a buffer size of 5.

Types

type Stream

type Stream interface {
	// Tokens returns a channel of Tokens, that can be read from.
	Tokens() <-chan Token
	// Push pushes a token onto this stream. It will be accessible on the
	// channel retrieved with Tokens() eventually.
	Push(Token)
	// Close closes this stream, and also closes the channel retrieved with
	// Tokens().
	Close()
}

Stream is an interface providing methods for a usable token stream.

type Token

type Token struct {
	Type  Type
	Value string
	Pos   int
}

Token is a struct that holds a token type (defined by you), a value (the runes that were consumed by the lexer as string) and a position (where the token starts).

func New

func New(t Type, v string, pos int) Token

New creates a new Token with the given Type, Value and Position.

func (Token) String

func (t Token) String() string

String returns a string representation of the form

Type.Name() + "(" + Value + "), " + Pos

For example

Number(7.5), pos 9

type Type

type Type interface {
	Name() string
	Value() uint64
}

Type is an interface that you custom token types must implement in order to be used with this lexer.