token

Documentation

Overview

Package token contains token types and a token stream implementation that can be used by lexers and parsers.

Index

• Constants
• type Stream
  • func NewBufferedStream(bufferSize int) *Stream
  • func NewStream() *Stream
  • func (s *Stream) Close()
  • func (s *Stream) Next() (Token, bool)
  • func (s *Stream) Push(t Token)
  • func (s *Stream) Tokens() <-chan Token
• type Token
  • func New(t Type, v string, start, length int) Token
  • func (t Token) String() string
• type Type
  • func (i Type) String() string

Constants

const (
	// DefaultStreamBufferSize is the default channel buffer size of the
	// underlying channel in a token stream.
	DefaultStreamBufferSize = 5
)

Types

type Stream

type Stream struct {
	// contains filtered or unexported fields
}

Stream is a token stream that allows tokens to be pushed. It uses a buffered channel as underlying queue.

_ = NewStream()           // channel buffer size = 5 (DefaultStreamBufferSize)
_ = NewBufferedStream(10) // channel buffer size = 10

func NewBufferedStream

func NewBufferedStream(bufferSize int) *Stream

NewBufferedStream returns a new token stream with the given buffer size.

func NewStream

func NewStream() *Stream

NewStream returns a new token stream with the default buffer size.

func (*Stream) Close

func (s *Stream) Close()

Close closes the underlying channel of this token stream.

func (*Stream) Next

func (s *Stream) Next() (Token, bool)

Next returns the next token that was pushed onto the stream. This is a common channel receive operation.

func (*Stream) Push

func (s *Stream) Push(t Token)

Push will push the given token onto the token stream. This is a common channel send operation.

func (*Stream) Tokens

func (s *Stream) Tokens() <-chan Token

Tokens returns a channel where all tokens of this stream can be read from (once). Multiple calls to Tokens() will return the same channel.

type Token

type Token struct {
	Type  Type
	Value string
	Pos   int
}

Token represents a token with a type and a value, which is represented as string. It also holds its starting position in the source.

func New

func New(t Type, v string, start, length int) Token

New creates a new token with the given type, value, start position and token length.

func (Token) String

func (t Token) String() string

type Type

type Type uint64

Type is a type of a token.

const (
	Unknown Type = iota

	LineTerminator // 11.3
	Whitespace     // 11.2

	MultiLineComment  // 11.4
	SingleLineComment // 11.4

	CommonToken    // 11.5
	IdentifierName // 11.5, 11.6
	Punctuator     // 11.5, 11.7
	// NumericLiteral // 11.5
	StringLiteral // 11.5
	// Template       // 11.5
	TemplateHead
	TemplateMiddle
	TemplateTail
	NoSubstitutionTemplate

	Async
	Let

	Asterisk
	Await    // 11.6.2.1
	Break    // 11.6.2.1
	Case     // 11.6.2.1
	Catch    // 11.6.2.1
	Class    // 11.6.2.1
	Const    // 11.6.2.1
	Continue // 11.6.2.1
	Debugger // 11.6.2.1
	Default  // 11.6.2.1
	Delete   // 11.6.2.1
	Do       // 11.6.2.1
	Else     // 11.6.2.1
	Export   // 11.6.2.1
	Extends  // 11.6.2.1
	Finally  // 11.6.2.1
	For      // 11.6.2.1
	Function // 11.6.2.1
	Get
	If         // 11.6.2.1
	Import     // 11.6.2.1
	In         // 11.6.2.1
	Instanceof // 11.6.2.1
	NewT       // 11.6.2.1
	Return     // 11.6.2.1
	Set
	Static
	Super  // 11.6.2.1
	Switch // 11.6.2.1
	This   // 11.6.2.1
	Throw  // 11.6.2.1
	Try    // 11.6.2.1
	Typeof // 11.6.2.1
	Var    // 11.6.2.1
	Void   // 11.6.2.1
	While  // 11.6.2.1
	With   // 11.6.2.1
	Yield  // 11.6.2.1

	Enum
	Implements
	Package
	Protected
	Interface
	Private
	Public

	Null                 // 11.8.1
	Boolean              // 11.8.2
	DecimalLiteral       // 11.8.3
	BinaryIntegerLiteral // 11.8.3
	OctalIntegerLiteral  // 11.8.3
	HexIntegerLiteral    // 11.8.3
	SignedInteger        // 11.8.3

	RegularExpressionBody // 11.8.5
	RegularExpressionFlags

	Target

	AndAssign
	Arrow
	Assign
	BitwiseAnd
	BitwiseNot
	BitwiseOr
	BitwiseXor
	BraceClose
	BraceOpen
	BracketClose
	BracketOpen
	Colon
	Comma
	// Div =  Slash
	DivAssign
	Dot
	Ellipsis
	Equals
	ExclamationMark
	GreaterThan
	GreaterThanOrEqualTo
	LeftShift
	LeftShiftAssign
	LessThan
	LessThanOrEqualTo
	LogicalAnd
	// LogicalNot = ExclamationMark
	LogicalOr
	Minus
	MinusAssign
	Modulo
	ModuloAssign
	// Multiply = Asterisk
	MultiplyAssign
	NotEquals
	OrAssign
	ParClose
	ParOpen
	Plus
	PlusAssign
	Power
	PowerAssign
	QuestionMark
	RightShift
	RightShiftAssign
	SemiColon
	Slash
	StrictEquals
	StrictNotEquals
	Tilde
	UnsignedRightShift
	UnsignedRightShiftAssign
	UpdateMinus
	UpdatePlus
	XorAssign
)

Available token type.

func (Type) String

func (i Type) String() string