tokens

Documentation

Index

• type Char
  • func NewChar(line, column, size int, rune rune) Char
  • func (p Char) At() (line, column int)
  • func (p Char) Is(r rune) bool
  • func (p Char) IsOneOf(runes ...rune) bool
• type Token
  • func NewToken(t TokenType, l string) *Token
  • func (t *Token) DebugString() string
  • func (t *Token) From() (line, column int)
  • func (t *Token) IsLiteral(lit string) bool
  • func (t *Token) IsOneOfLiterals(lits ...string) bool
  • func (t *Token) IsOneOfTypes(tys ...TokenType) bool
  • func (t *Token) IsType(ty TokenType) bool
  • func (t *Token) Range() (fromLine, fromColumn, toLine, toColumn int)
  • func (t *Token) RangeColumns() (from, to int)
  • func (t *Token) RangeLines() (from, to int)
  • func (t *Token) To() (line, column int)
  • func (t Token) WithLiteral(lit string) *Token
  • func (t Token) WithRange(fromLine, fromColumn, toLine, toColumn int) *Token
  • func (t Token) WithRangeChars(from, to Char) *Token
  • func (t Token) WithType(tp TokenType) *Token
• type TokenType

Types

type Char

type Char struct {
	Rune   rune
	Size   int // Size in bytes
	Line   int
	Column int
}

Represents a character in the source code, with its position.

func NewChar

func NewChar(line, column, size int, rune rune) Char

Creates a new Char at line and column with the given rune and size.

func (Char) At

func (p Char) At() (line, column int)

Returns the line and column of the character.

func (Char) Is

func (p Char) Is(r rune) bool

Checks if the character rune matches the given rune.

func (Char) IsOneOf

func (p Char) IsOneOf(runes ...rune) bool

Checks if the character rune is one of the given runes.

type Token

type Token struct {
	Type       TokenType
	Literal    string
	FromLine   int
	FromColumn int
	ToLine     int
	ToColumn   int
}

Represents a token in the source code.

tokens.NewToken(EOF, "").WithRangeChars(from, to)

func NewToken

func NewToken(t TokenType, l string) *Token

Creates a new token.

func (*Token) DebugString

func (t *Token) DebugString() string

Pretty string representation of the token.

func (*Token) From

func (t *Token) From() (line, column int)

Returns the initial position of the token.

func (*Token) IsLiteral

func (t *Token) IsLiteral(lit string) bool

Returns true if the token is the given literal.

func (*Token) IsOneOfLiterals

func (t *Token) IsOneOfLiterals(lits ...string) bool

Returns true if the token is one of the given literals.

func (*Token) IsOneOfTypes

func (t *Token) IsOneOfTypes(tys ...TokenType) bool

Returns true if the token is of the given type.

func (*Token) IsType

func (t *Token) IsType(ty TokenType) bool

Returns true if the token is of the given type.

func (*Token) Range

func (t *Token) Range() (fromLine, fromColumn, toLine, toColumn int)

Returns the range of the token.

func (*Token) RangeColumns

func (t *Token) RangeColumns() (from, to int)

Returns the range of the token ignoring the lines. Starting at 1.

func (*Token) RangeLines

func (t *Token) RangeLines() (from, to int)

Returns the range of the starting and final lines. Starting at 1.

func (*Token) To

func (t *Token) To() (line, column int)

Returns the final position of the token.

func (Token) WithLiteral

func (t Token) WithLiteral(lit string) *Token

Creates a new token with the given literal. You can use this method as a chain.

func (Token) WithRange

func (t Token) WithRange(fromLine, fromColumn, toLine, toColumn int) *Token

Creates a new token with the given range. You can use this method as a chain.

func (Token) WithRangeChars

func (t Token) WithRangeChars(from, to Char) *Token

Creates a new token with the given char range. You can use this method as a chain.

func (Token) WithType

func (t Token) WithType(tp TokenType) *Token

Creates a new token with the given type. You can use this method as a chain.

type TokenType

type TokenType string

const (
	UNKNOWN TokenType = "unknown"
	INVALID TokenType = "invalid"
	EOF     TokenType = "eof"
)