token

Documentation

Overview

Package token provides generated internal tokenizing functions for DOT parsing.

Index

• Variables
• type Context
• type Pos
  • func (p Pos) String() string
• type Sourcer
• type Token
  • func (t *Token) CharLiteralValue() string
  • func (t *Token) Equals(rhs interface{}) bool
  • func (t *Token) Float32Value() (float32, error)
  • func (t *Token) Float64Value() (float64, error)
  • func (t *Token) IDValue() string
  • func (t *Token) Int32Value() (int32, error)
  • func (t *Token) Int64Value() (int64, error)
  • func (t *Token) StringValue() string
  • func (t *Token) UTF8Rune() (rune, error)
• type TokenMap
  • func (m TokenMap) Id(tok Type) string
  • func (m TokenMap) StringType(typ Type) string
  • func (m TokenMap) TokenString(tok *Token) string
  • func (m TokenMap) Type(tok string) Type
• type Type

Variables

var TokMap = TokenMap{
	// contains filtered or unexported fields
}

Types

type Context

type Context interface{}

Context allows user-defined data to be associated with the lexer/scanner to be associated with each token that lexer produces.

type Pos

type Pos struct {
	Offset  int
	Line    int
	Column  int
	Context Context
}

func (Pos) String

func (p Pos) String() string

type Sourcer

type Sourcer interface {
	Source() string
}

Sourcer is a Context interface which presents a Source() method identifying e.g the filename for the current code.

type Token

type Token struct {
	Type
	Lit []byte
	Pos
}

func (*Token) CharLiteralValue

func (t *Token) CharLiteralValue() string

CharLiteralValue returns the string value of the char literal.

func (*Token) Equals

func (t *Token) Equals(rhs interface{}) bool

Equals returns returns true if the token Type and Lit are matches.

func (*Token) Float32Value

func (t *Token) Float32Value() (float32, error)

Float32Value returns the float32 value of the token or an error if the token literal does not denote a valid float32.

func (*Token) Float64Value

func (t *Token) Float64Value() (float64, error)

Float64Value returns the float64 value of the token or an error if the token literal does not denote a valid float64.

func (*Token) IDValue

func (t *Token) IDValue() string

IDValue returns the string representation of an identifier token.

func (*Token) Int32Value

func (t *Token) Int32Value() (int32, error)

Int32Value returns the int32 value of the token or an error if the token literal does not denote a valid float64.

func (*Token) Int64Value

func (t *Token) Int64Value() (int64, error)

Int64Value returns the int64 value of the token or an error if the token literal does not denote a valid float64.

func (*Token) StringValue

func (t *Token) StringValue() string

StringValue returns the string value of the token literal.

func (*Token) UTF8Rune

func (t *Token) UTF8Rune() (rune, error)

UTF8Rune decodes the UTF8 rune in the token literal. It returns utf8.RuneError if the token literal contains an invalid rune.

type TokenMap

type TokenMap struct {
	// contains filtered or unexported fields
}

func (TokenMap) Id

func (m TokenMap) Id(tok Type) string

func (TokenMap) StringType

func (m TokenMap) StringType(typ Type) string

func (TokenMap) TokenString

func (m TokenMap) TokenString(tok *Token) string

func (TokenMap) Type

func (m TokenMap) Type(tok string) Type

type Type

type Type int

const (
	INVALID Type = iota
	EOF
)