toki

README

toki

Regexp tokenizer/scanner in Go.

The scanner takes in a list of token definitions and string input. it is general and easy to use.

token definition

a token is a pair of constant and regexp string.

Documentation

Index

• type Def
• type Position
• type Result
  • func (this *Result) String() string
• type Scanner
  • func NewScanner(def []Def) *Scanner
  • func (this *Scanner) Next() *Result
  • func (this *Scanner) Peek() *Result
  • func (this *Scanner) SetInput(input string)
• type Token

Types

type Def

type Def struct {
	Token   Token
	Pattern string
	// contains filtered or unexported fields
}

type Position

type Position struct {
	Line   int
	Column int
}

type Result

type Result struct {
	Token Token
	Value []byte
	Pos   Position
}

func (*Result) String

func (this *Result) String() string

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

func NewScanner

func NewScanner(def []Def) *Scanner

func (*Scanner) Next

func (this *Scanner) Next() *Result

func (*Scanner) Peek

func (this *Scanner) Peek() *Result

func (*Scanner) SetInput

func (this *Scanner) SetInput(input string)

type Token

type Token uint32

const (
	EOF Token = 1<<32 - 1 - iota
	Error
)