token

Documentation

Index

• func NewTokenizer(n Notifier) charm.State
• type Notifier
• type Pos
• type Signature
  • func (sig *Signature) Decoder() charm.State
  • func (sig *Signature) Pending() bool
• type Tokenizer
  • func (cfg Tokenizer) Decode() charm.State
• type Type
  • func (i Type) String() string

Functions

func NewTokenizer

func NewTokenizer(n Notifier) charm.State

Types

type Notifier

type Notifier interface {
	Decoded(Pos, Type, any) error
}

tbd: maybe a channel instead?

type Pos

type Pos struct{ X, Y int }

type Signature

type Signature struct {
	strings.Builder
	// contains filtered or unexported fields
}

parses a dictionary key of ascii words separated by, and terminating with, a colon. the words must start with a letter, but can contain spaces and underscores. ex. `a:`, `a:b:`, `and:more complex:keys_like_this:`

func (*Signature) Decoder

func (sig *Signature) Decoder() charm.State

first character of the signature must be a letter subsequent characters of words can be letters, numbers, spaces, or "connectors" (underscore) colons separate word parts

func (*Signature) Pending

func (sig *Signature) Pending() bool

type Tokenizer

type Tokenizer struct {
	Notifier Notifier
	// configure the upcoming Decode to produce only floating point numbers.
	// otherwise it will produce int for integers, and unit for hex specifications.
	UseFloats bool // controls number decoding
}

func (Tokenizer) Decode

func (cfg Tokenizer) Decode() charm.State

return a state to parse a stream of runes and notify as they are detected.

type Type

type Type int

const (
	Invalid Type = iota // placeholder, not generated by the tokenizer
	Array               // the value is the open or close rune
	Bool
	Comment // a completely empty comment is a blank line
	Key     // an empty key means a sequence; otherwise a mapping
	Number
	String
)

func (Type) String

func (i Type) String() string