shell

Documentation

Index

• func Split(s string) ([]string, error)
• type Lexer
  • func NewLexer(r io.Reader) *Lexer
  • func (l *Lexer) Next() ([]rune, bool, error)
• type TokenType
• type Tokenizer
  • func NewTokenizer(r io.Reader) *Tokenizer
  • func (t *Tokenizer) Next() (TokenType, []rune, bool, error)

Functions

func Split

func Split(s string) ([]string, error)

Split partitions a string into a slice of strings.

Types

type Lexer

type Lexer Tokenizer

Lexer turns an input stream into a sequence of tokens. Whitespace and comments are skipped.

func NewLexer

func NewLexer(r io.Reader) *Lexer

NewLexer creates a new lexer from an input stream.

func (*Lexer) Next

func (l *Lexer) Next() ([]rune, bool, error)

Next returns the next word, or an error. If there are no more words, the error will be io.EOF.

type TokenType

type TokenType int

TokenType is a top-level token classification: A word, space, comment, unknown.

const (
	UnknownToken TokenType = iota
	WordToken
	SpaceToken
	CommentToken
)

Classes of lexographic token

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer turns an input stream into a sequence of typed tokens

func NewTokenizer

func NewTokenizer(r io.Reader) *Tokenizer

NewTokenizer creates a new tokenizer from an input stream.

func (*Tokenizer) Next

func (t *Tokenizer) Next() (TokenType, []rune, bool, error)

Next returns the next token in the stream.