tokenizer

Documentation

Overview

Package tokenizer converts LaTeX text input into a stream of tokens.

The package can perform macro expansion and has built-in knowledge about the macros and environments defined by different LaTeX classes and packages.

Index

• type Arg
  • func (arg *Arg) String() string
• type MissingEndError
  • func (err MissingEndError) Error() string
• type Token
  • func (tok *Token) String() string
• type TokenList
  • func (toks TokenList) FormatMaths() string
  • func (toks TokenList) FormatText() string
• type TokenType
• type Tokenizer
  • func NewTokenizer() *Tokenizer
  • func (p *Tokenizer) ParseTex(res chan<- *Token) error

Types

type Arg

type Arg struct {
	Optional bool
	Value    TokenList
}

Arg specifies a single macro argument.

func (*Arg) String

func (arg *Arg) String() string

type MissingEndError

type MissingEndError string

MissingEndError is used to indicate unterminated maths and tikz environments.

func (MissingEndError) Error

func (err MissingEndError) Error() string

type Token

type Token struct {
	// Type describes which kind of token this is.
	Type TokenType

	// For TokenMacro, this is the name of the macro, including the
	// leading backslash.  For most other token types, this is the
	// textual content of the token.
	Name string

	// For tokens of type TokenMacro, this field specifies the values
	// of the macro arguments.  Unused for all other token types.
	Args []*Arg
}

Token contains information about a single syntactic unit in the TeX source.

func (*Token) String

func (tok *Token) String() string

type TokenList

type TokenList []*Token

TokenList describes tokenized data in the argument of a macro call.

func (TokenList) FormatMaths

func (toks TokenList) FormatMaths() string

FormatMaths converts a TokenList to a string, assuming maths mode. Redundant spaces are omitted by this method.

func (TokenList) FormatText

func (toks TokenList) FormatText() string

FormatText converts a TokenList to a string.

type TokenType

type TokenType int

TokenType is used to enumerate different types of token

const (
	TokenMacro TokenType = iota
	TokenEmptyLine
	TokenComment
	TokenSpace
	TokenWord
	TokenOther
	TokenVerbatim
)

The different token types used by this package.

type Tokenizer

type Tokenizer struct {
	scanner.Scanner
	// contains filtered or unexported fields
}

A Tokenizer can be used to split a LaTeX file into syntactic units. User-defined macros are expanded in the process.

func NewTokenizer

func NewTokenizer() *Tokenizer

NewTokenizer creates and initialises a new Tokenizer.

func (*Tokenizer) ParseTex

func (p *Tokenizer) ParseTex(res chan<- *Token) error

ParseTex splits the Tokenizer's input into tokens and writes these tokens to the channel `res`.