token

Documentation

Overview

Package token defines constants representing the lexical tokens of the bibtex language and basic operations on tokens (printing, predicates).

Index

• type Token
  • func (tok Token) IsCommand() bool
  • func (tok Token) IsLiteral() bool
  • func (tok Token) IsOperator() bool
  • func (tok Token) IsStringLiteral() bool
  • func (tok Token) String() string

Types

type Token

type Token int

Token is the set of lexical tokens for bibtex.

const (
	Illegal Token = iota
	EOF
	TexComment // % foo

	Abbrev   // @STRING, @string
	Comment  // @COMMENT, @comment
	Preamble // @PREAMBLE, @pReAmble
	BibEntry // @article, @book, etc

	Ident       // author
	String      // "abc"
	BraceString // {abc}
	Number      // 2005

	DoubleQuote     // " - delimits a string
	StringLBrace    // {
	StringRBrace    // }
	StringSpace     // any consecutive whitespace '\n', '\r', '\t', ' ' in a bibtex string
	StringNBSP      // ~ - non-breakable space in LaTeX
	StringContents  // anything else inside a string
	StringHyphen    // - a hyphen counts as a token separator when parsing names
	StringMath      // $...$
	StringComma     // , - useful for parsing author names
	StringBackslash // \&, \$, \{, \_ - single character escape for special bibtex chars
	StringAccent    // \'{o} == ȯ, \ae == æ
	StringMacro     // \url, \(, \[, \, - TeX macro, either alphabetical or a single non-alphabetical char

	Assign // =
	LParen // (
	LBrace // {
	RParen // )
	RBrace // }
	Concat // #
	Comma  // ,

)

func (Token) IsCommand

func (tok Token) IsCommand() bool

IsCommand returns true for tokens corresponding to commands. It returns false otherwise.

func (Token) IsLiteral

func (tok Token) IsLiteral() bool

IsLiteral returns true for tokens corresponding to identifiers and basic type literals. It returns false otherwise.

func (Token) IsOperator

func (tok Token) IsOperator() bool

IsOperator returns true for tokens corresponding to operators and delimiters. It returns false otherwise.

func (Token) IsStringLiteral

func (tok Token) IsStringLiteral() bool

IsStringLiteral returns true for tokens corresponding to tokens within a string literal. It returns false otherwise.

func (Token) String

func (tok Token) String() string