Documentation
¶
Overview ¶
Package token defines a complete set of all lexical tokens for any kind of language! It is based on the alecthomas/chroma / pygments lexical tokens plus all the more detailed tokens needed for actually parsing languages
Index ¶
- Variables
- func InitCatMap()
- func InitSubCatMap()
- type KeyToken
- type KeyTokenList
- type Tokens
- func (tk Tokens) Cat() Tokens
- func (tk Tokens) ClassName() string
- func (tk Tokens) CombineRepeats() bool
- func (i *Tokens) FromString(s string) error
- func (tk Tokens) IconName() string
- func (tk Tokens) InCat(other Tokens) bool
- func (tk Tokens) InSubCat(other Tokens) bool
- func (tk Tokens) IsAmbigUnaryOp() bool
- func (tk Tokens) IsCat() bool
- func (tk Tokens) IsKeyword() bool
- func (tk Tokens) IsPunctGpLeft() bool
- func (tk Tokens) IsPunctGpRight() bool
- func (tk Tokens) IsSubCat() bool
- func (tk Tokens) IsUnaryOp() bool
- func (ev Tokens) MarshalJSON() ([]byte, error)
- func (ev Tokens) MarshalText() ([]byte, error)
- func (tk Tokens) Match(otk Tokens) bool
- func (tk Tokens) Parent() Tokens
- func (tk Tokens) PunctGpMatch() Tokens
- func (i Tokens) String() string
- func (tk Tokens) StyleName() string
- func (tk Tokens) SubCat() Tokens
- func (ev *Tokens) UnmarshalJSON(b []byte) error
- func (ev *Tokens) UnmarshalText(b []byte) error
Constants ¶
This section is empty.
Variables ¶
var CatMap map[Tokens]Tokens
CatMap is the map into the category level for each token
var Cats = []Tokens{ None, Keyword, Name, Literal, Operator, Punctuation, Comment, Text, TokensN, }
Categories
var KeyTokenZero = KeyToken{}
var Names = map[Tokens]string{}/* 125 elements not displayed */
Names are the short tag names for each token, used e.g., for syntax highlighting These are based on alecthomas/chroma / pygments
var OpPunctMap = map[string]Tokens{ "+": OpMathAdd, "-": OpMathSub, "*": OpMathMul, "/": OpMathDiv, "%": OpMathRem, "&": OpBitAnd, "|": OpBitOr, "~": OpBitNot, "^": OpBitXor, "<<": OpBitShiftLeft, ">>": OpBitShiftRight, "&^": OpBitAndNot, "=": OpAsgnAssign, "++": OpAsgnInc, "--": OpAsgnDec, "<-": OpAsgnArrow, ":=": OpAsgnDefine, "+=": OpMathAsgnAdd, "-=": OpMathAsgnSub, "*=": OpMathAsgnMul, "/=": OpMathAsgnDiv, "%=": OpMathAsgnRem, "&=": OpBitAsgnAnd, "|=": OpBitAsgnOr, "^=": OpBitAsgnXor, "<<=": OpBitAsgnShiftLeft, ">>=": OpBitAsgnShiftRight, "&^=": OpBitAsgnAndNot, "&&": OpLogAnd, "||": OpLogOr, "!": OpLogNot, "==": OpRelEqual, "!=": OpRelNotEqual, "<": OpRelLess, ">": OpRelGreater, "<=": OpRelLtEq, ">=": OpRelGtEq, "...": OpListEllipsis, "(": PunctGpLParen, ")": PunctGpRParen, "[": PunctGpLBrack, "]": PunctGpRBrack, "{": PunctGpLBrace, "}": PunctGpRBrace, ",": PunctSepComma, ".": PunctSepPeriod, ";": PunctSepSemicolon, ":": PunctSepColon, "\"": PunctStrDblQuote, "'": PunctStrQuote, "`": PunctStrBacktick, "\\": PunctStrEsc, }
OpPunctMap provides a lookup of operators and punctuation tokens by their usual string representation
var SubCatMap map[Tokens]Tokens
SubCatMap is the map into the sub-category level for each token
var SubCats = []Tokens{ None, Keyword, Name, NameType, NameFunction, NameScope, NameVar, NameValue, Literal, LitStr, LitNum, Operator, OpMath, OpBit, OpAsgn, OpMathAsgn, OpBitAsgn, OpLog, OpRel, OpList, Punctuation, PunctGp, PunctSep, PunctStr, Comment, CommentPreproc, Text, TextStyle, TokensN, }
Sub-Categories
Functions ¶
Types ¶
type KeyToken ¶
KeyToken combines a token and an optional keyword name for Keyword token types if Tok is in Keyword category, then Key string can be used to check for same keyword. Also has a Depth for matching against a particular nesting depth
func (KeyToken) Equal ¶
Equal compares equality of two tokens including keywords if token is in Keyword category. See also Match for version that uses category / subcategory matching
func (KeyToken) Match ¶
Match compares equality of two tokens including keywords if token is in Keyword category. returns true if the two tokens match, in a category / subcategory sensitive manner: if receiver token is a category, then it matches other token if it is the same category and likewise for subcategory
func (KeyToken) MatchDepth ¶
MatchDepth compares equality of two tokens including depth -- see Match for other matching criteria
type KeyTokenList ¶
type KeyTokenList []KeyToken
KeyTokenList is a list (slice) of KeyTokens
func (KeyTokenList) Match ¶
func (kl KeyTokenList) Match(okt KeyToken) bool
Match returns true if given keytoken matches any of the items on the list
type Tokens ¶
type Tokens int
Tokens is a complete set of lexical tokens that encompasses all programming and text markup languages. It includes everything in alecthomas/chroma (pygments) and everything needed for Go, C, C++, Python, etc.
There are categories and sub-categories, and methods to get those from a given element. The first category is 'None'.
See http://pygments.org/docs/tokens/ for more docs on the different categories
Anything missing should be added via a pull request etc
const ( // None is the nil token value -- for non-terminal cases or TBD None Tokens = iota // Error is an input that could not be tokenized due to syntax error etc Error // EOF is end of file EOF // EOL is end of line (typically implicit -- used for rule matching) EOL // EOS is end of statement -- a key meta-token -- in C it is ;, in Go it is either ; or EOL EOS // Background is for syntax highlight styles based on these tokens Background // Cat: Keywords (actual keyword is just the string) Keyword KeywordConstant KeywordDeclaration KeywordNamespace // incl package, import KeywordPseudo KeywordReserved KeywordType // Cat: Names. Name NameBuiltin // e.g., true, false -- builtin values.. NameBuiltinPseudo // e.g., this, self NameOther NamePseudo // SubCat: Type names NameType NameClass NameStruct NameField NameInterface NameConstant NameEnum NameEnumMember NameArray // includes slice etc NameMap NameObject NameTypeParam // for generics, templates // SubCat: Function names NameFunction NameDecorator // function-like wrappers in python NameFunctionMagic // e.g., __init__ in python NameMethod NameOperator NameConstructor // includes destructor.. NameException NameLabel // e.g., goto label NameEvent // for LSP -- not really sure what it is.. // SubCat: Scoping names NameScope NameNamespace NameModule NamePackage NameLibrary // SubCat: NameVar -- variable names NameVar NameVarAnonymous NameVarClass NameVarGlobal NameVarInstance NameVarMagic NameVarParam // SubCat: Value -- data-like elements NameValue NameTag // e.g., HTML tag NameProperty NameAttribute // e.g., HTML attr NameEntity // special entities. (e.g. in HTML). seems like other.. // Cat: Literals. Literal LiteralDate LiteralOther LiteralBool // SubCat: Literal Strings. LitStr LitStrAffix // unicode specifiers etc LitStrAtom LitStrBacktick LitStrBoolean LitStrChar LitStrDelimiter LitStrDoc // doc-specific strings where syntactically noted LitStrDouble LitStrEscape // esc sequences within strings LitStrHeredoc // in ruby, perl LitStrInterpol // interpolated parts of strings in #{foo} in Ruby LitStrName LitStrOther LitStrRegex LitStrSingle LitStrSymbol LitStrFile // filename // SubCat: Literal Numbers. LitNum LitNumBin LitNumFloat LitNumHex LitNumInteger LitNumIntegerLong LitNumOct LitNumImag // Cat: Operators. Operator OperatorWord // SubCat: Math operators OpMath OpMathAdd // + OpMathSub // - OpMathMul // * OpMathDiv // / OpMathRem // % // SubCat: Bitwise operators OpBit OpBitAnd // & OpBitOr // | OpBitNot // ~ OpBitXor // ^ OpBitShiftLeft // << OpBitShiftRight // >> OpBitAndNot // &^ // SubCat: Assign operators OpAsgn OpAsgnAssign // = OpAsgnInc // ++ OpAsgnDec // -- OpAsgnArrow // <- OpAsgnDefine // := // SubCat: Math Assign operators OpMathAsgn OpMathAsgnAdd // += OpMathAsgnSub // -= OpMathAsgnMul // *= OpMathAsgnDiv // /= OpMathAsgnRem // %= // SubCat: Bitwise Assign operators OpBitAsgn OpBitAsgnAnd // &= OpBitAsgnOr // |= OpBitAsgnXor // ^= OpBitAsgnShiftLeft // <<= OpBitAsgnShiftRight // >>= OpBitAsgnAndNot // &^= // SubCat: Logical operators OpLog OpLogAnd // && OpLogOr // || OpLogNot // ! // SubCat: Relational operators OpRel OpRelEqual // == OpRelNotEqual // != OpRelLess // < OpRelGreater // > OpRelLtEq // <= OpRelGtEq // >= // SubCat: List operators OpList OpListEllipsis // ... // Cat: Punctuation. Punctuation // SubCat: Grouping punctuation PunctGp PunctGpLParen // ( PunctGpRParen // ) PunctGpLBrack // [ PunctGpRBrack // ] PunctGpLBrace // { PunctGpRBrace // } // SubCat: Separator punctuation PunctSep PunctSepComma // , PunctSepPeriod // . PunctSepSemicolon // ; PunctSepColon // : // SubCat: String punctuation PunctStr PunctStrDblQuote // " PunctStrQuote // ' PunctStrBacktick // ` PunctStrEsc // \ // Cat: Comments. Comment CommentHashbang CommentMultiline CommentSingle CommentSpecial // SubCat: Preprocessor "comments". CommentPreproc CommentPreprocFile // Cat: Text. Text TextWhitespace TextSymbol TextPunctuation TextSpellErr // SubCat: TextStyle (corresponds to Generic in chroma / pygments) todo: look in font deco for more TextStyle TextStyleDeleted // strike-through TextStyleEmph // italics TextStyleError TextStyleHeading TextStyleInserted TextStyleOutput TextStylePrompt TextStyleStrong // bold TextStyleSubheading TextStyleTraceback TextStyleUnderline TextStyleLink TokensN )
The list of tokens
func (Tokens) ClassName ¶
ClassName returns the . prefixed CSS classname of the tag style for styling, a CSS property should exist with this name
func (Tokens) CombineRepeats ¶
CombineRepeats are token types where repeated tokens of the same type should be combined together -- literals, comments, text
func (*Tokens) FromString ¶
func (Tokens) IconName ¶ added in v0.5.5
IconName returns the appropriate icon name for the type of lexical item this is
func (Tokens) IsAmbigUnaryOp ¶
IsAmbigUnaryOp returns true if this token is an operator that could either be a Unary or Binary operator -- need special matching for this. includes * and & which are used for address operations in C-like languages
func (Tokens) IsPunctGpLeft ¶
IsPunctGpLeft returns true if token is a PunctGpL token -- left paren, brace, bracket
func (Tokens) IsPunctGpRight ¶
IsPunctGpRight returns true if token is a PunctGpR token -- right paren, brace, bracket
func (Tokens) IsUnaryOp ¶ added in v0.5.5
IsUnaryOp returns true if this token is an operator that is typically used as a Unary operator: - + & * ! ^ ! <-
func (Tokens) MarshalJSON ¶
func (Tokens) MarshalText ¶
map keys require text marshaling:
func (Tokens) Match ¶
Match returns true if the two tokens match, in a category / subcategory sensitive manner: if receiver token is a category, then it matches other token if it is the same category and likewise for subcategory
func (Tokens) PunctGpMatch ¶
PunctGpMatch returns the matching token for given PunctGp token
Source Files