lexers

README

Lexer tests

The tests in this directory feed a known input testdata/<name>.actual into the parser for <name> and check that its output matches <name>.exported.

Running the tests

Run the tests as normal:

go test ./lexers

Update existing tests

When you add a new test data file (*.actual), you need to regenerate all tests. That's how Chroma creates the *.expected test file based on the corresponding lexer.

To regenerate all tests, type in your terminal:

RECORD=true go test ./lexers

This first sets the RECORD environment variable to true. Then it runs go test on the ./lexers directory of the Chroma project.

(That environment variable tells Chroma it needs to output test data. After running go test ./lexers you can remove or reset that variable.)

Windows users

Windows users will find that the RECORD=true go test ./lexers command fails in both the standard command prompt terminal and in PowerShell.

Instead we have to perform both steps separately:

• Set the RECORD environment variable to true.
  • In the regular command prompt window, the set command sets an environment variable for the current session: set RECORD=true. See this page for more.
  • In PowerShell, you can use the $env:RECORD = 'true' command for that. See this article for more.
  • You can also make a persistent environment variable by hand in the Windows computer settings. See this article for how.
• When the environment variable is set, run go tests ./lexers.

Chroma will now regenerate the test files and print its results to the console window.

Documentation

Overview

Package lexers contains the registry of all lexers.

Sub-packages contain lexer implementations.

Index

• Variables
• func Analyse(text string) chroma.Lexer
• func Get(name string) chroma.Lexer
• func Match(filename string) chroma.Lexer
• func MatchMimeType(mimeType string) chroma.Lexer
• func Names(withAliases bool) []string
• func Register(lexer chroma.Lexer) chroma.Lexer

Variables

var Fallback = internal.Fallback

Fallback lexer if no other is found.

var HLB = internal.Register(MustNewLexer(
	&Config{
		Name:      "HLB",
		Aliases:   []string{"hlb"},
		Filenames: []string{"*.hlb"},
		MimeTypes: []string{},
	},
	Rules{
		"root": {
			{`(#.*)`, ByGroups(CommentSingle), nil},
			{`((\b(0(b|B|o|O|x|X)[a-fA-F0-9]+)\b)|(\b(0|[1-9][0-9]*)\b))`, ByGroups(LiteralNumber), nil},
			{`((\b(true|false)\b))`, ByGroups(NameBuiltin), nil},
			{`(\bstring\b|\bint\b|\bbool\b|\bfs\b|\boption\b)`, ByGroups(KeywordType), nil},
			{`(\b[a-zA-Z_][a-zA-Z0-9]*\b)(\()`, ByGroups(NameFunction, Punctuation), Push("params")},
			{`(\{)`, ByGroups(Punctuation), Push("block")},
			{`(\n|\r|\r\n)`, Text, nil},
			{`.`, Text, nil},
		},
		"string": {
			{`"`, LiteralString, Pop(1)},
			{`\\"`, LiteralString, nil},
			{`[^\\"]+`, LiteralString, nil},
		},
		"block": {
			{`(\})`, ByGroups(Punctuation), Pop(1)},
			{`(#.*)`, ByGroups(CommentSingle), nil},
			{`((\b(0(b|B|o|O|x|X)[a-fA-F0-9]+)\b)|(\b(0|[1-9][0-9]*)\b))`, ByGroups(LiteralNumber), nil},
			{`((\b(true|false)\b))`, ByGroups(KeywordConstant), nil},
			{`"`, LiteralString, Push("string")},
			{`(with)`, ByGroups(KeywordReserved), nil},
			{`(as)([\t ]+)(\b[a-zA-Z_][a-zA-Z0-9]*\b)`, ByGroups(KeywordReserved, Text, NameFunction), nil},
			{`(\bstring\b|\bint\b|\bbool\b|\bfs\b|\boption\b)([\t ]+)(\{)`, ByGroups(KeywordType, Text, Punctuation), Push("block")},
			{`(?!\b(?:scratch|image|resolve|http|checksum|chmod|filename|git|keepGitDir|local|includePatterns|excludePatterns|followPaths|generate|frontendInput|shell|run|readonlyRootfs|env|dir|user|network|security|host|ssh|secret|mount|target|localPath|uid|gid|mode|readonly|tmpfs|sourcePath|cache|mkdir|createParents|chown|createdTime|mkfile|rm|allowNotFound|allowWildcards|copy|followSymlinks|contentsOnly|unpack|createDestPath)\b)(\b[a-zA-Z_][a-zA-Z0-9]*\b)`, ByGroups(NameOther), nil},
			{`(\n|\r|\r\n)`, Text, nil},
			{`.`, Text, nil},
		},
		"params": {
			{`(\))`, ByGroups(Punctuation), Pop(1)},
			{`(variadic)`, ByGroups(Keyword), nil},
			{`(\bstring\b|\bint\b|\bbool\b|\bfs\b|\boption\b)`, ByGroups(KeywordType), nil},
			{`(\b[a-zA-Z_][a-zA-Z0-9]*\b)`, ByGroups(NameOther), nil},
			{`(\n|\r|\r\n)`, Text, nil},
			{`.`, Text, nil},
		},
	},
))

HLB lexer.

var Qml = internal.Register(MustNewLexer(
	&Config{
		Name:      "QML",
		Aliases:   []string{"qml", "qbs"},
		Filenames: []string{"*.qml", "*.qbs"},
		MimeTypes: []string{"application/x-qml", "application/x-qt.qbs+qml"},
		DotAll:    true,
	},
	Rules{
		"commentsandwhitespace": {
			{`\s+`, Text, nil},
			{`<!--`, Comment, nil},
			{`//.*?\n`, CommentSingle, nil},
			{`/\*.*?\*/`, CommentMultiline, nil},
		},
		"slashstartsregex": {
			Include("commentsandwhitespace"),
			{`/(\\.|[^[/\\\n]|\[(\\.|[^\]\\\n])*])+/([gim]+\b|\B)`, LiteralStringRegex, Pop(1)},
			{`(?=/)`, Text, Push("#pop", "badregex")},
			Default(Pop(1)),
		},
		"badregex": {
			{`\n`, Text, Pop(1)},
		},
		"root": {
			{`^(?=\s|/|<!--)`, Text, Push("slashstartsregex")},
			Include("commentsandwhitespace"),
			{`\+\+|--|~|&&|\?|:|\|\||\\(?=\n)|(<<|>>>?|==?|!=?|[-<>+*%&|^/])=?`, Operator, Push("slashstartsregex")},
			{`[{(\[;,]`, Punctuation, Push("slashstartsregex")},
			{`[})\].]`, Punctuation, nil},
			{`\bid\s*:\s*[A-Za-z][\w.]*`, KeywordDeclaration, Push("slashstartsregex")},
			{`\b[A-Za-z][\w.]*\s*:`, Keyword, Push("slashstartsregex")},
			{`(for|in|while|do|break|return|continue|switch|case|default|if|else|throw|try|catch|finally|new|delete|typeof|instanceof|void|this)\b`, Keyword, Push("slashstartsregex")},
			{`(var|let|with|function)\b`, KeywordDeclaration, Push("slashstartsregex")},
			{`(abstract|boolean|byte|char|class|const|debugger|double|enum|export|extends|final|float|goto|implements|import|int|interface|long|native|package|private|protected|public|short|static|super|synchronized|throws|transient|volatile)\b`, KeywordReserved, nil},
			{`(true|false|null|NaN|Infinity|undefined)\b`, KeywordConstant, nil},
			{`(Array|Boolean|Date|Error|Function|Math|netscape|Number|Object|Packages|RegExp|String|sun|decodeURI|decodeURIComponent|encodeURI|encodeURIComponent|Error|eval|isFinite|isNaN|parseFloat|parseInt|document|this|window)\b`, NameBuiltin, nil},
			{`[$a-zA-Z_]\w*`, NameOther, nil},
			{`[0-9][0-9]*\.[0-9]+([eE][0-9]+)?[fd]?`, LiteralNumberFloat, nil},
			{`0x[0-9a-fA-F]+`, LiteralNumberHex, nil},
			{`[0-9]+`, LiteralNumberInteger, nil},
			{`"(\\\\|\\"|[^"])*"`, LiteralStringDouble, nil},
			{`'(\\\\|\\'|[^'])*'`, LiteralStringSingle, nil},
		},
	},
))

Qml lexer.

var Registry = internal.Registry

Registry of Lexers.

Functions

func Analyse

func Analyse(text string) chroma.Lexer

Analyse text content and return the "best" lexer..

func Get

func Get(name string) chroma.Lexer

Get a Lexer by name, alias or file extension.

func Match

func Match(filename string) chroma.Lexer

Match returns the first lexer matching filename.

func MatchMimeType

func MatchMimeType(mimeType string) chroma.Lexer

MatchMimeType attempts to find a lexer for the given MIME type.

func Names

func Names(withAliases bool) []string

Names of all lexers, optionally including aliases.

func Register

func Register(lexer chroma.Lexer) chroma.Lexer

Register a Lexer with the global registry.