enry

README

enry

File programming language detector and toolbox to ignore binary or vendored files. enry, started as a port to Go of the original linguist Ruby library, that has an improved 2x performance.

Installation

The recommended way to install enry is

go get gopkg.in/src-d/enry.v1/cmd/enry

To build enry's CLI you must run

make build

this will generate a binary in the project's root directory called enry. You can then move this binary to anywhere in your PATH.

This project is now part of source{d} Engine, which provides the simplest way to get started with a single command. Visit sourced.tech/engine for more information.

Faster regexp engine (optional)

Oniguruma is CRuby's regular expression engine. It is very fast and performs better than the one built into Go runtime. enry supports swapping between those two engines thanks to rubex project. The typical overall speedup from using Oniguruma is 1.5-2x. However, it requires CGo and the external shared library. On macOS with brew, it is

brew install oniguruma

On Ubuntu, it is

sudo apt install libonig-dev

To build enry with Oniguruma regexps use the oniguruma build tag

go get -v -t --tags oniguruma ./...

and then rebuild the project.

Examples

lang, safe := enry.GetLanguageByExtension("foo.go")
fmt.Println(lang, safe)
// result: Go true

lang, safe := enry.GetLanguageByContent("foo.m", []byte("<matlab-code>"))
fmt.Println(lang, safe)
// result: Matlab true

lang, safe := enry.GetLanguageByContent("bar.m", []byte("<objective-c-code>"))
fmt.Println(lang, safe)
// result: Objective-C true

// all strategies together
lang := enry.GetLanguage("foo.cpp", []byte("<cpp-code>"))
// result: C++ true

Note that the returned boolean value safe is set either to true, if there is only one possible language detected, or to false otherwise.

To get a list of possible languages for a given file, you can use the plural version of the detecting functions.

langs := enry.GetLanguages("foo.h",  []byte("<cpp-code>"))
// result: []string{"C", "C++", "Objective-C}

langs := enry.GetLanguagesByExtension("foo.asc", []byte("<content>"), nil)
// result: []string{"AGS Script", "AsciiDoc", "Public Key"}

langs := enry.GetLanguagesByFilename("Gemfile", []byte("<content>"), []string{})
// result: []string{"Ruby"}

CLI

You can use enry as a command,

$ enry --help
  enry v1.5.0 build: 10-02-2017_14_01_07 commit: 95ef0a6cf3, based on linguist commit: 37979b2
  enry, A simple (and faster) implementation of github/linguist
  usage: enry <path>
         enry [-json] [-breakdown] <path>
         enry [-json] [-breakdown]
         enry [-version]

and it'll return an output similar to linguist's output,

$ enry
55.56%    Shell
22.22%    Ruby
11.11%    Gnuplot
11.11%    Go

but not only the output; its flags are also the same as linguist's ones,

$ enry --breakdown
55.56%    Shell
22.22%    Ruby
11.11%    Gnuplot
11.11%    Go

Gnuplot
plot-histogram.gp

Ruby
linguist-samples.rb
linguist-total.rb

Shell
parse.sh
plot-histogram.sh
run-benchmark.sh
run-slow-benchmark.sh
run.sh

Go
parser/main.go

even the JSON flag,

$ enry --json
{"Gnuplot":["plot-histogram.gp"],"Go":["parser/main.go"],"Ruby":["linguist-samples.rb","linguist-total.rb"],"Shell":["parse.sh","plot-histogram.sh","run-benchmark.sh","run-slow-benchmark.sh","run.sh"]}

Note that even if enry's CLI is compatible with linguist's, its main point is that enry doesn't need a git repository to work!

Java bindings

Generated Java bindings using a C-shared library and JNI are located under java

Development

enry re-uses parts of original linguist to generate internal data structures. In order to update to the latest upstream and generate all the necessary code you must run:

git clone https://github.com/github/linguist.git .linguist
# update commit in generator_test.go (to re-generate .gold fixtures)
# https://github.com/src-d/enry/blob/13d3d66d37a87f23a013246a1b0678c9ee3d524b/internal/code-generator/generator/generator_test.go#L18
go generate

We update enry when changes are done in linguist's master branch on the following files:

• languages.yml
• heuristics.yml
• vendor.yml
• documentation.yml

Currently we don't have any procedure established to automatically detect changes in the linguist project and regenerate the code. So we update the generated code as needed, without any specific criteria.

If you want to update enry because of changes in linguist, you can run the go generate command and do a pull request that only contains the changes in generated files (those files in the subdirectory data).

To run the tests,

make test

Divergences from linguist

enry CLI tool does not require a full Git repository to be present in the filesystem in order to report languages.

Using linguist/samples as a set for the tests, the following issues were found:

• Heuristics for ".es" extension in JavaScript could not be parsed, due to unsupported backreference in RE2 regexp engine

• As of (Linguist v5.3.2)[https://github.com/github/linguist/releases/tag/v5.3.2] it is using flex-based scanner in C for tokenization. Enry still uses extract_token regex-based algorithm. See #193.

• Bayesian classifier can't distinguish "SQL" from "PLpgSQL. See #194.

• Detection of generated files is not supported yet. (Thus they are not excluded from CLI output). See #213.

• XML detection strategy is not implemented. See #192.

• Overriding languages and types though .gitattributes is not yet supported. See #18.

• enry CLI output does NOT exclude .gitignoreed files and git submodules, as linguist does

In all the cases above that have an issue number - we plan to update enry to match Linguist behaviour.

Benchmarks

Enry's language detection has been compared with Linguist's one. In order to do that, Linguist's project directory linguist/samples was used as a set of files to run benchmarks against.

We got these results:

The histogram represents the number of files for which spent time in language detection was in the range of the time interval indicated in the x axis.

So you can see that most of the files were detected quicker in enry.

We found some few cases where enry turns slower than linguist. This is due to Golang's regexp engine being slower than Ruby's, which uses the oniguruma library, written in C.

You can find scripts and additional information (like software and hardware used and benchmarks' results per sample file) in benchmarks directory.

Benchmark Dependencies

As benchmarks depend on Ruby and Github-Linguist gem make sure you have:

• Ruby (e.g using rbenv), bundler installed
• Docker
• native dependencies installed
• Build the gem cd .linguist && bundle install && rake build_gem && cd -
• Install it gem install --no-rdoc --no-ri --local .linguist/github-linguist-*.gem

How to reproduce current results

If you want to reproduce the same benchmarks as reported above:

• Make sure all dependencies are installed
• Install gnuplot (in order to plot the histogram)
• Run ENRY_TEST_REPO="$PWD/.linguist" benchmarks/run.sh (takes ~15h)

It will run the benchmarks for enry and linguist, parse the output, create csv files and plot the histogram. This takes some time.

Quick

To run quicker benchmarks you can either:

make benchmarks

to get average times for the main detection function and strategies for the whole samples set or:

make benchmarks-samples

if you want to see measures per sample file.

Why Enry?

In the movie My Fair Lady, Professor Henry Higgins is one of the main characters. Henry is a linguist and at the very beginning of the movie enjoys guessing the origin of people based on their accent.

Enry Iggins is how Eliza Doolittle, pronounces the name of the Professor during the first half of the movie.

License

Apache License, Version 2.0. See LICENSE

Documentation

Overview

Package enry implements multiple strategies for programming language identification.

Identification is made based on file name and file content using a seriece of strategies to narrow down possible option. Each strategy is available as a separate API call, as well as a main enty point

GetLanguage(filename string, content []byte) (language string)

It is a port of the https://github.com/github/linguist from Ruby. Upstream Linguist YAML files are used to generate datastructures for data package.

Index

• Constants
• Variables
• func GetLanguage(filename string, content []byte) (language string)
• func GetLanguageByAlias(alias string) (lang string, ok bool)
• func GetLanguageByClassifier(content []byte, candidates []string) (language string, safe bool)
• func GetLanguageByContent(filename string, content []byte) (language string, safe bool)
• func GetLanguageByEmacsModeline(content []byte) (language string, safe bool)
• func GetLanguageByExtension(filename string) (language string, safe bool)
• func GetLanguageByFilename(filename string) (language string, safe bool)
• func GetLanguageByModeline(content []byte) (language string, safe bool)
• func GetLanguageByShebang(content []byte) (language string, safe bool)
• func GetLanguageBySpecificClassifier(content []byte, candidates []string, classifier Classifier) (language string, safe bool)
• func GetLanguageByVimModeline(content []byte) (language string, safe bool)
• func GetLanguageExtensions(language string) []string
• func GetLanguages(filename string, content []byte) []string
• func GetLanguagesByClassifier(filename string, content []byte, candidates []string) (languages []string)
• func GetLanguagesByContent(filename string, content []byte, _ []string) []string
• func GetLanguagesByEmacsModeline(_ string, content []byte, _ []string) []string
• func GetLanguagesByExtension(filename string, _ []byte, _ []string) []string
• func GetLanguagesByFilename(filename string, _ []byte, _ []string) []string
• func GetLanguagesByModeline(_ string, content []byte, candidates []string) []string
• func GetLanguagesByShebang(_ string, content []byte, _ []string) (languages []string)
• func GetLanguagesBySpecificClassifier(content []byte, candidates []string, classifier Classifier) (languages []string)
• func GetLanguagesByVimModeline(_ string, content []byte, _ []string) []string
• func GetMIMEType(path string, language string) string
• func IsBinary(data []byte) bool
• func IsConfiguration(path string) bool
• func IsDocumentation(path string) bool
• func IsDotFile(path string) bool
• func IsImage(path string) bool
• func IsVendor(path string) bool
• type Classifier
• type Strategy
• type Type
  • func GetLanguageType(language string) (langType Type)

Constants

const OtherLanguage = ""

OtherLanguage is used as a zero value when a function can not return a specific language.

Variables

var DefaultStrategies = []Strategy{
	GetLanguagesByModeline,
	GetLanguagesByFilename,
	GetLanguagesByShebang,
	GetLanguagesByExtension,
	GetLanguagesByContent,
	GetLanguagesByClassifier,
}

DefaultStrategies is a sequence of strategies used by GetLanguage to detect languages.

Functions

func GetLanguage

func GetLanguage(filename string, content []byte) (language string)

GetLanguage applies a sequence of strategies based on the given filename and content to find out the most probably language to return.

func GetLanguageByAlias

func GetLanguageByAlias(alias string) (lang string, ok bool)

GetLanguageByAlias returns either the language related to the given alias and ok set to true or Otherlanguage and ok set to false if the alias is not recognized.

func GetLanguageByClassifier

func GetLanguageByClassifier(content []byte, candidates []string) (language string, safe bool)

GetLanguageByClassifier returns the most probably language detected for the given content. It uses DefaultClassifier, if no candidates are provided it returns OtherLanguage.

func GetLanguageByContent

func GetLanguageByContent(filename string, content []byte) (language string, safe bool)

GetLanguageByContent returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByEmacsModeline

func GetLanguageByEmacsModeline(content []byte) (language string, safe bool)

GetLanguageByEmacsModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByExtension

func GetLanguageByExtension(filename string) (language string, safe bool)

GetLanguageByExtension returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByFilename

func GetLanguageByFilename(filename string) (language string, safe bool)

GetLanguageByFilename returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByModeline

func GetLanguageByModeline(content []byte) (language string, safe bool)

GetLanguageByModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByShebang

func GetLanguageByShebang(content []byte) (language string, safe bool)

GetLanguageByShebang returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageBySpecificClassifier

func GetLanguageBySpecificClassifier(content []byte, candidates []string, classifier Classifier) (language string, safe bool)

GetLanguageBySpecificClassifier returns the most probably language for the given content using classifier to detect language.

func GetLanguageByVimModeline

func GetLanguageByVimModeline(content []byte) (language string, safe bool)

GetLanguageByVimModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageExtensions

func GetLanguageExtensions(language string) []string

GetLanguageExtensions returns the different extensions being used by the language.

func GetLanguages

func GetLanguages(filename string, content []byte) []string

GetLanguages applies a sequence of strategies based on the given filename and content to find out the most probably languages to return. At least one of arguments should be set. If content is missing, language detection will be based on the filename. The function won't read the file, given an empty content.

func GetLanguagesByClassifier

func GetLanguagesByClassifier(filename string, content []byte, candidates []string) (languages []string)

GetLanguagesByClassifier uses DefaultClassifier as a Classifier and returns a sorted slice of possible languages ordered by decreasing language's probability. If there are not candidates it returns nil. It complies with the signature to be a Strategy type.

func GetLanguagesByContent

func GetLanguagesByContent(filename string, content []byte, _ []string) []string

GetLanguagesByContent returns a slice of languages for the given content. It is a Strategy that uses content-based regexp heuristics and a filename extension.

func GetLanguagesByEmacsModeline

func GetLanguagesByEmacsModeline(_ string, content []byte, _ []string) []string

GetLanguagesByEmacsModeline returns a slice of possible languages for the given content. It complies with the signature to be a Strategy type.

func GetLanguagesByExtension

func GetLanguagesByExtension(filename string, _ []byte, _ []string) []string

GetLanguagesByExtension returns a slice of possible languages for the given filename. It complies with the signature to be a Strategy type.

func GetLanguagesByFilename

func GetLanguagesByFilename(filename string, _ []byte, _ []string) []string

GetLanguagesByFilename returns a slice of possible languages for the given filename. It complies with the signature to be a Strategy type.

func GetLanguagesByModeline

func GetLanguagesByModeline(_ string, content []byte, candidates []string) []string

GetLanguagesByModeline returns a slice of possible languages for the given content. It complies with the signature to be a Strategy type.

func GetLanguagesByShebang

func GetLanguagesByShebang(_ string, content []byte, _ []string) (languages []string)

GetLanguagesByShebang returns a slice of possible languages for the given content. It complies with the signature to be a Strategy type.

func GetLanguagesBySpecificClassifier

func GetLanguagesBySpecificClassifier(content []byte, candidates []string, classifier Classifier) (languages []string)

GetLanguagesBySpecificClassifier returns a slice of possible languages. It takes in a Classifier to be used.

func GetLanguagesByVimModeline

func GetLanguagesByVimModeline(_ string, content []byte, _ []string) []string

GetLanguagesByVimModeline returns a slice of possible languages for the given content. It complies with the signature to be a Strategy type.

func GetMIMEType

func GetMIMEType(path string, language string) string

GetMIMEType returns a MIME type of a given file based on its languages.

func IsBinary

func IsBinary(data []byte) bool

IsBinary detects if data is a binary value based on: http://git.kernel.org/cgit/git/git.git/tree/xdiff-interface.c?id=HEAD#n198

func IsConfiguration

func IsConfiguration(path string) bool

IsConfiguration tells if filename is in one of the configuration languages.

func IsDocumentation

func IsDocumentation(path string) bool

IsDocumentation returns whether or not path is a documentation path.

func IsDotFile

func IsDotFile(path string) bool

IsDotFile returns whether or not path has dot as a prefix.

func IsImage

func IsImage(path string) bool

IsImage tells if a given file is an image (PNG, JPEG or GIF format).

func IsVendor

func IsVendor(path string) bool

IsVendor returns whether or not path is a vendor path.

Types

type Classifier

type Classifier interface {
	Classify(content []byte, candidates map[string]float64) (languages []string)
}

Classifier is the interface in charge to detect the possible languages of the given content based on a set of candidates. Candidates is a map which can be used to assign weights to languages dynamically.

var DefaultClassifier Classifier = &classifier{
	languagesLogProbabilities: data.LanguagesLogProbabilities,
	tokensLogProbabilities:    data.TokensLogProbabilities,
	tokensTotal:               data.TokensTotal,
}

DefaultClassifier is a Naive Bayes classifier trained on Linguist samples.

type Strategy

type Strategy func(filename string, content []byte, candidates []string) (languages []string)

Strategy type fix the signature for the functions that can be used as a strategy.

type Type

type Type int

Type represent language's type. Either data, programming, markup, prose, or unknown.

const (
	Unknown Type = iota
	Data
	Programming
	Markup
	Prose
)

Type's values.

func GetLanguageType

func GetLanguageType(language string) (langType Type)

GetLanguageType returns the type of the given language.