enry

package module
v1.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2017 License: MIT Imports: 9 Imported by: 0

README

enry GoDoc Build Status codecov

File programming language detector and toolbox to ignore binary or vendored files. enry, started as a port to Go of the original linguist Ruby library, that has an improved 2x performance.

Installation

The recommended way to install simple-linguist

go get gopkg.in/src-d/enry.v1/...

To build enry's CLI you must run

make build-cli

it generates a binary in the project's root directory called enry. You can move this binary to anywhere in your PATH.

Examples

lang, safe := enry.GetLanguageByExtension("foo.go")
fmt.Println(lang)
// result: Go

lang, safe := enry.GetLanguageByContent("foo.m", "<matlab-code>")
fmt.Println(lang)
// result: Matlab

lang, safe := enry.GetLanguageByContent("bar.m", "<objective-c-code>")
fmt.Println(lang)
// result: Objective-C

// all strategies together
lang := enry.GetLanguage("foo.cpp", "<cpp-code>")

Note the returned boolean value "safe" is set either to true, if there is only one possible language detected or, to false otherwise.

To get a list of possible languages for a given file, you can use the plural version of the detecting functions.

langs := enry.GetLanguages("foo.h",  "<cpp-code>")
// result: []string{"C++", "C"}

langs := enry.GetLanguagesByExtension("foo.asc", "<content>", nil)
// result: []string{"AGS Script", "AsciiDoc", "Public Key"}

langs := enry.GetLanguagesByFilename("Gemfile", "<content>", []string{})
// result: []string{"Ruby"}

CLI

You can use enry as a command,

$ enry --help
enry, A simple (and faster) implementation of github/linguist
usage: enry <path>
              enry <path> [--json] [--breakdown]
              enry [--json] [--breakdown]

and it will return an output similar to linguist's output,

$ enry
11.11%    Gnuplot
22.22%    Ruby
55.56%    Shell
11.11%    Go

but not only the output, also its flags are the same as linguist's ones,

$ enry --breakdown
11.11%    Gnuplot
22.22%    Ruby
55.56%    Shell
11.11%    Go

Gnuplot
plot-histogram.gp

Ruby
linguist-samples.rb
linguist-total.rb

Shell
parse.sh
plot-histogram.sh
run-benchmark.sh
run-slow-benchmark.sh
run.sh

Go
parser/main.go

even the JSON flag,

$ enry --json
{"Gnuplot":["plot-histogram.gp"],"Go":["parser/main.go"],"Ruby":["linguist-samples.rb","linguist-total.rb"],"Shell":["parse.sh","plot-histogram.sh","run-benchmark.sh","run-slow-benchmark.sh","run.sh"]}

Note that even if enry's CLI is compatible with linguist's, its main point is that, contrary to linguist, enry doesn't need a git repository to work!

Development

enry re-uses parts of original linguist to generate internal data structures. In order to update to latest upstream and generate the necessary code you must run:

go generate

We update enry due to changes in linguist's master branch related to the following files:

For the moment we don't have any procedure established to detect changes in the linguist project automatically and regenerate the code. So we are updating the generated code as needed, without any specific criteria.

If you want update enry because of changes in linguist, you can run the go generate command and do a pull request that only contains the changes in generated files (those files in the subdirectory data).

To run the tests

make test

Divergences from linguist

Using linguist/samples as a set against run tests the following issues were found:

  • with hello.ms we can't detect the language (Unix Assembly) because we don't have a matcher in contentMatchers (content.go) for Unix Assembly. Linguist uses this regexp in its code,

    elsif /(?<!\S)\.(include|globa?l)\s/.match(data) || /(?<!\/\*)(\A|\n)\s*\.[A-Za-z][_A-Za-z0-9]*:/.match(data.gsub(/"([^\\"]|\\.)*"|'([^\\']|\\.)*'|\\\s*(?:--.*)?\n/, ""))

    which we can't port.

  • all files for SQL language fall to the classifier because we don't parse this disambiguator expresion for *.sql files right. This expression doesn't comply with the pattern for the rest of heuristics.rb file.

Benchmarks

Enry's language detection has been compared with Linguist's language detection. In order to do that, linguist's project directory linguist/samples was used as a set of files to run benchmarks against.

Following results were obtained:

histogram

The histogram represents the number of files for which spent time in language detection was in the range of the time interval indicated in x axis.

So reviewing the comparison enry/linguist, you can see the most of the files were detected in less time than linguist does.

We detected some few cases enry turns slower than linguist. This is due to Golang's regexp engine being slower than Ruby's, which uses oniguruma library, written in C.

You can find scripts and additional information (as software and hardware used, and benchmarks' results per sample file) in benchmarks directory.

If you want to reproduce the same experiment you can run:

benchmarks/run.sh

from the root's project directory and It runs benchmarks for enry and linguist, parse the output, create csv files and create a histogram (you must have installed gnuplot in your system to get the histogram). It can take to much time, so to run local benchmarks to take a quick look you can run either:

make benchmarks

to get time averages for main detection function and strategies for the whole samples set or:

make benchmarks-samples

if you want see measures by sample file

Why Enry?

In the movie My Fair Lady, Professor Henry Higgins is one of the main characters. Henry is a linguist and at the very beginning of the movie enjoys guessing the nationality of people based on their accent.

Enry Iggins is how Eliza Doolittle, pronounces the name of the Professor during the first half of the movie.

License

MIT, see LICENSE

Documentation

Index

Constants

View Source 
const OtherLanguage = ""

OtherLanguage is used as a zero value when a function can not return a specific language.

Variables

View Source 
var DefaultStrategies = []Strategy{
	GetLanguagesByModeline,
	GetLanguagesByFilename,
	GetLanguagesByShebang,
	GetLanguagesByExtension,
	GetLanguagesByContent,
	GetLanguagesByClassifier,
}

DefaultStrategies is the strategies' sequence GetLanguage uses to detect languages.

Functions

func GetLanguage 

func GetLanguage(filename string, content []byte) (language string)

GetLanguage applies a sequence of strategies based on the given filename and content to find out the most probably language to return.

func GetLanguageByAlias 

func GetLanguageByAlias(alias string) (lang string, ok bool)

GetLanguageByAlias returns either the language related to the given alias and ok set to true or Otherlanguage and ok set to false if the alias is not recognized.

func GetLanguageByClassifier added in v1.2.1 

func GetLanguageByClassifier(content []byte, candidates []string) (language string, safe bool)

GetLanguageByClassifier returns the most probably language detected for the given content. It uses DefaultClassifier, if no candidates are provided it returns OtherLanguage.

func GetLanguageByContent 

func GetLanguageByContent(content []byte) (language string, safe bool)

GetLanguageByContent returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByEmacsModeline 

func GetLanguageByEmacsModeline(content []byte) (language string, safe bool)

GetLanguageByEmacsModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByExtension 

func GetLanguageByExtension(filename string) (language string, safe bool)

GetLanguageByExtension returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByFilename 

func GetLanguageByFilename(filename string) (language string, safe bool)

GetLanguageByFilename returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByModeline 

func GetLanguageByModeline(content []byte) (language string, safe bool)

GetLanguageByModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByShebang 

func GetLanguageByShebang(content []byte) (language string, safe bool)

GetLanguageByShebang returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageBySpecificClassifier added in v1.2.1 

func GetLanguageBySpecificClassifier(content []byte, candidates []string, classifier Classifier) (language string, safe bool)

GetLanguageBySpecificClassifier returns the most probably language for the given content using classifier to detect language.

func GetLanguageByVimModeline 

func GetLanguageByVimModeline(content []byte) (language string, safe bool)

GetLanguageByVimModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageExtensions 

func GetLanguageExtensions(language string) []string

GetLanguageExtensions returns the different extensions being used by the language.

func GetLanguages added in v1.2.1 

func GetLanguages(filename string, content []byte) []string

GetLanguages applies a sequence of strategies based on the given filename and content to find out the most probably languages to return.

func GetLanguagesByClassifier added in v1.2.1 

func GetLanguagesByClassifier(filename string, content []byte, candidates []string) (languages []string)

GetLanguagesByClassifier uses DefaultClassifier as a Classifier and returns a sorted slice of possible languages ordered by decreasing language's probability. If there are not candidates it returns nil. It complies with the signature to be a Strategy type.

func GetLanguagesByContent added in v1.2.1 

func GetLanguagesByContent(filename string, content []byte, candidates []string) []string

GetLanguagesByContent returns a slice of possible languages for the given content, filename and candidates will be ignored. It complies with the signature to be a Strategy type.

func GetLanguagesByEmacsModeline added in v1.2.1 

func GetLanguagesByEmacsModeline(filename string, content []byte, candidates []string) []string

GetLanguagesByEmacsModeline returns a slice of possible languages for the given content, filename and candidates will be ignored. It complies with the signature to be a Strategy type.

func GetLanguagesByExtension added in v1.2.1 

func GetLanguagesByExtension(filename string, content []byte, candidates []string) []string

GetLanguagesByExtension returns a slice of possible languages for the given filename, content and candidates will be ignored. It complies with the signature to be a Strategy type.

func GetLanguagesByFilename added in v1.2.1 

func GetLanguagesByFilename(filename string, content []byte, candidates []string) []string

GetLanguagesByFilename returns a slice of possible languages for the given filename, content and candidates will be ignored. It complies with the signature to be a Strategy type.

func GetLanguagesByModeline added in v1.2.1 

func GetLanguagesByModeline(filename string, content []byte, candidates []string) []string

GetLanguagesByModeline returns a slice of possible languages for the given content, filename will be ignored. It complies with the signature to be a Strategy type.

func GetLanguagesByShebang added in v1.2.1 

func GetLanguagesByShebang(filename string, content []byte, candidates []string) (languages []string)

GetLanguagesByShebang returns a slice of possible languages for the given content, filename and candidates will be ignored. It complies with the signature to be a Strategy type.

func GetLanguagesBySpecificClassifier added in v1.2.1 

func GetLanguagesBySpecificClassifier(content []byte, candidates []string, classifier Classifier) (languages []string)

GetLanguagesBySpecificClassifier returns a slice of possible languages. It takes in a Classifier to be used.

func GetLanguagesByVimModeline added in v1.2.1 

func GetLanguagesByVimModeline(filename string, content []byte, candidates []string) []string

GetLanguagesByVimModeline returns a slice of possible languages for the given content, filename and candidates will be ignored. It complies with the signature to be a Strategy type.

func IsAuxiliaryLanguage 

func IsAuxiliaryLanguage(lang string) bool

IsAuxiliaryLanguage returns whether or not lang is an auxiliary language.

func IsBinary 

func IsBinary(data []byte) bool

IsBinary detects if data is a binary value based on: http://git.kernel.org/cgit/git/git.git/tree/xdiff-interface.c?id=HEAD#n198

func IsConfiguration 

func IsConfiguration(path string) bool

IsConfiguration returns whether or not path is using a configuration language.

func IsDocumentation 

func IsDocumentation(path string) bool

IsDocumentation returns whether or not path is a documentation path.

func IsDotFile 

func IsDotFile(path string) bool

IsDotFile returns whether or not path has dot as a prefix.

func IsVendor 

func IsVendor(path string) bool

IsVendor returns whether or not path is a vendor path.

Types

type Classifier added in v1.2.1 

type Classifier interface {
	Classify(content []byte, candidates map[string]float64) (languages []string)
}

Classifier is the interface in charge to detect the possible languages of the given content based on a set of candidates. Candidates is a map which can be used to assign weights to languages dynamically. 

var DefaultClassifier Classifier = &classifier{
	languagesLogProbabilities: data.LanguagesLogProbabilities,
	tokensLogProbabilities:    data.TokensLogProbabilities,
	tokensTotal:               data.TokensTotal,
}

type Strategy added in v1.2.1 

type Strategy func(filename string, content []byte, candidates []string) (languages []string)

Strategy type fix the signature for the functions that can be used as a strategy.

type Type 

type Type int

Type represent language's type. Either data, programming, markup, prose, or unknown. 

const (
	Unknown Type = iota
	Data
	Programming
	Markup
	Prose
)

Type's values.

func GetLanguageType 

func GetLanguageType(language string) (langType Type)

GetLanguageType returns the type of the given language.

Source Files

View all Source files

Directories

Path Synopsis
benchmarks
parser
cli
enry
internal
code-generator
code-generator/generator
tokenizer

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.