enry

package module
v2.0.0-...-6ef3e87 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 15, 2023 License: Apache-2.0 Imports: 12 Imported by: 0

README

go-enry GoDoc Test codecov

Programming language detector and toolbox to ignore binary or vendored files. enry, started as a port to Go of the original Linguist Ruby library, that has an improved 2x performance.

CLI

The CLI binary is hosted in a separate repository go-enry/enry.

Library

enry is also a Go library for guessing a programming language that exposes API through FFI to multiple programming environments.

Use cases

enry guesses a programming language using a sequence of matching strategies that are applied progressively to narrow down the possible options. Each strategy varies on the type of input data that it needs to make a decision: file name, extension, the first line of the file, the full content of the file, etc.

Depending on available input data, enry API can be roughly divided into the next categories or use cases.

By filename

Next functions require only a name of the file to make a guess:

  • GetLanguageByExtension uses only file extension (wich may be ambiguous)
  • GetLanguageByFilename useful for cases like .gitignore, .bashrc, etc
  • all filtering helpers

Please note that such guesses are expected not to be very accurate.

By text

To make a guess only based on the content of the file or a text snippet, use

  • GetLanguageByShebang reads only the first line of text to identify the shebang.

  • GetLanguageByModeline for cases when Vim/Emacs modeline e.g. /* vim: set ft=cpp: */ may be present at a head or a tail of the text.

  • GetLanguageByClassifier uses a Bayesian classifier trained on all the ./samples/ from Linguist.

    It usually is a last-resort strategy that is used to disambiguate the guess of the previous strategies, and thus it requires a list of "candidate" guesses. One can provide a list of all known languages - keys from the data.LanguagesLogProbabilities as possible candidates if more intelligent hypotheses are not available, at the price of possibly suboptimal accuracy.

By file

The most accurate guess would be when both, a file name and it's content are available:

  • GetLanguagesByContent only uses file extension and a set of regexp-based content heuristics.
  • GetLanguages uses the full set of matching strategies and is expected to be most accurate.
Filtering: vendoring, binaries, etc

enry expose a set of file-level helpers Is* to simplify filtering out the files that are less interesting for the purpose of source code analysis:

  • IsBinary
  • IsVendor
  • IsConfiguration
  • IsDocumentation
  • IsDotFile
  • IsImage
  • IsTest
  • IsGenerated
Language colors and groups

enry exposes function to get language color to use for example in presenting statistics in graphs:

  • GetColor
  • GetLanguageGroup can be used to group similar languages together e.g. for Less this function will return CSS

Languages

Go

In a Go module, import enry to the module by running:

go get github.com/go-enry/go-enry/v2

The rest of the examples will assume you have either done this or fetched the library into your GOPATH.

// The examples here and below assume you have imported the library.
import "github.com/go-enry/go-enry/v2"

lang, safe := enry.GetLanguageByExtension("foo.go")
fmt.Println(lang, safe)
// result: Go true

lang, safe := enry.GetLanguageByContent("foo.m", []byte("<matlab-code>"))
fmt.Println(lang, safe)
// result: Matlab true

lang, safe := enry.GetLanguageByContent("bar.m", []byte("<objective-c-code>"))
fmt.Println(lang, safe)
// result: Objective-C true

// all strategies together
lang := enry.GetLanguage("foo.cpp", []byte("<cpp-code>"))
// result: C++ true

Note that the returned boolean value safe is true if there is only one possible language detected.

A plural version of the same API allows getting a list of all possible languages for a given file.

langs := enry.GetLanguages("foo.h",  []byte("<cpp-code>"))
// result: []string{"C", "C++", "Objective-C}

langs := enry.GetLanguagesByExtension("foo.asc", []byte("<content>"), nil)
// result: []string{"AGS Script", "AsciiDoc", "Public Key"}

langs := enry.GetLanguagesByFilename("Gemfile", []byte("<content>"), []string{})
// result: []string{"Ruby"}
Java bindings

Generated Java bindings using a C shared library and JNI are available under java.

A library is published on Maven as tech.sourced:enry-java for macOS and linux platforms. Windows support is planned under src-d/enry#150.

Python bindings

Generated Python bindings using a C shared library and cffi are WIP under src-d/enry#154.

A library is going to be published on pypi as enry for macOS and linux platforms. Windows support is planned under src-d/enry#150.

Rust bindings

Generated Rust bindings using a C static library are available at https://github.com/go-enry/rs-enry.

Divergences from Linguist

The enry library is based on the data from github/linguist version v7.21.0.

Parsing linguist/samples the following enry results are different from the Linguist:

In all the cases above that have an issue number - we plan to update enry to match Linguist behavior.

Benchmarks

Enry's language detection has been compared with Linguist's on linguist/samples.

We got these results:

histogram

The histogram shows the number of files (y-axis) per time interval bucket (x-axis). Most of the files were detected faster by enry.

There are several cases where enry is slower than Linguist due to Go regexp engine being slower than Ruby's on, wich is based on oniguruma library, written in C.

See instructions for running enry with oniguruma.

Why Enry?

In the movie My Fair Lady, Professor Henry Higgins is a linguist who at the very beginning of the movie enjoys guessing the origin of people based on their accent.

"Enry Iggins" is how Eliza Doolittle, pronounces the name of the Professor.

Development

To run the tests use:

go test ./...

Setting ENRY_TEST_REPO to a path to the existing checkout of the Linguist will avoid cloning it and speeds tests up. Setting ENRY_DEBUG=1 will provide insight into the Bayesian classifier built during make code-generate.

Sync with github/linguist upstream

enry re-uses parts of the original github/linguist to generate internal data structures. In order to update to the latest release of linguist do:

$ git clone https://github.com/github/linguist.git .linguist
$ cd .linguist; git checkout <release-tag>; cd ..

# put the new release's commit sha in the generator_test.go (to re-generate .gold test fixtures)
# https://github.com/go-enry/go-enry/blob/13d3d66d37a87f23a013246a1b0678c9ee3d524b/internal/code-generator/generator/generator_test.go#L18

$ make code-generate

To stay in sync, enry needs to be updated when a new release of the linguist includes changes to any of the following files:

There now is automation for detecting the changes in the upstream Linguist project: every day Github CI runs a job that will create a PR to this repo for each new Linguist release. It will include all the steps from the above.

When submitting a pull request syncing up to a new release manually, please make sure it only contains the changes in the generated files (in data subdirectory).

Separating all the necessary "manual" code changes to a different PR that includes some background description and an update to the documentation on "divergences from linguist" is encouraged and very much appreciated, as it simplifies the maintenance (review/release notes/etc).

Misc

Running a benchmark & faster regexp engine
Benchmark

All benchmark scripts are in benchmarks directory.

Dependencies

As benchmarks depend on Ruby and GitHub-Linguist gem make sure you have:

  • Ruby (e.g using rbenv), bundler installed
  • Docker
  • native dependencies installed
  • Build the gem cd .linguist && bundle install && rake build_gem && cd -
  • Install it gem install --no-rdoc --no-ri --local .linguist/github-linguist-*.gem
Quick benchmark

To run quicker benchmarks

make benchmarks

to get average times for the primary detection function and strategies for the whole samples set. If you want to see measures per sample file use:

make benchmarks-samples
Full benchmark

If you want to reproduce the same benchmarks as reported above:

  • Make sure all dependencies are installed
  • Install gnuplot (in order to plot the histogram)
  • Run ENRY_TEST_REPO="$PWD/.linguist" benchmarks/run.sh (takes ~15h)

It will run the benchmarks for enry and Linguist, parse the output, create csv files and plot the histogram.

Faster regexp engine (optional)

Oniguruma is CRuby's regular expression engine. It is very fast and performs better than the one built into Go runtime. enry supports swapping between those two engines thanks to rubex project. The typical overall speedup from using Oniguruma is 1.5-2x. However, it requires CGo and the external shared library. On macOS with Homebrew, it is:

brew install oniguruma

On Ubuntu, it is

sudo apt install libonig-dev

To build enry with Oniguruma regexps use the oniguruma build tag

go get -v -t --tags oniguruma ./...

and then rebuild the project.

License

Apache License, Version 2.0. See LICENSE

Documentation

Overview

Package enry implements multiple strategies for programming language identification.

Identification is made based on file name and file content using a service of strategies to narrow down possible option. Each strategy is available as a separate API call, as well as a main enty point 

GetLanguage(filename string, content []byte) (language string)

It is a port of the https://github.com/github/linguist from Ruby. Upstream Linguist YAML files are used to generate datastructures for data package.

Index

Constants

View Source 
const (
	Unknown     Type = Type(data.TypeUnknown)
	Data             = Type(data.TypeData)
	Programming      = Type(data.TypeProgramming)
	Markup           = Type(data.TypeMarkup)
	Prose            = Type(data.TypeProse)
)

Type's values.

View Source 
const OtherLanguage = ""

OtherLanguage is used as a zero value when a function can not return a specific language.

Variables

View Source 
var DefaultStrategies = []Strategy{
	GetLanguagesByModeline,
	GetLanguagesByFilename,
	GetLanguagesByShebang,
	GetLanguagesByExtension,
	GetLanguagesByXML,
	GetLanguagesByManpage,
	GetLanguagesByContent,
	GetLanguagesByClassifier,
}

DefaultStrategies is a sequence of strategies used by GetLanguage to detect languages.

Functions

func GetColor 

func GetColor(language string) string

GetColor returns a HTML color code of a given language.

func GetLanguage 

func GetLanguage(filename string, content []byte) (language string)

GetLanguage applies a sequence of strategies based on the given filename and content to find out the most probable language to return.

func GetLanguageByAlias 

func GetLanguageByAlias(alias string) (lang string, ok bool)

GetLanguageByAlias returns either the language related to the given alias and ok set to true or Otherlanguage and ok set to false if the alias is not recognized.

func GetLanguageByClassifier 

func GetLanguageByClassifier(content []byte, candidates []string) (language string, safe bool)

GetLanguageByClassifier returns the most probably language detected for the given content. It uses defaultClassifier, if no candidates are provided it returns OtherLanguage.

func GetLanguageByContent 

func GetLanguageByContent(filename string, content []byte) (language string, safe bool)

GetLanguageByContent returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByEmacsModeline 

func GetLanguageByEmacsModeline(content []byte) (language string, safe bool)

GetLanguageByEmacsModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByExtension 

func GetLanguageByExtension(filename string) (language string, safe bool)

GetLanguageByExtension returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByFilename 

func GetLanguageByFilename(filename string) (language string, safe bool)

GetLanguageByFilename returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByModeline 

func GetLanguageByModeline(content []byte) (language string, safe bool)

GetLanguageByModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByShebang 

func GetLanguageByShebang(content []byte) (language string, safe bool)

GetLanguageByShebang returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageByVimModeline 

func GetLanguageByVimModeline(content []byte) (language string, safe bool)

GetLanguageByVimModeline returns detected language. If there are more than one possibles languages it returns the first language by alphabetically order and safe to false.

func GetLanguageExtensions 

func GetLanguageExtensions(language string) []string

GetLanguageExtensions returns all extensions associated with the given language.

func GetLanguageGroup 

func GetLanguageGroup(language string) string

GetLanguageGroup returns language group or empty string if language does not have group.

func GetLanguageID 

func GetLanguageID(language string) (int, bool)

GetLanguageID returns the ID for the language. IDs are assigned by GitHub. The input must be the canonical language name. Aliases are not supported.

NOTE: The zero value (0) is a valid language ID, so this API mimics the Go map API. Use the second return value to check if the language was found.

func GetLanguageInfo 

func GetLanguageInfo(language string) (data.LanguageInfo, error)

GetLanguageInfo returns the LanguageInfo for a given language name, or an error if not found.

func GetLanguageInfoByID 

func GetLanguageInfoByID(id int) (data.LanguageInfo, error)

GetLanguageInfoByID returns the LanguageInfo for a given language ID, or an error if not found.

func GetLanguages 

func GetLanguages(filename string, content []byte) []string

GetLanguages applies a sequence of strategies based on the given filename and content to find out the most probable languages to return.

If it finds a strategy that produces a single result, it will be returned; otherise the last strategy that returned multiple results will be returned. If the content is binary, no results will be returned. This matches the behavior of Linguist.detect: https://github.com/github/linguist/blob/aad49acc0624c70d654a8dce447887dbbc713c7a/lib/linguist.rb#L14-L49

At least one of arguments should be set. If content is missing, language detection will be based on the filename. The function won't read the file, given an empty content.

func GetLanguagesByClassifier 

func GetLanguagesByClassifier(filename string, content []byte, candidates []string) (languages []string)

GetLanguagesByClassifier returns a sorted slice of possible languages ordered by decreasing language's probability. If there are not candidates it returns nil. It is a Strategy that uses a pre-trained defaultClassifier.

func GetLanguagesByContent 

func GetLanguagesByContent(filename string, content []byte, _ []string) []string

GetLanguagesByContent returns a slice of languages for the given content. It is a Strategy that uses content-based regexp heuristics and a filename extension.

func GetLanguagesByEmacsModeline 

func GetLanguagesByEmacsModeline(_ string, content []byte, _ []string) []string

GetLanguagesByEmacsModeline returns a slice of possible languages for the given content. It complies with the signature to be a Strategy type.

func GetLanguagesByExtension 

func GetLanguagesByExtension(filename string, _ []byte, _ []string) []string

GetLanguagesByExtension returns a slice of possible languages for the given filename. It complies with the signature to be a Strategy type.

func GetLanguagesByFilename 

func GetLanguagesByFilename(filename string, _ []byte, _ []string) []string

GetLanguagesByFilename returns a slice of possible languages for the given filename. It complies with the signature to be a Strategy type.

func GetLanguagesByManpage 

func GetLanguagesByManpage(filename string, _ []byte, _ []string) []string

GetLanguagesByManpage returns a slice of possible manpage languages for the given filename. It complies with the signature to be a Strategy type.

func GetLanguagesByModeline 

func GetLanguagesByModeline(_ string, content []byte, candidates []string) []string

GetLanguagesByModeline returns a slice of possible languages for the given content. It complies with the signature to be a Strategy type.

func GetLanguagesByShebang 

func GetLanguagesByShebang(_ string, content []byte, _ []string) (languages []string)

GetLanguagesByShebang returns a slice of possible languages for the given content. It complies with the signature to be a Strategy type.

func GetLanguagesByVimModeline 

func GetLanguagesByVimModeline(_ string, content []byte, _ []string) []string

GetLanguagesByVimModeline returns a slice of possible languages for the given content. It complies with the signature to be a Strategy type.

func GetLanguagesByXML 

func GetLanguagesByXML(_ string, content []byte, candidates []string) []string

GetLanguagesByXML returns a slice of possible XML language for the given filename. It complies with the signature to be a Strategy type.

func GetMIMEType 

func GetMIMEType(path string, language string) string

GetMIMEType returns a MIME type of a given file based on its languages.

func IsBinary 

func IsBinary(data []byte) bool

IsBinary detects if data is a binary value based on: http://git.kernel.org/cgit/git/git.git/tree/xdiff-interface.c?id=HEAD#n198

func IsConfiguration 

func IsConfiguration(path string) bool

IsConfiguration tells if filename is in one of the configuration languages.

func IsDocumentation 

func IsDocumentation(path string) bool

IsDocumentation returns whether or not path is a documentation path.

func IsDotFile 

func IsDotFile(path string) bool

IsDotFile returns whether or not path has dot as a prefix.

func IsGenerated 

func IsGenerated(path string, content []byte) bool

IsGenerated returns whether the file with the given path and content is a generated file.

func IsImage 

func IsImage(path string) bool

IsImage tells if a given file is an image (PNG, JPEG or GIF format).

func IsTest 

func IsTest(path string) bool

IsTest returns whether or not path is a test path.

func IsVendor 

func IsVendor(path string) bool

IsVendor returns whether or not path is a vendor path.

Types

type Strategy 

type Strategy func(filename string, content []byte, candidates []string) (languages []string)

Strategy type fix the signature for the functions that can be used as a strategy.

type Type 

type Type int

Type represent language's type. Either data, programming, markup, prose, or unknown.

func GetLanguageType 

func GetLanguageType(language string) (langType Type)

GetLanguageType returns the type of the given language.

Source Files

View all Source files

Directories

Path Synopsis
benchmarks
parser
cmd
enry
Package data contains only auto-generated data-structures for all the language identification strategies from the Linguist project sources.
 Package data contains only auto-generated data-structures for all the language identification strategies from the Linguist project sources.
rule
Package rule contains rule-based heuristic implementations.
 Package rule contains rule-based heuristic implementations.
internal
code-generator
code-generator/generator
Package generator provides facilities to generate Go code for the package data in enry from YAML files describing supported languages in Linguist.
 Package generator provides facilities to generate Go code for the package data in enry from YAML files describing supported languages in Linguist.
tokenizer
Package tokenizer implements file tokenization used by the enry content classifier.
 Package tokenizer implements file tokenization used by the enry content classifier.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.