lingua

package module

v1.4.0 Latest Latest Go to latest Published: Sep 5, 2023 License: Apache-2.0 Imports: 21 Imported by: 40

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/pemistahl/lingua-go

Links

Open Source Insights

README ¶

1. What does this library do?

Its task is simple: It tells you which language some text is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages.

2. Why does this library exist?

Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy.

So far, the only other comprehensive open source library in the Go ecosystem for this task is Whatlanggo. Unfortunately, it has two major drawbacks:

Detection only works with quite lengthy text fragments. For very short text snippets such as Twitter messages, it does not provide adequate results.
The more languages take part in the decision process, the less accurate are the detection results.

Lingua aims at eliminating these problems. She nearly does not need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. She draws on both rule-based and statistical methods but does not use any dictionaries of words. She does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.

3. Which languages are supported?

Compared to other language detection libraries, Lingua's focus is on quality over quantity, that is, getting detection right for a small set of languages first before adding new ones. Currently, the following 75 languages are supported:

A
- Afrikaans
- Albanian
- Arabic
- Armenian
- Azerbaijani
B
- Basque
- Belarusian
- Bengali
- Norwegian Bokmal
- Bosnian
- Bulgarian
C
- Catalan
- Chinese
- Croatian
- Czech
D
- Danish
- Dutch
E
- English
- Esperanto
- Estonian
F
- Finnish
- French
G
- Ganda
- Georgian
- German
- Greek
- Gujarati
H
- Hebrew
- Hindi
- Hungarian
I
- Icelandic
- Indonesian
- Irish
- Italian
J
- Japanese
K
- Kazakh
- Korean
L
- Latin
- Latvian
- Lithuanian
M
- Macedonian
- Malay
- Maori
- Marathi
- Mongolian
N
- Norwegian Nynorsk
P
- Persian
- Polish
- Portuguese
- Punjabi
R
- Romanian
- Russian
S
- Serbian
- Shona
- Slovak
- Slovene
- Somali
- Sotho
- Spanish
- Swahili
- Swedish
T
- Tagalog
- Tamil
- Telugu
- Thai
- Tsonga
- Tswana
- Turkish
U
- Ukrainian
- Urdu
V
- Vietnamese
W
- Welsh
X
- Xhosa
Y
- Yoruba
Z
- Zulu

4. How good is it?

Lingua is able to report accuracy statistics for some bundled test data available for each supported language. The test data for each language is split into three parts:

a list of single words with a minimum length of 5 characters
a list of word pairs with a minimum length of 10 characters
a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the Wortschatz corpora offered by Leipzig University, Germany. Data crawled from various news websites have been used for training, each corpus comprising one million sentences. For testing, corpora made of arbitrarily chosen websites have been used, each comprising ten thousand sentences. From each test corpus, a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of Lingua and Whatlanggo running over the data of Lingua's supported 75 languages. Additionally, I have added Google's CLD3 to the comparison with the help of the gocld3 bindings. Languages that are not supported by CLD3 or Whatlanggo are simply ignored during the detection process.

Each of the following sections contains two plots. The bar plot shows the detailed accuracy results for each supported language. The box plot illustrates the distributions of the accuracy values for each classifier. The boxes themselves represent the areas which the middle 50 % of data lie within. Within the colored boxes, the horizontal lines mark the median of the distributions.

4.1 Single word detection

Bar plot

4.2 Word pair detection

Bar plot

4.3 Sentence detection

Bar plot

4.4 Average detection

Bar plot

4.5 Mean, median and standard deviation

The table below shows detailed statistics for each language and classifier including mean, median and standard deviation.

Open table

Language	Average				Single Words				Word Pairs				Sentences
	Lingua (high accuracy mode)	Lingua (low accuracy mode)	CLD3	Whatlang	Lingua (high accuracy mode)	Lingua (low accuracy mode)	CLD3	Whatlang	Lingua (high accuracy mode)	Lingua (low accuracy mode)	CLD3	Whatlang	Lingua (high accuracy mode)	Lingua (low accuracy mode)	CLD3	Whatlang
Afrikaans	79	64	55	51	58	38	22	21	81	62	46	39	97	93	98	92
Albanian	88	80	55	-	69	54	18	-	95	86	48	-	100	99	98	-
Arabic	98	94	90	89	96	88	79	77	99	96	92	91	100	99	100	99
Armenian	100	100	99	-	100	100	100	-	100	100	100	-	100	100	97	-
Azerbaijani	90	82	81	64	77	71	62	45	92	78	82	58	99	96	99	91
Basque	84	75	62	-	71	56	33	-	87	76	62	-	93	92	92	-
Belarusian	97	92	84	81	92	80	67	64	99	95	86	80	100	100	100	98
Bengali	100	100	99	100	100	100	98	100	100	100	99	100	100	100	99	100
Bokmal	58	50	-	34	39	27	-	15	59	47	-	28	77	75	-	60
Bosnian	35	29	33	-	29	23	19	-	35	29	28	-	41	36	52	-
Bulgarian	87	78	70	61	70	56	45	37	91	81	66	57	99	96	98	89
Catalan	70	58	48	-	51	33	19	-	74	60	42	-	87	82	84	-
Chinese	100	100	92	100	100	100	92	100	100	100	83	100	100	100	100	100
Croatian	73	60	42	55	53	36	26	28	74	57	42	44	90	86	58	91
Czech	80	71	64	50	66	54	39	31	84	72	65	46	91	87	88	71
Danish	81	70	58	47	61	45	26	24	84	70	54	38	98	95	95	79
Dutch	77	64	58	47	55	36	29	22	81	61	47	36	96	94	97	82
English	81	63	54	49	55	29	22	17	89	62	44	35	99	97	97	94
Esperanto	84	66	57	52	67	44	22	25	85	61	51	45	98	93	98	88
Estonian	92	83	70	61	80	62	41	36	96	88	69	53	100	99	99	94
Finnish	96	91	80	71	90	77	58	45	98	95	84	70	100	100	99	98
French	89	77	55	64	74	52	22	37	94	83	49	59	99	98	94	97
Ganda	91	84	-	-	79	65	-	-	95	87	-	-	100	100	-	-
Georgian	100	100	98	100	100	100	99	100	100	100	100	100	100	100	96	100
German	89	80	66	65	74	57	40	38	94	84	62	60	100	99	98	97
Greek	100	100	100	100	100	100	100	100	100	100	100	100	100	100	100	100
Gujarati	100	100	100	100	100	100	99	100	100	100	100	100	100	100	100	100
Hebrew	100	100	-	90	100	100	-	76	100	100	-	94	100	100	-	99
Hindi	73	33	58	52	61	11	34	27	64	20	45	40	95	67	95	88
Hungarian	95	90	76	62	87	77	53	37	98	94	76	53	100	100	99	95
Icelandic	93	88	71	-	83	72	42	-	97	92	70	-	100	99	99	-
Indonesian	61	47	46	67	39	25	26	39	61	46	45	66	83	71	66	95
Irish	91	85	67	-	82	70	42	-	94	90	66	-	96	95	94	-
Italian	87	71	62	56	69	42	31	25	92	74	57	47	100	98	98	96
Japanese	100	100	98	99	100	100	97	100	100	100	96	100	100	100	100	97
Kazakh	92	90	82	-	80	78	62	-	96	93	83	-	99	99	99	-
Korean	100	100	99	100	100	100	100	100	100	100	100	100	100	100	98	100
Latin	87	73	62	-	72	49	44	-	93	76	58	-	97	94	83	-
Latvian	93	87	75	59	85	75	51	36	97	90	77	54	99	97	98	87
Lithuanian	95	87	72	62	86	76	42	38	98	89	75	56	100	98	99	92
Macedonian	84	72	60	62	66	52	30	39	86	70	54	55	99	95	97	94
Malay	31	31	22	-	26	22	11	-	38	36	22	-	28	35	34	-
Maori	91	82	52	-	82	62	22	-	92	87	43	-	99	98	91	-
Marathi	85	39	84	73	74	16	69	52	85	30	84	74	96	72	98	93
Mongolian	97	95	83	-	93	89	63	-	99	98	87	-	99	99	99	-
Nynorsk	66	52	-	34	41	25	-	10	66	49	-	24	91	81	-	69
Persian	90	80	76	70	78	62	57	46	94	80	70	66	100	98	99	99
Polish	95	90	77	66	85	77	51	45	98	93	80	59	100	99	99	94
Portuguese	81	69	53	57	59	42	21	26	85	70	40	48	99	95	97	96
Punjabi	100	100	100	100	100	100	99	100	100	100	100	100	100	100	100	100
Romanian	87	72	53	59	69	49	24	34	92	74	48	52	99	94	88	90
Russian	90	78	71	53	76	59	48	40	95	84	72	52	98	92	93	68
Serbian	88	78	78	57	74	62	63	34	90	80	75	51	99	91	95	86
Shona	91	81	76	68	78	56	51	44	96	86	79	65	100	100	99	95
Slovak	84	75	63	-	64	49	32	-	90	78	61	-	99	97	96	-
Slovene	82	67	63	48	61	39	29	25	87	68	60	38	99	93	99	81
Somali	92	85	69	68	82	64	38	38	96	90	70	66	100	100	100	99
Sotho	86	72	49	-	67	43	15	-	90	75	33	-	100	97	98	-
Spanish	70	56	48	48	44	26	16	19	69	49	32	33	97	94	96	93
Swahili	81	70	57	-	60	43	25	-	84	68	49	-	98	97	98	-
Swedish	84	72	61	49	64	46	30	24	88	76	56	40	99	94	96	83
Tagalog	78	66	-	52	52	36	-	23	83	67	-	43	98	96	-	90
Tamil	100	100	100	100	100	100	100	100	100	100	100	100	100	100	99	100
Telugu	100	100	99	100	100	100	99	100	100	100	100	100	100	100	99	100
Thai	100	100	99	100	100	100	100	100	100	100	100	100	100	100	98	99
Tsonga	84	72	-	-	66	46	-	-	89	73	-	-	98	97	-	-
Tswana	84	71	-	-	65	44	-	-	88	73	-	-	99	96	-	-
Turkish	94	87	69	54	84	71	41	26	98	91	70	44	100	100	97	92
Ukrainian	92	86	81	72	84	75	62	53	97	92	83	71	95	93	98	93
Urdu	91	80	61	57	80	65	39	31	94	78	53	46	98	96	92	94
Vietnamese	91	87	66	73	79	76	26	36	94	87	74	85	99	98	99	97
Welsh	91	82	69	-	78	61	43	-	96	87	66	-	99	99	98	-
Xhosa	82	69	66	-	64	45	40	-	85	67	65	-	98	94	92	-
Yoruba	74	62	15	22	50	33	5	11	77	61	11	14	96	92	28	41
Zulu	81	70	63	70	62	45	35	44	83	72	63	68	97	94	92	98

Mean	86	77	69	67	74	61	48	48	89	78	67	63	96	93	93	91

Median	89.0	80.0	68.0	62.0	74.0	57.0	41.0	38.0	94.0	81.0	66.0	57.0	99.0	97.0	98.0	94.0
Standard Deviation	13.08	17.29	19.04	20.22	18.41	24.9	27.86	29.28	13.12	18.93	21.83	24.22	11.05	11.91	13.95	11.24

5. Why is it better than other libraries?

Every language detector uses a probabilistic n-gram model trained on the character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.

A second important difference is that Lingua does not only use such a statistical model, but also a rule-based engine. This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration. This makes sense because loading less language models means less memory consumption and better runtime performance.

In general, it is always a good idea to restrict the set of languages to be considered in the classification process using the respective api methods. If you know beforehand that certain languages are never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.

6. Test report generation

If you want to reproduce the accuracy results above, you can generate the test reports yourself for both classifiers and all languages by doing:

cd cmd
go run accuracy_reporter.go

For gocld3 to run successfully, you need to install the exact version 3.17.3 of Google's protocol buffers which is a bit unfortunate. For each detector and language, a test report file is then written into /accuracy-reports. As an example, here is the current output of the Lingua German report:

##### German #####

>>> Accuracy on average: 89.23%

>> Detection of 1000 single words (average length: 9 chars)
Accuracy: 73.90%
Erroneously classified as Dutch: 2.30%, Danish: 2.10%, English: 2.00%, Latin: 1.90%, Bokmal: 1.60%, Basque: 1.20%, French: 1.20%, Italian: 1.20%, Esperanto: 1.10%, Swedish: 1.00%, Afrikaans: 0.80%, Tsonga: 0.70%, Nynorsk: 0.60%, Portuguese: 0.60%, Yoruba: 0.60%, Finnish: 0.50%, Sotho: 0.50%, Welsh: 0.50%, Estonian: 0.40%, Irish: 0.40%, Polish: 0.40%, Spanish: 0.40%, Swahili: 0.40%, Tswana: 0.40%, Bosnian: 0.30%, Icelandic: 0.30%, Tagalog: 0.30%, Albanian: 0.20%, Catalan: 0.20%, Croatian: 0.20%, Indonesian: 0.20%, Lithuanian: 0.20%, Maori: 0.20%, Romanian: 0.20%, Xhosa: 0.20%, Zulu: 0.20%, Latvian: 0.10%, Malay: 0.10%, Slovak: 0.10%, Slovene: 0.10%, Somali: 0.10%, Turkish: 0.10%

>> Detection of 1000 word pairs (average length: 18 chars)
Accuracy: 94.10%
Erroneously classified as Dutch: 0.90%, Latin: 0.80%, English: 0.70%, Swedish: 0.60%, Danish: 0.50%, French: 0.40%, Bokmal: 0.30%, Irish: 0.20%, Tagalog: 0.20%, Afrikaans: 0.10%, Esperanto: 0.10%, Estonian: 0.10%, Finnish: 0.10%, Italian: 0.10%, Maori: 0.10%, Nynorsk: 0.10%, Somali: 0.10%, Swahili: 0.10%, Tsonga: 0.10%, Turkish: 0.10%, Welsh: 0.10%, Zulu: 0.10%

>> Detection of 1000 sentences (average length: 111 chars)
Accuracy: 99.70%
Erroneously classified as Dutch: 0.20%, Latin: 0.10%

7. How to add it to your project?

go get github.com/pemistahl/lingua-go

8. How to build?

Lingua requires at least Go version 1.18.

git clone https://github.com/pemistahl/lingua-go.git
cd lingua-go
go build

The source code is accompanied by an extensive unit test suite. To run the tests, simply say:

go test

9. How to use?

9.1 Basic usage

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    if language, exists := detector.DetectLanguageOf("languages are awesome"); exists {
        fmt.Println(language)
    }

    // Output: English
}

9.2 Minimum relative distance

By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language. The word prologue, for instance, is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed up probabilities for each possible language have to satisfy. It can be stated in the following way:

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        WithMinimumRelativeDistance(0.9).
        Build()

    language, exists := detector.DetectLanguageOf("languages are awesome")

    fmt.Println(language)
    fmt.Println(exists)

    // Output:
    // Unknown
    // false
}

Be aware that the distance between the language probabilities is dependent on the length of the input text. The longer the input text, the larger the distance between the languages. So if you want to classify very short text phrases, do not set the minimum relative distance too high. Otherwise Unknown will be returned most of the time as in the example above. This is the return value for cases where language detection is not reliably possible. This value is not meant to be included in the set of input languages when building the language detector. If you include it, it will be automatically removed from the set of input languages.

9.3 Confidence values

Knowing about the most likely language is nice but how reliable is the computed likelihood? And how less likely are the other examined languages in comparison to the most likely one? These questions can be answered as well:

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    confidenceValues := detector.ComputeLanguageConfidenceValues("languages are awesome")

    for _, elem := range confidenceValues {
        fmt.Printf("%s: %.2f\n", elem.Language(), elem.Value())
    }

    // Output:
    // English: 0.93
    // French: 0.04
    // German: 0.02
    // Spanish: 0.01
}

In the example above, a slice of ConfidenceValue is returned containing all possible languages sorted by their confidence value in descending order. Each value is a probability between 0.0 and 1.0. The probabilities of all languages will sum to 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned for this language. The other languages will receive a value of 0.0.

There is also a method for returning the confidence value for one specific language only:

confidence := detector.ComputeLanguageConfidence("languages are awesome", lingua.French)
fmt.Printf("%.2f", confidence)

// Output:
// 0.04

The value that this method computes is a number between 0.0 and 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned. If the given language is not supported by this detector instance, the value 0.0 will always be returned.

9.4 Eager loading versus lazy loading

By default, Lingua uses lazy-loading to load only those language models on demand which are considered relevant by the rule-based filter engine. For web services, for instance, it is rather beneficial to preload all language models into memory to avoid unexpected latency while waiting for the service response. If you want to enable the eager-loading mode, you can do it like this:

lingua.NewLanguageDetectorBuilder().
    FromAllLanguages().
    WithPreloadedLanguageModels().
    Build()

Multiple instances of LanguageDetector share the same language models in memory which are accessed asynchronously by the instances.

9.5 Low accuracy mode versus high accuracy mode

Lingua's high detection accuracy comes at the cost of being noticeably slower than other language detectors. The large language models also consume significant amounts of memory. These requirements might not be feasible for systems running low on resources. If you want to classify mostly long texts or need to save resources, you can enable a low accuracy mode that loads only a small subset of the language models into memory:

lingua.NewLanguageDetectorBuilder().
    FromAllLanguages().
    WithLowAccuracyMode().
    Build()

The downside of this approach is that detection accuracy for short texts consisting of less than 120 characters will drop significantly. However, detection accuracy for texts which are longer than 120 characters will remain mostly unaffected.

In high accuracy mode (the default), the language detector consumes approximately 1,800 MB of memory if all language models are loaded. In low accuracy mode, memory consumption is reduced to approximately 110 MB. The goal is to further reduce memory consumption in later releases.

An alternative for a smaller memory footprint and faster performance is to reduce the set of languages when building the language detector. In most cases, it is not advisable to build the detector from all supported languages. When you have knowledge about the texts you want to classify you can almost always rule out certain languages as impossible or unlikely to occur.

9.6 Detection of multiple languages in mixed-language texts

In contrast to most other language detectors, Lingua is able to detect multiple languages in mixed-language texts. This feature can yield quite reasonable results but it is still in an experimental state and therefore the detection result is highly dependent on the input text. It works best in high-accuracy mode with multiple long words for each language. The shorter the phrases and their words are, the less accurate are the results. Reducing the set of languages when building the language detector can also improve accuracy for this task if the languages occurring in the text are equal to the languages supported by the respective language detector instance.

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    sentence := "Parlez-vous français? " + 
        "Ich spreche Französisch nur ein bisschen. " +
        "A little bit is better than nothing."

    for _, result := range detector.DetectMultipleLanguagesOf(sentence) {
        fmt.Printf("%s: '%s'\n", result.Language(), sentence[result.StartIndex():result.EndIndex()])
    }

    // Output:
    // French: 'Parlez-vous français? '
    // German: 'Ich spreche Französisch nur ein bisschen. '
    // English: 'A little bit is better than nothing.'
}

In the example above, a slice of DetectionResult is returned. Each entry in the slice describes a contiguous single-language text section, providing start and end indices of the respective substring.

9.7 Methods to build the LanguageDetector

There might be classification tasks where you know beforehand that your language data is definitely not written in Latin, for instance. The detection accuracy can become better in such cases if you exclude certain languages from the decision process or just explicitly include relevant languages:

// Include all languages available in the library.
lingua.NewLanguageDetectorBuilder().FromAllLanguages()

// Include only languages that are not yet extinct (= currently excludes Latin).
lingua.NewLanguageDetectorBuilder().FromAllSpokenLanguages()

// Include only languages written with Cyrillic script.
lingua.NewLanguageDetectorBuilder().FromAllLanguagesWithCyrillicScript()

// Exclude only the Spanish language from the decision algorithm.
lingua.NewLanguageDetectorBuilder().FromAllLanguagesWithout(lingua.Spanish)

// Only decide between English and German.
lingua.NewLanguageDetectorBuilder().FromLanguages(lingua.English, lingua.German)

// Select languages by ISO 639-1 code.
lingua.NewLanguageDetectorBuilder().FromIsoCodes639_1(lingua.EN, lingua.DE)

// Select languages by ISO 639-3 code.
lingua.NewLanguageDetectorBuilder().FromIsoCodes639_3(lingua.ENG, lingua.DEU)

10. What's next for version 1.5.0?

Take a look at the planned issues.

11. Contributions

Any contributions to Lingua are very much appreciated. Please read the instructions in CONTRIBUTING.md for how to add new languages to the library.

Documentation ¶

Overview ¶

Package lingua accurately detects the natural language of written text, be it long or short.

What this library does ¶

Its task is simple: It tells you which language some text is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages.

Why this library exists ¶

Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy.

So far, the only other comprehensive open source library in the Go ecosystem for this task is Whatlanggo (https://github.com/abadojack/whatlanggo). Unfortunately, it has two major drawbacks:

1. Detection only works with quite lengthy text fragments. For very short text snippets such as Twitter messages, it does not provide adequate results.

2. The more languages take part in the decision process, the less accurate are the detection results.

Lingua aims at eliminating these problems. It nearly does not need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. It draws on both rule-based and statistical methods but does not use any dictionaries of words. It does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.

Supported languages ¶

Compared to other language detection libraries, Lingua's focus is on quality over quantity, that is, getting detection right for a small set of languages first before adding new ones. Currently, 75 languages are supported. They are listed as variants of type Language.

How good it is ¶

Lingua is able to report accuracy statistics for some bundled test data available for each supported language. The test data for each language is split into three parts:

1. a list of single words with a minimum length of 5 characters

2. a list of word pairs with a minimum length of 10 characters

3. a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the Wortschatz corpora (https://wortschatz.uni-leipzig.de) offered by Leipzig University, Germany. Data crawled from various news websites have been used for training, each corpus comprising one million sentences. For testing, corpora made of arbitrarily chosen websites have been used, each comprising ten thousand sentences. From each test corpus, a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of Lingua, and Whatlanggo running over the data of Lingua's supported 75 languages. Additionally, I have added Google's CLD3 (https://github.com/google/cld3/) to the comparison with the help of the gocld3 bindings (https://github.com/jmhodges/gocld3). Languages that are not supported by CLD3 or Whatlanggo are simply ignored during the detection process. Lingua clearly outperforms its contenders.

Why it is better than other libraries ¶

Every language detector uses a probabilistic n-gram (https://en.wikipedia.org/wiki/N-gram) model trained on the character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.

A second important difference is that Lingua does not only use such a statistical model, but also a rule-based engine. This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration. This makes sense because loading less language models means less memory consumption and better runtime performance.

In general, it is always a good idea to restrict the set of languages to be considered in the classification process using the respective api methods. If you know beforehand that certain languages are never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.

Example (Basic) ¶

languages := []lingua.Language{
	lingua.English,
	lingua.French,
	lingua.German,
	lingua.Spanish,
}

detector := lingua.NewLanguageDetectorBuilder().
	FromLanguages(languages...).
	Build()

if language, exists := detector.DetectLanguageOf("languages are awesome"); exists {
	fmt.Println(language)
}

Output:

English

Example (BuilderApi) ¶

There might be classification tasks where you know beforehand that your language data is definitely not written in Latin, for instance. The detection accuracy can become better in such cases if you exclude certain languages from the decision process or just explicitly include relevant languages.

// Include all languages available in the library.
lingua.NewLanguageDetectorBuilder().FromAllLanguages()

// Include only languages that are not yet extinct (= currently excludes Latin).
lingua.NewLanguageDetectorBuilder().FromAllSpokenLanguages()

// Include only languages written with Cyrillic script.
lingua.NewLanguageDetectorBuilder().FromAllLanguagesWithCyrillicScript()

// Exclude only the Spanish language from the decision algorithm.
lingua.NewLanguageDetectorBuilder().FromAllLanguagesWithout(lingua.Spanish)

// Only decide between English and German.
lingua.NewLanguageDetectorBuilder().FromLanguages(lingua.English, lingua.German)

// Select languages by ISO 639-1 code.
lingua.NewLanguageDetectorBuilder().FromIsoCodes639_1(lingua.EN, lingua.DE)

// Select languages by ISO 639-3 code.
lingua.NewLanguageDetectorBuilder().FromIsoCodes639_3(lingua.ENG, lingua.DEU)

Output:

Example (ConfidenceValues) ¶

Knowing about the most likely language is nice but how reliable is the computed likelihood? And how less likely are the other examined languages in comparison to the most likely one? In the example below, a slice of ConfidenceValue is returned containing those languages which the calling instance of LanguageDetector has been built from. The entries are sorted by their confidence value in descending order. Each value is a probability between 0.0 and 1.0. The probabilities of all languages will sum to 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned for this language. The other languages will receive a value of 0.0.

languages := []lingua.Language{
	lingua.English,
	lingua.French,
	lingua.German,
	lingua.Spanish,
}

detector := lingua.NewLanguageDetectorBuilder().
	FromLanguages(languages...).
	Build()

confidenceValues := detector.ComputeLanguageConfidenceValues("languages are awesome")

for _, elem := range confidenceValues {
	fmt.Printf("%s: %.2f\n", elem.Language(), elem.Value())
}

Output:

English: 0.93
French: 0.04
German: 0.02
Spanish: 0.01

Example (EagerLoading) ¶

By default, Lingua uses lazy-loading to load only those language models on demand which are considered relevant by the rule-based filter engine. For web services, for instance, it is rather beneficial to preload all language models into memory to avoid unexpected latency while waiting for the service response. If you want to enable the eager-loading mode, you can do it as seen below. Multiple instances of LanguageDetector share the same language models in memory which are accessed asynchronously by the instances.

lingua.NewLanguageDetectorBuilder().
	FromAllLanguages().
	WithPreloadedLanguageModels().
	Build()

Output:

Example (MinimumRelativeDistance) ¶

By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language. The word `prologue`, for instance, is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed up probabilities for each possible language have to satisfy. It can be stated as seen below.

Be aware that the distance between the language probabilities is dependent on the length of the input text. The longer the input text, the larger the distance between the languages. So if you want to classify very short text phrases, do not set the minimum relative distance too high. Otherwise Unknown will be returned most of the time as in the example below. This is the return value for cases where language detection is not reliably possible.

languages := []lingua.Language{
	lingua.English,
	lingua.French,
	lingua.German,
	lingua.Spanish,
}

detector := lingua.NewLanguageDetectorBuilder().
	FromLanguages(languages...).
	WithMinimumRelativeDistance(0.9).
	Build()

language, exists := detector.DetectLanguageOf("languages are awesome")

fmt.Println(language)
fmt.Println(exists)

Output:

Unknown
false

Example (MultipleLanguagesDetection) ¶

languages := []lingua.Language{
	lingua.English,
	lingua.French,
	lingua.German,
}

detector := lingua.NewLanguageDetectorBuilder().
	FromLanguages(languages...).
	Build()

sentence := "Parlez-vous français? " +
	"Ich spreche Französisch nur ein bisschen. " +
	"A little bit is better than nothing."

for _, result := range detector.DetectMultipleLanguagesOf(sentence) {
	fmt.Printf("%s: '%s'\n", result.Language(), sentence[result.StartIndex():result.EndIndex()])
}

Output:

French: 'Parlez-vous français? '
German: 'Ich spreche Französisch nur ein bisschen. '
English: 'A little bit is better than nothing.'

Index ¶

func CreateAndWriteLanguageModelFiles(inputFilePath string, outputDirectoryPath string, language Language, ...) error
func CreateAndWriteTestDataFiles(inputFilePath string, outputDirectoryPath string, charClass string, ...) error
type ConfidenceValue
type DetectionResult
type IsoCode639_1
- func GetIsoCode639_1FromValue(name string) IsoCode639_1
- func (i IsoCode639_1) String() string
type IsoCode639_3
- func GetIsoCode639_3FromValue(name string) IsoCode639_3
- func (i IsoCode639_3) String() string
type Language
type LanguageDetector
type LanguageDetectorBuilder
type UnconfiguredLanguageDetectorBuilder
- func NewLanguageDetectorBuilder() UnconfiguredLanguageDetectorBuilder

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func CreateAndWriteLanguageModelFiles ¶

func CreateAndWriteLanguageModelFiles(
	inputFilePath string,
	outputDirectoryPath string,
	language Language,
	charClass string,
) error

CreateAndWriteLanguageModelFiles creates language model files for accuracy report generation and writes them to a directory.

`inputFilePath` is the path to a txt file used for language model creation. The assumed enconding of the txt file is UTF-8.

`outputDirectoryPath` is the path to an existing directory where the language model files are to be written.

`language` is the language for which to create the language models.

`charClass` is a regex character class such as `\\p{L}` to restrict the set of characters that the language models are built from.

An error is returned if:

- the input file path is not absolute or does not point to an existing txt file

- the input file's encoding is not UTF-8

- the output directory path is not absolute or does not point to an existing directory

Panics if the character class cannot be compiled to a valid regular expression.

func CreateAndWriteTestDataFiles ¶

func CreateAndWriteTestDataFiles(
	inputFilePath string,
	outputDirectoryPath string,
	charClass string,
	maximumLines int,
) error

CreateAndWriteTestDataFiles creates test data files for accuracy report generation and writes them to a directory.

`inputFilePath` is the path to a txt file used for test data creation. The assumed enconding of the txt file is UTF-8.

`outputDirectoryPath` is the path to an existing directory where the test data files are to be written.

`charClass` is a regex character class such as `\\p{L}` to restrict the set of characters that the language models are built from.

`maximumLines` is the maximum number of lines each test data file should have.

An error is returned if:

- the input file path is not absolute or does not point to an existing txt file

- the input file's encoding is not UTF-8

- the output directory path is not absolute or does not point to an existing directory

Panics if the character class cannot be compiled to a valid regular expression.

Types ¶

type ConfidenceValue ¶

type ConfidenceValue interface {
	// Language returns the language being part of this ConfidenceValue.
	Language() Language

	// Value returns a language's confidence value which lies between 0.0 and 1.0.
	Value() float64
}

ConfidenceValue is the interface describing a language's confidence value that is computed by LanguageDetector.ComputeLanguageConfidenceValues.

type DetectionResult ¶ added in v1.2.0

type DetectionResult interface {
	// StartIndex returns the start index of the identified single-language substring.
	StartIndex() int
	// EndIndex returns the end index of the identified single-language substring.
	EndIndex() int
	// Language returns the language being part of this DetectionResult.
	Language() Language
}

DetectionResult is the interface describing a contiguous single-language text section within a possibly mixed-language text. It is computed by LanguageDetector.DetectMultipleLanguagesOf.

type IsoCode639_1 ¶

type IsoCode639_1 int

IsoCode639_1 is the type used for enumerating the ISO 639-1 code representations of the supported languages.

const (
	// AF is the ISO 639-1 code for Afrikaans.
	AF IsoCode639_1 = iota

	// AR is the ISO 639-1 code for Arabic.
	AR

	// AZ is the ISO 639-1 code for Azerbaijani.
	AZ

	// BE is the ISO 639-1 code for Belarusian.
	BE

	// BG is the ISO 639-1 code for Bulgarian.
	BG

	// BN is the ISO 639-1 code for Bengali.
	BN

	// BS is the ISO 639-1 code for Bosnian.
	BS

	// CA is the ISO 639-1 code for Catalan.
	CA

	// CS is the ISO 639-1 code for Czech.
	CS

	// CY is the ISO 639-1 code for Welsh.
	CY

	// DA is the ISO 639-1 code for Danish.
	DA

	// DE is the ISO 639-1 code for German.
	DE

	// EL is the ISO 639-1 code for Greek.
	EL

	// EN is the ISO 639-1 code for English.
	EN

	// EO is the ISO 639-1 code for Esperanto.
	EO

	// ES is the ISO 639-1 code for Spanish.
	ES

	// ET is the ISO 639-1 code for Estonian.
	ET

	// EU is the ISO 639-1 code for Basque.
	EU

	// FA is the ISO 639-1 code for Persian.
	FA

	// FI is the ISO 639-1 code for Finnish.
	FI

	// FR is the ISO 639-1 code for French.
	FR

	// GA is the ISO 639-1 code for Irish.
	GA

	// GU is the ISO 639-1 code for Gujarati.
	GU

	// HE is the ISO 639-1 code for Hebrew.
	HE

	// HI is the ISO 639-1 code for Hindi.
	HI

	// HR is the ISO 639-1 code for Croatian.
	HR

	// HU is the ISO 639-1 code for Hungarian.
	HU

	// HY is the ISO 639-1 code for Armenian.
	HY

	// ID is the ISO 639-1 code for Indonesian.
	ID

	// IS is the ISO 639-1 code for Icelandic.
	IS

	// IT is the ISO 639-1 code for Italian.
	IT

	// JA is the ISO 639-1 code for Japanese.
	JA

	// KA is the ISO 639-1 code for Georgian.
	KA

	// KK is the ISO 639-1 code for Kazakh.
	KK

	// KO is the ISO 639-1 code for Korean.
	KO

	// LA is the ISO 639-1 code for Latin.
	LA

	// LG is the ISO 639-1 code for Ganda.
	LG

	// LT is the ISO 639-1 code for Lithuanian.
	LT

	// LV is the ISO 639-1 code for Latvian.
	LV

	// MI is the ISO 639-1 code for Maori.
	MI

	// MK is the ISO 639-1 code for Macedonian.
	MK

	// MN is the ISO 639-1 code for Mongolian.
	MN

	// MR is the ISO 639-1 code for Marathi.
	MR

	// MS is the ISO 639-1 code for Malay.
	MS

	// NB is the ISO 639-1 code for Bokmal.
	NB

	// NL is the ISO 639-1 code for Dutch.
	NL

	// NN is the ISO 639-1 code for Nynorsk.
	NN

	// PA is the ISO 639-1 code for Punjabi.
	PA

	// PL is the ISO 639-1 code for Polish.
	PL

	// PT is the ISO 639-1 code for Portuguese.
	PT

	// RO is the ISO 639-1 code for Romanian.
	RO

	// RU is the ISO 639-1 code for Russian.
	RU

	// SK is the ISO 639-1 code for Slovak.
	SK

	// SL is the ISO 639-1 code for Slovene.
	SL

	// SN is the ISO 639-1 code for Shona.
	SN

	// SO is the ISO 639-1 code for Somali.
	SO

	// SQ is the ISO 639-1 code for Albanian.
	SQ

	// SR is the ISO 639-1 code for Serbian.
	SR

	// ST is the ISO 639-1 code for Sotho.
	ST

	// SV is the ISO 639-1 code for Swedish.
	SV

	// SW is the ISO 639-1 code for Swahili.
	SW

	// TA is the ISO 639-1 code for Tamil.
	TA

	// TE is the ISO 639-1 code for Telugu.
	TE

	// TH is the ISO 639-1 code for Thai.
	TH

	// TL is the ISO 639-1 code for Tagalog.
	TL

	// TN is the ISO 639-1 code for Tswana.
	TN

	// TR is the ISO 639-1 code for Turkish.
	TR

	// TS is the ISO 639-1 code for Tsonga.
	TS

	// UK is the ISO 639-1 code for Ukrainian.
	UK

	// UR is the ISO 639-1 code for Urdu.
	UR

	// VI is the ISO 639-1 code for Vietnamese.
	VI

	// XH is the ISO 639-1 code for Xhosa.
	XH

	// YO is the ISO 639-1 code for Yoruba.
	YO

	// ZH is the ISO 639-1 code for Chinese.
	ZH

	// ZU is the ISO 639-1 code for Zulu.
	ZU

	// UnknownIsoCode639_1 is the ISO 639-1 code for Unknown.
	UnknownIsoCode639_1
)

func GetIsoCode639_1FromValue ¶ added in v1.4.0

func GetIsoCode639_1FromValue(name string) IsoCode639_1

GetIsoCode639_1FromValue returns the ISO 639-1 code for the given name.

func (IsoCode639_1) String ¶

func (i IsoCode639_1) String() string

type IsoCode639_3 ¶

type IsoCode639_3 int

IsoCode639_3 is the type used for enumerating the ISO 639-3 code representations of the supported languages.

const (
	// AFR is the ISO 639-3 code for Afrikaans.
	AFR IsoCode639_3 = iota

	// ARA is the ISO 639-3 code for Arabic.
	ARA

	// AZE is the ISO 639-3 code for Azerbaijani.
	AZE

	// BEL is the ISO 639-3 code for Belarusian.
	BEL

	// BEN is the ISO 639-3 code for Bengali.
	BEN

	// BOS is the ISO 639-3 code for Bosnian.
	BOS

	// BUL is the ISO 639-3 code for Bulgarian.
	BUL

	// CAT is the ISO 639-3 code for Catalan.
	CAT

	// CES is the ISO 639-3 code for Czech.
	CES

	// CYM is the ISO 639-3 code for Welsh.
	CYM

	// DAN is the ISO 639-3 code for Danish.
	DAN

	// DEU is the ISO 639-3 code for German.
	DEU

	// ELL is the ISO 639-3 code for Greek.
	ELL

	// ENG is the ISO 639-3 code for English.
	ENG

	// EPO is the ISO 639-3 code for Esperanto.
	EPO

	// EST is the ISO 639-3 code for Estonian.
	EST

	// EUS is the ISO 639-3 code for Basque.
	EUS

	// FAS is the ISO 639-3 code for Persian.
	FAS

	// FIN is the ISO 639-3 code for Finnish.
	FIN

	// FRA is the ISO 639-3 code for French.
	FRA

	// GLE is the ISO 639-3 code for Irish.
	GLE

	// GUJ is the ISO 639-3 code for Gujarati.
	GUJ

	// HEB is the ISO 639-3 code for Hebrew.
	HEB

	// HIN is the ISO 639-3 code for Hindi.
	HIN

	// HRV is the ISO 639-3 code for Croatian.
	HRV

	// HUN is the ISO 639-3 code for Hungarian.
	HUN

	// HYE is the ISO 639-3 code for Armenian.
	HYE

	// IND is the ISO 639-3 code for Indonesian.
	IND

	// ISL is the ISO 639-3 code for Icelandic.
	ISL

	// ITA is the ISO 639-3 code for Italian.
	ITA

	// JPN is the ISO 639-3 code for Japanese.
	JPN

	// KAT is the ISO 639-3 code for Georgian.
	KAT

	// KAZ is the ISO 639-3 code for Kazakh.
	KAZ

	// KOR is the ISO 639-3 code for Korean.
	KOR

	// LAT is the ISO 639-3 code for Latin.
	LAT

	// LAV is the ISO 639-3 code for Latvian.
	LAV

	// LIT is the ISO 639-3 code for Lithuanian.
	LIT

	// LUG is the ISO 639-3 code for Ganda.
	LUG

	// MAR is the ISO 639-3 code for Marathi.
	MAR

	// MKD is the ISO 639-3 code for Macedonian.
	MKD

	// MON is the ISO 639-3 code for Mongolian.
	MON

	// MRI is the ISO 639-3 code for Maori.
	MRI

	// MSA is the ISO 639-3 code for Malay.
	MSA

	// NLD is the ISO 639-3 code for Dutch.
	NLD

	// NNO is the ISO 639-3 code for Nynorsk.
	NNO

	// NOB is the ISO 639-3 code for Bokmal.
	NOB

	// PAN is the ISO 639-3 code for Punjabi.
	PAN

	// POL is the ISO 639-3 code for Polish.
	POL

	// POR is the ISO 639-3 code for Portuguese.
	POR

	// RON is the ISO 639-3 code for Romanian.
	RON

	// RUS is the ISO 639-3 code for Russian.
	RUS

	// SLK is the ISO 639-3 code for Slovak.
	SLK

	// SLV is the ISO 639-3 code for Slovene.
	SLV

	// SNA is the ISO 639-3 code for Shona.
	SNA

	// SOM is the ISO 639-3 code for Somali.
	SOM

	// SOT is the ISO 639-3 code for Sotho.
	SOT

	// SPA is the ISO 639-3 code for Spanish.
	SPA

	// SQI is the ISO 639-3 code for Albanian.
	SQI

	// SRP is the ISO 639-3 code for Serbian.
	SRP

	// SWA is the ISO 639-3 code for Swahili.
	SWA

	// SWE is the ISO 639-3 code for Swedish.
	SWE

	// TAM is the ISO 639-3 code for Tamil.
	TAM

	// TEL is the ISO 639-3 code for Telugu.
	TEL

	// TGL is the ISO 639-3 code for Tagalog.
	TGL

	// THA is the ISO 639-3 code for Thai.
	THA

	// TSN is the ISO 639-3 code for Tswana.
	TSN

	// TSO is the ISO 639-3 code for Tsonga.
	TSO

	// TUR is the ISO 639-3 code for Turkish.
	TUR

	// UKR is the ISO 639-3 code for Ukrainian.
	UKR

	// URD is the ISO 639-3 code for Urdu.
	URD

	// VIE is the ISO 639-3 code for Vietnamese.
	VIE

	// XHO is the ISO 639-3 code for Xhosa.
	XHO

	// YOR is the ISO 639-3 code for Yoruba.
	YOR

	// ZHO is the ISO 639-3 code for Chinese.
	ZHO

	// ZUL is the ISO 639-3 code for Zulu.
	ZUL

	// UnknownIsoCode639_3 is the ISO 639-3 code for Unknown.
	UnknownIsoCode639_3
)

func GetIsoCode639_3FromValue ¶ added in v1.4.0

func GetIsoCode639_3FromValue(name string) IsoCode639_3

GetIsoCode639_3FromValue returns the ISO 639-3 code for the given name.

func (IsoCode639_3) String ¶

func (i IsoCode639_3) String() string

type Language ¶

type Language int

Language is the type used for enumerating the so far 75 languages which can be detected by Lingua.

const (
	Afrikaans Language = iota
	Albanian
	Arabic
	Armenian
	Azerbaijani
	Basque
	Belarusian
	Bengali
	Bokmal
	Bosnian
	Bulgarian
	Catalan
	Chinese
	Croatian
	Czech
	Danish
	Dutch
	English
	Esperanto
	Estonian
	Finnish
	French
	Ganda
	Georgian
	German
	Greek
	Gujarati
	Hebrew
	Hindi
	Hungarian
	Icelandic
	Indonesian
	Irish
	Italian
	Japanese
	Kazakh
	Korean
	Latin
	Latvian
	Lithuanian
	Macedonian
	Malay
	Maori
	Marathi
	Mongolian
	Nynorsk
	Persian
	Polish
	Portuguese
	Punjabi
	Romanian
	Russian
	Serbian
	Shona
	Slovak
	Slovene
	Somali
	Sotho
	Spanish
	Swahili
	Swedish
	Tagalog
	Tamil
	Telugu
	Thai
	Tsonga
	Tswana
	Turkish
	Ukrainian
	Urdu
	Vietnamese
	Welsh
	Xhosa
	Yoruba
	Zulu
	Unknown
)

func AllLanguages ¶

func AllLanguages() []Language

AllLanguages returns a sorted slice of all currently supported languages.

func AllLanguagesWithArabicScript ¶

func AllLanguagesWithArabicScript() []Language

AllLanguagesWithArabicScript returns a sorted slice of all built-in languages supporting the Arabic script.

func AllLanguagesWithCyrillicScript ¶

func AllLanguagesWithCyrillicScript() []Language

AllLanguagesWithCyrillicScript returns a sorted slice of all built-in languages supporting the Cyrillic script.

func AllLanguagesWithDevanagariScript ¶

func AllLanguagesWithDevanagariScript() []Language

AllLanguagesWithDevanagariScript returns a sorted slice of all built-in languages supporting the Devanagari script.

func AllLanguagesWithLatinScript ¶

func AllLanguagesWithLatinScript() []Language

AllLanguagesWithLatinScript returns a sorted slice of all built-in languages supporting the Latin script.

func AllSpokenLanguages ¶

func AllSpokenLanguages() (languages []Language)

AllSpokenLanguages returns a sorted slice of all supported languages that are not extinct but still spoken.

func GetLanguageFromIsoCode639_1 ¶

func GetLanguageFromIsoCode639_1(isoCode IsoCode639_1) Language

GetLanguageFromIsoCode639_1 returns the language for the given ISO 639-1 code enum value.

func GetLanguageFromIsoCode639_3 ¶

func GetLanguageFromIsoCode639_3(isoCode IsoCode639_3) Language

GetLanguageFromIsoCode639_3 returns the language for the given ISO 639-3 code enum value.

func (Language) IsoCode639_1 ¶

func (language Language) IsoCode639_1() IsoCode639_1

IsoCode639_1 returns a language's ISO 639-1 code.

func (Language) IsoCode639_3 ¶

func (language Language) IsoCode639_3() IsoCode639_3

IsoCode639_3 returns a language's ISO 639-3 code.

func (Language) String ¶

func (language Language) String() string

type LanguageDetector ¶

type LanguageDetector interface {
	// DetectLanguageOf detects the language of the given text.
	// The boolean return value indicates whether a language can be reliably
	// detected. If this is not possible, (Unknown, false) is returned.
	DetectLanguageOf(text string) (Language, bool)

	// DetectMultipleLanguagesOf attempts to detect multiple languages in
	// mixed-language text. This feature is experimental and under continuous
	// development.
	//
	// A slice of DetectionResult is returned containing an entry for each
	// contiguous single-language text section as identified by the library.
	// Each entry consists of the identified language, a start index and an
	// end index. The indices denote the substring that has been identified
	// as a contiguous single-language text section.
	DetectMultipleLanguagesOf(text string) []DetectionResult

	// ComputeLanguageConfidenceValues computes confidence values for each
	// language supported by this detector for the given input text. These
	// values denote how likely it is that the given text has been written
	// in any of the languages supported by this detector.
	//
	// A slice of ConfidenceValue is returned containing those languages which
	// the calling instance of LanguageDetector has been built from. The entries
	// are sorted by their confidence value in descending order. Each value is
	// a probability between 0.0 and 1.0. The probabilities of all languages
	// will sum to 1.0. If the language is unambiguously identified by the rule
	// engine, the value 1.0 will always be returned for this language. The
	// other languages will receive a value of 0.0.
	ComputeLanguageConfidenceValues(text string) []ConfidenceValue

	// ComputeLanguageConfidence computes the confidence value for the given
	// language and input text. This value denotes how likely it is that the
	// given text has been written in the given language.
	//
	// The value that this method computes is a number between 0.0 and 1.0.
	// If the language is unambiguously identified by the rule engine, the value
	// 1.0 will always be returned. If the given language is not supported by
	// this detector instance, the value 0.0 will always be returned.
	ComputeLanguageConfidence(text string, language Language) float64
}

LanguageDetector is the interface describing the available methods for detecting the language of some textual input.

type LanguageDetectorBuilder ¶

type LanguageDetectorBuilder interface {
	// WithMinimumRelativeDistance sets the desired value for the minimum
	// relative distance measure.
	//
	// By default, Lingua returns the most likely language for a given
	// input text. However, there are certain words that are spelled the
	// same in more than one language. The word `prologue`, for instance,
	// is both a valid English and French word. Lingua would output either
	// English or French which might be wrong in the given context.
	// For cases like that, it is possible to specify a minimum relative
	// distance that the logarithmized and summed up probabilities for
	// each possible language have to satisfy.
	//
	// Be aware that the distance between the language probabilities is
	// dependent on the length of the input text. The longer the input
	// text, the larger the distance between the languages. So if you
	// want to classify very short text phrases, do not set the minimum
	// relative distance too high. Otherwise, you will get most results
	// returned as Unknown which is the return value for cases
	// where language detection is not reliably possible.
	//
	// Panics if distance is smaller than 0.0 or greater than 0.99.
	WithMinimumRelativeDistance(distance float64) LanguageDetectorBuilder

	// WithPreloadedLanguageModels configures LanguageDetectorBuilder to
	// preload all language models when creating the instance of LanguageDetector.
	//
	// By default, Lingua uses lazy-loading to load only those language
	// models on demand which are considered relevant by the rule-based
	// filter engine. For web services, for instance, it is rather
	// beneficial to preload all language models into memory to avoid
	// unexpected latency while waiting for the service response. This
	// method allows to switch between these two loading modes.
	WithPreloadedLanguageModels() LanguageDetectorBuilder

	// WithLowAccuracyMode disables the high accuracy mode in order to save
	// memory and increase performance.
	//
	// By default, Lingua's high detection accuracy comes at the cost of
	// loading large language models into memory which might not be feasible
	// for systems running low on resources.
	//
	// This method disables the high accuracy mode so that only a small subset
	// of language models is loaded into memory. The downside of this approach
	// is that detection accuracy for short texts consisting of less than 120
	// characters will drop significantly. However, detection accuracy for texts
	// which are longer than 120 characters will remain mostly unaffected.
	WithLowAccuracyMode() LanguageDetectorBuilder

	// Build creates and returns the configured instance of LanguageDetector.
	Build() LanguageDetector
	// contains filtered or unexported methods
}

LanguageDetectorBuilder is the interface that defines any other settings to use for building an instance of LanguageDetector, except for the languages to use.

type UnconfiguredLanguageDetectorBuilder ¶

type UnconfiguredLanguageDetectorBuilder interface {
	// FromAllLanguages configures the LanguageDetectorBuilder
	// to use all built-in languages.
	FromAllLanguages() LanguageDetectorBuilder

	// FromAllSpokenLanguages configures the LanguageDetectorBuilder
	// to use all built-in spoken languages.
	FromAllSpokenLanguages() LanguageDetectorBuilder

	// FromAllLanguagesWithArabicScript configures the LanguageDetectorBuilder
	// to use all built-in languages supporting the Arabic script.
	FromAllLanguagesWithArabicScript() LanguageDetectorBuilder

	// FromAllLanguagesWithCyrillicScript configures the LanguageDetectorBuilder
	// to use all built-in languages supporting the Cyrillic script.
	FromAllLanguagesWithCyrillicScript() LanguageDetectorBuilder

	// FromAllLanguagesWithDevanagariScript configures the LanguageDetectorBuilder
	// to use all built-in languages supporting the Devanagari script.
	FromAllLanguagesWithDevanagariScript() LanguageDetectorBuilder

	// FromAllLanguagesWithLatinScript configures the LanguageDetectorBuilder
	// to use all built-in languages supporting the Latin script.
	FromAllLanguagesWithLatinScript() LanguageDetectorBuilder

	// FromAllLanguagesWithout configures the LanguageDetectorBuilder
	// to use all built-in languages except for those specified as arguments
	// passed to this method. Panics if less than two languages are used to
	// build the LanguageDetector.
	FromAllLanguagesWithout(languages ...Language) LanguageDetectorBuilder

	// FromLanguages configures the LanguageDetectorBuilder to use the languages
	// specified as arguments passed to this method. Panics if less than two
	// languages are specified.
	FromLanguages(languages ...Language) LanguageDetectorBuilder

	// FromIsoCodes639_1 configures the LanguageDetectorBuilder to use those
	// languages whose ISO 639-1 codes are specified as arguments passed to
	// this method. Panics if less than two iso codes are specified.
	FromIsoCodes639_1(isoCodes ...IsoCode639_1) LanguageDetectorBuilder

	// FromIsoCodes639_3 configures the LanguageDetectorBuilder to use those
	// languages whose ISO 639-3 codes are specified as arguments passed to
	// this method. Panics if less than two iso codes are specified.
	FromIsoCodes639_3(isoCodes ...IsoCode639_3) LanguageDetectorBuilder
}

UnconfiguredLanguageDetectorBuilder is the interface describing the methods for specifying which languages will be used to build an instance of LanguageDetector. All methods return an implementation of the interface LanguageDetectorBuilder.

func NewLanguageDetectorBuilder ¶

func NewLanguageDetectorBuilder() UnconfiguredLanguageDetectorBuilder

NewLanguageDetectorBuilder returns a new instance that implements the interface UnconfiguredLanguageDetectorBuilder.

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
cmd module
serialization

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL