cld2

README

cld2 token filter

A bleve token filter which passes the text of each token and passes it to the cld2 library. The library determines what it thinks the language most likely is. The ISO-639 language code replaces the token term.

In normal usage, you use this with the "single" tokenizer, so there is only one input token. Further, you should precede it with the "to_lower" filter so that the input term is in all lower-case unicode characters.

Building

1. Acquire the source to cld2 in this directory.

   $ svn checkout http://cld2.googlecode.com/svn/trunk/ cld2-read-only
   

2. Build cld2

   $ cd cld2-read-only/internal/
   $ ./compile_libs.sh
   

3. Put the resulting libraries somewhere your dynamic linker can find.

   $ cp *.so /usr/local/lib
   

4. Run the unit tests

   $ cd ../..
   $ go test -v
   === RUN TestCld2Filter
   --- PASS: TestCld2Filter (0.00 seconds)
   PASS
   ok      github.com/couchbaselabs/bleve/analysis/token_filters/cld2      0.033s
   

Documentation

Overview

Copyright (c) 2014 Couchbase, Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• Constants
• func Cld2FilterConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenFilter, error)
• type Cld2Filter
  • func NewCld2Filter() *Cld2Filter
  • func (f *Cld2Filter) Filter(input analysis.TokenStream) analysis.TokenStream

Constants

const Name = "detect_lang"

Functions

func Cld2FilterConstructor

func Cld2FilterConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenFilter, error)

Types

type Cld2Filter

type Cld2Filter struct {
}

func NewCld2Filter

func NewCld2Filter() *Cld2Filter

func (*Cld2Filter) Filter

func (f *Cld2Filter) Filter(input analysis.TokenStream) analysis.TokenStream