tantivy_go

README

Go Tantivy Bindings

This project provides Go bindings for the Tantivy search engine library. Tantivy is a full-text search engine library written in Rust, and this project aims to make its powerful search capabilities available to Go developers.

Disclaimer

This project is still in development and might not be ready for production use. The API is subject to change, and the project may contain bugs. Please use with caution.

Installation

go get github.com/anyproto/tantivy-go

Ensure your libraries are in your ld path.

Example Run

• Run make download-tantivy-all inside the rust folder
• Run main.go in the example folder

Development

Development and compilation are done on MacBooks and for Apple platforms. Therefore, the development steps provided are for macOS.

Install environment

• Install rustup
• Install Rust architectures: make setup
• Add Android libraries to your path: export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/emulator:$ANDROID_HOME/platform-tools:$ANDROID_HOME/ndk/25.2.9519653/toolchains/llvm/prebuilt/darwin-x86_64/bin
• Install Windows compiler: brew install mingw-w64
• Install musl: brew tap messense/macos-cross-toolchains && brew install x86_64-unknown-linux-musl

Install rust libraries

Run inside the rust folder:

make install-all - install release versions for all platforms

make install-debug-all - install debug versions for all platforms

make install-ARCH-GOOS - install release version for ARCH GOOS

make install-debug-ARCH-GOOS - install debug version for ARCH GOOS

GCC support

To be done

Validate min macos version

otool -l libtantivy_go.a | rg LC_BUILD_VERSION -A4 | rg minos | sort | uniq -c Expected output:

 880     minos 11.0

Documentation

Index

• Constants
• func GetSearchResults[T any](searchResult *SearchResult, schema *Schema, f func(json string) (T, error), ...) ([]T, error)
• func LibInit(directive ...string) error
• func ToModel[T any](doc *Document, schema *Schema, includeFields []string, ...) (T, error)
• type Document
  • func NewDocument() *Document
  • func (d *Document) AddField(fieldName, fieldValue string, index *Index) error
  • func (d *Document) Free()
  • func (d *Document) ToJson(schema *Schema, includeFields ...string) (string, error)
• type Index
  • func NewIndexWithSchema(path string, schema *Schema) (*Index, error)
  • func (i *Index) AddAndConsumeDocuments(docs ...*Document) error
  • func (i *Index) DeleteDocuments(field string, deleteIds ...string) error
  • func (i *Index) Free()
  • func (i *Index) NumDocs() (uint64, error)
  • func (i *Index) RegisterTextAnalyzerEdgeNgram(tokenizerName string, minGram, maxGram uintptr, limit uintptr) error
  • func (i *Index) RegisterTextAnalyzerNgram(tokenizerName string, minGram, maxGram uintptr, prefixOnly bool) error
  • func (i *Index) RegisterTextAnalyzerRaw(tokenizerName string) error
  • func (i *Index) RegisterTextAnalyzerSimple(tokenizerName string, textLimit uintptr, lang string) error
  • func (i *Index) Search(query string, docsLimit uintptr, withHighlights bool, fieldNames ...string) (*SearchResult, error)
• type Schema
• type SchemaBuilder
  • func NewSchemaBuilder() (*SchemaBuilder, error)
  • func (b *SchemaBuilder) AddTextField(name string, stored bool, isText bool, isFast bool, indexRecordOption int, ...) error
  • func (b *SchemaBuilder) BuildSchema() (*Schema, error)
• type SearchResult
  • func (r *SearchResult) Free()
  • func (r *SearchResult) Get(index uint64) (*Document, error)
  • func (r *SearchResult) GetSize() (uint64, error)

Constants

const (
	// IndexRecordOptionBasic specifies that only basic indexing information should be used.
	IndexRecordOptionBasic = iota
	// IndexRecordOptionWithFreqs specifies that indexing should include term frequencies.
	IndexRecordOptionWithFreqs
	// IndexRecordOptionWithFreqsAndPositions specifies that indexing should include term frequencies and term positions.
	IndexRecordOptionWithFreqsAndPositions
)

const (
	Arabic     = "ar"
	Danish     = "da"
	Dutch      = "nl"
	English    = "en"
	Finnish    = "fi"
	French     = "fr"
	German     = "de"
	Greek      = "el"
	Hungarian  = "hu"
	Italian    = "it"
	Norwegian  = "no"
	Portuguese = "pt"
	Romanian   = "ro"
	Russian    = "ru"
	Spanish    = "es"
	Swedish    = "sv"
	Tamil      = "ta"
	Turkish    = "tr"
)

const DefaultTokenizer = "default"

const TokenizerEdgeNgram = "edge_ngram"

const TokenizerNgram = "ngram"

const TokenizerRaw = "raw"

const TokenizerSimple = "simple"

Functions

func GetSearchResults

func GetSearchResults[T any](
	searchResult *SearchResult,
	schema *Schema,
	f func(json string) (T, error),
	includeFields ...string,
) ([]T, error)

GetSearchResults extracts search results from a SearchResult and converts them into a slice of models.

Parameters:

• searchResult (*SearchResult): The search results to process.
• schema (*Schema): The schema to use for converting documents to models.
• f (func(json string) (T, error)): A function to convert JSON strings to models.
• includeFields (...string): Optional list of fields to include in the result.

Returns:

• ([]T, error): A slice of models obtained from the search results, and an error if something goes wrong.

func LibInit

func LibInit(directive ...string) error

LibInit initializes the library with an optional directive.

Parameters:

• directive: A variadic parameter that allows specifying an initialization directive. If no directive is provided, the default value "info" is used.

Returns: - An error if the initialization fails.

func ToModel

func ToModel[T any](doc *Document, schema *Schema, includeFields []string, f func(json string) (T, error)) (T, error)

ToModel converts a document to a model of type T using the provided schema and a conversion function.

Parameters:

• doc: the document to convert
• schema: the schema to use for converting the document to JSON
• includeFields: optional fields to include in the JSON output
• f: a function that takes a JSON string and converts it to a model of type T

Returns:

• T: the model of type T resulting from the conversion
• error: an error if the conversion fails, or nil if the operation is successful

Types

type Document

type Document struct {
	// contains filtered or unexported fields
}

func NewDocument

func NewDocument() *Document

NewDocument creates a new instance of Document.

Returns:

• *Document: a pointer to a newly created Document instance.

func (*Document) AddField

func (d *Document) AddField(fieldName, fieldValue string, index *Index) error

AddField adds a field with the specified name and value to the document using the given index. Returns an error if adding the field fails.

Parameters:

• fieldName: the name of the field to add
• fieldValue: the value of the field to add
• index: the index to use for adding the field

Returns:

• error: an error if adding the field fails, or nil if the operation is successful

func (*Document) Free

func (d *Document) Free()

func (*Document) ToJson

func (d *Document) ToJson(schema *Schema, includeFields ...string) (string, error)

ToJson converts the document to its JSON representation based on the provided schema. Optionally, specific fields can be included in the JSON output.

Parameters:

• schema: the schema to use for converting the document to JSON
• includeFields: optional variadic parameter specifying the fields to include in the JSON output

Returns:

• string: the JSON representation of the document
• error: an error if the conversion fails, or nil if the operation is successful

type Index

type Index struct {
	// contains filtered or unexported fields
}

func NewIndexWithSchema

func NewIndexWithSchema(path string, schema *Schema) (*Index, error)

NewIndexWithSchema creates a new instance of Index with the provided schema.

Parameters:

• path: The path to the index as a string.
• schema: A pointer to the Schema to be used.

Returns:

• *Index: A pointer to a newly created Index instance.
• error: An error if the index creation fails.

func (*Index) AddAndConsumeDocuments

func (i *Index) AddAndConsumeDocuments(docs ...*Document) error

AddAndConsumeDocuments adds and consumes the provided documents to the index.

Parameters:

• docs: A variadic parameter of pointers to Document to be added and consumed.

Returns:

• error: An error if adding and consuming the documents fails.

func (*Index) DeleteDocuments

func (i *Index) DeleteDocuments(field string, deleteIds ...string) error

DeleteDocuments deletes documents from the index based on the specified field and IDs.

Parameters:

• field: The field name to match against the document IDs.
• deleteIds: A variadic parameter of document IDs to be deleted.

Returns:

• error: An error if deleting the documents fails.

func (*Index) Free

func (i *Index) Free()

func (*Index) NumDocs

func (i *Index) NumDocs() (uint64, error)

NumDocs returns the number of documents in the index.

Returns:

• uint64: The number of documents.
• error: An error if retrieving the document count fails.

func (*Index) RegisterTextAnalyzerEdgeNgram

func (i *Index) RegisterTextAnalyzerEdgeNgram(tokenizerName string, minGram, maxGram uintptr, limit uintptr) error

RegisterTextAnalyzerEdgeNgram registers a text analyzer using edge n-grams with the index.

Parameters:

• tokenizerName (string): The name of the tokenizer to be used.
• minGram (uintptr): The minimum length of the edge n-grams.
• maxGram (uintptr): The maximum length of the edge n-grams.
• limit (uintptr): The maximum number of edge n-grams to generate.

Returns:

• error: An error if the registration fails.

func (*Index) RegisterTextAnalyzerNgram

func (i *Index) RegisterTextAnalyzerNgram(tokenizerName string, minGram, maxGram uintptr, prefixOnly bool) error

RegisterTextAnalyzerNgram registers a text analyzer using N-grams with the index.

Parameters:

• tokenizerName (string): The name of the tokenizer to be used.
• minGram (uintptr): The minimum length of the n-grams.
• maxGram (uintptr): The maximum length of the n-grams.
• prefixOnly (bool): Whether to generate only prefix n-grams.

Returns:

• error: An error if the registration fails.

func (*Index) RegisterTextAnalyzerRaw

func (i *Index) RegisterTextAnalyzerRaw(tokenizerName string) error

RegisterTextAnalyzerRaw registers a raw text analyzer with the index.

Parameters:

• tokenizerName (string): The name of the raw tokenizer to be used.

Returns:

• error: An error if the registration fails.

func (*Index) RegisterTextAnalyzerSimple

func (i *Index) RegisterTextAnalyzerSimple(tokenizerName string, textLimit uintptr, lang string) error

RegisterTextAnalyzerSimple registers a simple text analyzer with the index.

Parameters:

• tokenizerName (string): The name of the tokenizer to be used.
• textLimit (uintptr): The limit on the length of the text to be analyzed.
• lang (string): The language code for the text analyzer.

Returns:

• error: An error if the registration fails.

func (*Index) Search

func (i *Index) Search(query string, docsLimit uintptr, withHighlights bool, fieldNames ...string) (*SearchResult, error)

Search performs a search query on the index and returns the search results.

Parameters:

• query (string): The search query string.
• docsLimit (uintptr): The maximum number of documents to return.
• withHighlights (bool): Whether to include highlights in the results.
• fieldNames (...string): The names of the fields to be included in the search.

Returns:

• *SearchResult: A pointer to the SearchResult containing the search results.
• error: An error if the search fails.

type Schema

type Schema struct {
	// contains filtered or unexported fields
}

type SchemaBuilder

type SchemaBuilder struct {
	// contains filtered or unexported fields
}

func NewSchemaBuilder

func NewSchemaBuilder() (*SchemaBuilder, error)

NewSchemaBuilder creates a new SchemaBuilder instance. Returns a pointer to the SchemaBuilder and an error if creation fails.

func (*SchemaBuilder) AddTextField

func (b *SchemaBuilder) AddTextField(
	name string,
	stored bool,
	isText bool,
	isFast bool,
	indexRecordOption int,
	tokenizer string,
) error

AddTextField adds a text field to the schema being built.

Parameters: - name: The name of the field. - stored: Whether the field should be stored in the index. - isText: Whether the field should be treated as tantivy text or string for full-text search. - isFast: Whether the field should be indexed as tantivy quick field. - indexRecordOption: The indexing option to be used (e.g., basic, with frequencies, with frequencies and positions). - tokenizer: The name of the tokenizer to be used for the field.

Returns an error if the field could not be added.

func (*SchemaBuilder) BuildSchema

func (b *SchemaBuilder) BuildSchema() (*Schema, error)

BuildSchema finalizes the schema building process and returns the resulting Schema. Returns a pointer to the Schema and an error if the schema could not be built.

type SearchResult

type SearchResult struct {
	// contains filtered or unexported fields
}

func (*SearchResult) Free

func (r *SearchResult) Free()

func (*SearchResult) Get

func (r *SearchResult) Get(index uint64) (*Document, error)

Get retrieves a document from the search result at the specified index.

Parameters: - index: The index of the document to retrieve.

Returns: - A pointer to the Document if successful, or nil if not found. - An error if there was an issue retrieving the document.

func (*SearchResult) GetSize

func (r *SearchResult) GetSize() (uint64, error)

GetSize returns the number of documents in the search result.

Returns: - The size of the search result if successful. - An error if there was an issue getting the size.