find

Documentation

Overview

Functions for finding documents by full text search

Functions for parsing a search query

Index

• Variables
• func LoadDocInfo(r io.Reader) (map[string]DocInfo, map[string]DocInfo)
• type Collection
• type DictQueryParser
  • func (parser DictQueryParser) ParseQuery(query string) []TextSegment
• type DocFinder
  • func NewDocFinder(ctx context.Context, database *sql.DB, docMap map[string]DocInfo) DocFinder
• type DocInfo
• type DocTitleFinder
  • func NewDocTitleFinder(infoCache map[string]DocInfo) DocTitleFinder
• type Document
  • func (doc Document) String() string
• type QueryParser
  • func MakeQueryParser(dict map[string]*dicttypes.Word) QueryParser
• type QueryResults
• type TextSegment

Variables

var WEIGHT = []float64{0.080, 2.327, 3.040} // [BM25 words, BM25 bigrams, bit vector]

From logistic regression

Functions

func LoadDocInfo

func LoadDocInfo(r io.Reader) (map[string]DocInfo, map[string]DocInfo)

Load title info for all documents

Types

type Collection

type Collection struct {
	GlossFile, Title string
}

type DictQueryParser

type DictQueryParser struct{ Tokenizer tokenizer.DictTokenizer }

func (DictQueryParser) ParseQuery

func (parser DictQueryParser) ParseQuery(query string) []TextSegment

The method for parsing the query text in this function is based on dictionary lookups

type DocFinder

type DocFinder interface {
	FindDocuments(ctx context.Context, dictSearcher dictionary.ReverseIndex,
		parser QueryParser, query string, advanced bool) (*QueryResults, error)
	FindDocumentsInCol(ctx context.Context, dictSearcher dictionary.ReverseIndex,
		parser QueryParser, query, col_gloss_file string) (*QueryResults, error)
	GetColMap() map[string]string
	Inititialized() bool
}

DocFinder finds documents.

func NewDocFinder

func NewDocFinder(ctx context.Context,
	database *sql.DB,
	docMap map[string]DocInfo) DocFinder

Create and initialize an implementation of the DocFinder interface

type DocInfo

type DocInfo struct {
	CorpusFile, GlossFile, Title, TitleCN, TitleEN, CollectionFile, CollectionTitle string
}

type DocTitleFinder

type DocTitleFinder interface {
	FindDocuments(ctx context.Context, query string) (*QueryResults, error)
}

DocTitleFinder finds documents by title.

func NewDocTitleFinder

func NewDocTitleFinder(infoCache map[string]DocInfo) DocTitleFinder

NewDocTitleFinder initializes a DocTitleFinder implementation Params

infoCache: key to the map is the Chinese part of the title

type Document

type Document struct {
	GlossFile, Title, CollectionFile, CollectionTitle, ContainsWords string
	ContainsBigrams                                                  string
	SimTitle, SimWords, SimBigram, SimBitVector, Similarity          float64
	ContainsTerms                                                    []string
	MatchDetails                                                     fulltext.MatchingText
	TitleCNMatch                                                     bool
}

func (Document) String

func (doc Document) String() string

For printing out retrieved document metadata

type QueryParser

type QueryParser interface {
	ParseQuery(query string) []TextSegment
}

Parses input queries into a slice of text segments

func MakeQueryParser

func MakeQueryParser(dict map[string]*dicttypes.Word) QueryParser

Creates a QueryParser

type QueryResults

type QueryResults struct {
	Query, CollectionFile        string
	NumCollections, NumDocuments int
	Collections                  []Collection
	Documents                    []Document
	Terms                        []TextSegment
	SimilarTerms                 []TextSegment
}

type TextSegment

type TextSegment struct {
	QueryText string
	DictEntry dicttypes.Word
	Senses    []dicttypes.WordSense
}

A text segment contains the QueryText searched for and possibly a matching dictionary entry. There will only be matching dictionary entries for Chinese words in the dictionary. Non-Chinese text, punctuation, and unknown Chinese words will have nil DictEntry values and matching values will be included in the Senses field.