mapping

package
v2.4.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 29, 2024 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	IndexDynamic     = true
	StoreDynamic     = true
	DocValuesDynamic = true // TODO revisit default?
)

control the default behavior for dynamic fields (those not explicitly mapped)

View Source 
var MappingJSONStrict = false

Functions

func SetLog 

func SetLog(l *log.Logger)

SetLog sets the logger used for logging by default log messages are sent to io.Discard

Types

type Classifier 

type Classifier interface {
	Type() string
}

A Classifier is an interface describing any object which knows how to identify its own type. Alternatively, if a struct already has a Type field or method in conflict, one can use BleveType instead.

type DocumentMapping 

type DocumentMapping struct {
	Enabled         bool                        `json:"enabled"`
	Dynamic         bool                        `json:"dynamic"`
	Properties      map[string]*DocumentMapping `json:"properties,omitempty"`
	Fields          []*FieldMapping             `json:"fields,omitempty"`
	DefaultAnalyzer string                      `json:"default_analyzer,omitempty"`

	// StructTagKey overrides "json" when looking for field names in struct tags
	StructTagKey string `json:"struct_tag_key,omitempty"`
}

A DocumentMapping describes how a type of document should be indexed. As documents can be hierarchical, named sub-sections of documents are mapped using the same structure in the Properties field. Each value inside a document can be indexed 0 or more ways. These index entries are called fields and are stored in the Fields field. Entire sections of a document can be ignored or excluded by setting Enabled to false. If not explicitly mapped, default mapping operations are used. To disable this automatic handling, set Dynamic to false.

func NewDocumentDisabledMapping 

func NewDocumentDisabledMapping() *DocumentMapping

NewDocumentDisabledMapping returns a new document mapping that will not perform any indexing.

func NewDocumentMapping 

func NewDocumentMapping() *DocumentMapping

NewDocumentMapping returns a new document mapping with all the default values.

func NewDocumentStaticMapping 

func NewDocumentStaticMapping() *DocumentMapping

NewDocumentStaticMapping returns a new document mapping that will not automatically index parts of a document without an explicit mapping.

func (*DocumentMapping) AddFieldMapping 

func (dm *DocumentMapping) AddFieldMapping(fm *FieldMapping)

AddFieldMapping adds the provided FieldMapping for this section of the document.

Example 
// you can only add field mapping to those properties which already have a document mapping
documentMapping := NewDocumentMapping()
subDocumentMapping := NewDocumentMapping()
documentMapping.AddSubDocumentMapping("Property", subDocumentMapping)

fieldMapping := NewTextFieldMapping()
fieldMapping.Analyzer = "en"
subDocumentMapping.AddFieldMapping(fieldMapping)

fmt.Println(len(documentMapping.Properties["Property"].Fields))

Output:

1

func (*DocumentMapping) AddFieldMappingsAt 

func (dm *DocumentMapping) AddFieldMappingsAt(property string, fms ...*FieldMapping)

AddFieldMappingsAt adds one or more FieldMappings at the named sub-document. If the named sub-document doesn't yet exist it is created for you. This is a convenience function to make most common mappings more concise. Otherwise, you would: 

subMapping := NewDocumentMapping()
subMapping.AddFieldMapping(fieldMapping)
parentMapping.AddSubDocumentMapping(property, subMapping)
Example 
// you can only add field mapping to those properties which already have a document mapping
documentMapping := NewDocumentMapping()
subDocumentMapping := NewDocumentMapping()
documentMapping.AddSubDocumentMapping("NestedProperty", subDocumentMapping)

fieldMapping := NewTextFieldMapping()
fieldMapping.Analyzer = "en"
documentMapping.AddFieldMappingsAt("NestedProperty", fieldMapping)

fmt.Println(len(documentMapping.Properties["NestedProperty"].Fields))

Output:

1

func (*DocumentMapping) AddSubDocumentMapping 

func (dm *DocumentMapping) AddSubDocumentMapping(property string, sdm *DocumentMapping)

AddSubDocumentMapping adds the provided DocumentMapping as a sub-mapping for the specified named subsection.

Example 
// adds a document mapping for a property in a document
// useful for mapping nested documents
documentMapping := NewDocumentMapping()
subDocumentMapping := NewDocumentMapping()
documentMapping.AddSubDocumentMapping("Property", subDocumentMapping)

fmt.Println(len(documentMapping.Properties))

Output:

1

func (*DocumentMapping) UnmarshalJSON 

func (dm *DocumentMapping) UnmarshalJSON(data []byte) error

UnmarshalJSON offers custom unmarshaling with optional strict validation

func (*DocumentMapping) Validate 

func (dm *DocumentMapping) Validate(
	cache *registry.Cache,
	parentName string, fieldAliasCtx map[string]*FieldMapping,
) error

type FieldMapping 

type FieldMapping struct {
	Name string `json:"name,omitempty"`
	Type string `json:"type,omitempty"`

	// Analyzer specifies the name of the analyzer to use for this field.  If
	// Analyzer is empty, traverse the DocumentMapping tree toward the root and
	// pick the first non-empty DefaultAnalyzer found. If there is none, use
	// the IndexMapping.DefaultAnalyzer.
	Analyzer string `json:"analyzer,omitempty"`

	// Store indicates whether to store field values in the index. Stored
	// values can be retrieved from search results using SearchRequest.Fields.
	Store bool `json:"store,omitempty"`
	Index bool `json:"index,omitempty"`

	// IncludeTermVectors, if true, makes terms occurrences to be recorded for
	// this field. It includes the term position within the terms sequence and
	// the term offsets in the source document field. Term vectors are required
	// to perform phrase queries or terms highlighting in source documents.
	IncludeTermVectors bool   `json:"include_term_vectors,omitempty"`
	IncludeInAll       bool   `json:"include_in_all,omitempty"`
	DateFormat         string `json:"date_format,omitempty"`

	// DocValues, if true makes the index uninverting possible for this field
	// It is useful for faceting and sorting queries.
	DocValues bool `json:"docvalues,omitempty"`

	// SkipFreqNorm, if true, avoids the indexing of frequency and norm values
	// of the tokens for this field. This option would be useful for saving
	// the processing of freq/norm details when the default score based relevancy
	// isn't needed.
	SkipFreqNorm bool `json:"skip_freq_norm,omitempty"`

	// Dimensionality of the vector
	Dims int `json:"dims,omitempty"`

	// Similarity is the similarity algorithm used for scoring
	// vector fields.
	// See: index.DefaultSimilarityMetric & index.SupportedSimilarityMetrics
	Similarity string `json:"similarity,omitempty"`

	// Applicable to vector fields only - optimization string
	VectorIndexOptimizedFor string `json:"vector_index_optimized_for,omitempty"`
}

A FieldMapping describes how a specific item should be put into the index.

func NewBooleanFieldMapping 

func NewBooleanFieldMapping() *FieldMapping

NewBooleanFieldMapping returns a default field mapping for booleans

func NewDateTimeFieldMapping 

func NewDateTimeFieldMapping() *FieldMapping

NewDateTimeFieldMapping returns a default field mapping for dates

func NewGeoPointFieldMapping 

func NewGeoPointFieldMapping() *FieldMapping

NewGeoPointFieldMapping returns a default field mapping for geo points

func NewGeoShapeFieldMapping 

func NewGeoShapeFieldMapping() *FieldMapping

NewGeoShapeFieldMapping returns a default field mapping for geoshapes

func NewIPFieldMapping 

func NewIPFieldMapping() *FieldMapping

NewIPFieldMapping returns a default field mapping for IP points

func NewKeywordFieldMapping 

func NewKeywordFieldMapping() *FieldMapping

NewKeyworFieldMapping returns a default field mapping for text with analyzer "keyword".

func NewNumericFieldMapping 

func NewNumericFieldMapping() *FieldMapping

NewNumericFieldMapping returns a default field mapping for numbers

func NewTextFieldMapping 

func NewTextFieldMapping() *FieldMapping

NewTextFieldMapping returns a default field mapping for text

func NewVectorFieldMapping 

func NewVectorFieldMapping() *FieldMapping

func (*FieldMapping) Options 

func (fm *FieldMapping) Options() index.FieldIndexingOptions

Options returns the indexing options for this field.

func (*FieldMapping) UnmarshalJSON 

func (fm *FieldMapping) UnmarshalJSON(data []byte) error

UnmarshalJSON offers custom unmarshaling with optional strict validation

type IndexMapping 

type IndexMapping interface {
	MapDocument(doc *document.Document, data interface{}) error
	Validate() error

	DateTimeParserNamed(name string) analysis.DateTimeParser

	DefaultSearchField() string

	AnalyzerNameForPath(path string) string
	AnalyzerNamed(name string) analysis.Analyzer

	FieldMappingForPath(path string) FieldMapping
}

type IndexMappingImpl 

type IndexMappingImpl struct {
	TypeMapping           map[string]*DocumentMapping `json:"types,omitempty"`
	DefaultMapping        *DocumentMapping            `json:"default_mapping"`
	TypeField             string                      `json:"type_field"`
	DefaultType           string                      `json:"default_type"`
	DefaultAnalyzer       string                      `json:"default_analyzer"`
	DefaultDateTimeParser string                      `json:"default_datetime_parser"`
	DefaultField          string                      `json:"default_field"`
	StoreDynamic          bool                        `json:"store_dynamic"`
	IndexDynamic          bool                        `json:"index_dynamic"`
	DocValuesDynamic      bool                        `json:"docvalues_dynamic"`
	CustomAnalysis        *customAnalysis             `json:"analysis,omitempty"`
	// contains filtered or unexported fields
}

An IndexMappingImpl controls how objects are placed into an index. First the type of the object is determined. Once the type is know, the appropriate DocumentMapping is selected by the type. If no mapping was determined for that type, a DefaultMapping will be used.

func NewIndexMapping 

func NewIndexMapping() *IndexMappingImpl

NewIndexMapping creates a new IndexMapping that will use all the default indexing rules

func (*IndexMappingImpl) AddCustomAnalyzer 

func (im *IndexMappingImpl) AddCustomAnalyzer(name string, config map[string]interface{}) error

AddCustomAnalyzer defines a custom analyzer for use in this mapping. The config map must have a "type" string entry to resolve the analyzer constructor. The constructor is invoked with the remaining entries and returned analyzer is registered in the IndexMapping.

bleve comes with predefined analyzers, like github.com/blevesearch/bleve/analysis/analyzer/custom. They are available only if their package is imported by client code. To achieve this, use their metadata to fill configuration entries: 

import (
    "github.com/MuratYMT2/bleve/v2/analysis/analyzer/custom"
    "github.com/MuratYMT2/bleve/v2/analysis/char/html"
    "github.com/MuratYMT2/bleve/v2/analysis/token/lowercase"
    "github.com/MuratYMT2/bleve/v2/analysis/tokenizer/unicode"
)

m := bleve.NewIndexMapping()
err := m.AddCustomAnalyzer("html", map[string]interface{}{
    "type": custom.Name,
    "char_filters": []string{
        html.Name,
    },
    "tokenizer":     unicode.Name,
    "token_filters": []string{
        lowercase.Name,
        ...
    },
})

func (*IndexMappingImpl) AddCustomCharFilter 

func (im *IndexMappingImpl) AddCustomCharFilter(name string, config map[string]interface{}) error

AddCustomCharFilter defines a custom char filter for use in this mapping

func (*IndexMappingImpl) AddCustomDateTimeParser 

func (im *IndexMappingImpl) AddCustomDateTimeParser(name string, config map[string]interface{}) error

AddCustomDateTimeParser defines a custom date time parser for use in this mapping

func (*IndexMappingImpl) AddCustomTokenFilter 

func (im *IndexMappingImpl) AddCustomTokenFilter(name string, config map[string]interface{}) error

AddCustomTokenFilter defines a custom token filter for use in this mapping

func (*IndexMappingImpl) AddCustomTokenMap 

func (im *IndexMappingImpl) AddCustomTokenMap(name string, config map[string]interface{}) error

AddCustomTokenMap defines a custom token map for use in this mapping

func (*IndexMappingImpl) AddCustomTokenizer 

func (im *IndexMappingImpl) AddCustomTokenizer(name string, config map[string]interface{}) error

AddCustomTokenizer defines a custom tokenizer for use in this mapping

func (*IndexMappingImpl) AddDocumentMapping 

func (im *IndexMappingImpl) AddDocumentMapping(doctype string, dm *DocumentMapping)

AddDocumentMapping sets a custom document mapping for the specified type

func (*IndexMappingImpl) AnalyzeText 

func (im *IndexMappingImpl) AnalyzeText(analyzerName string, text []byte) (analysis.TokenStream, error)

func (*IndexMappingImpl) AnalyzerNameForPath 

func (im *IndexMappingImpl) AnalyzerNameForPath(path string) string

AnalyzerNameForPath attempts to find the best analyzer to use with only a field name will walk all the document types, look for field mappings at the provided path, if one exists and it has an explicit analyzer that is returned.

func (*IndexMappingImpl) AnalyzerNamed 

func (im *IndexMappingImpl) AnalyzerNamed(name string) analysis.Analyzer

func (*IndexMappingImpl) DateTimeParserNamed 

func (im *IndexMappingImpl) DateTimeParserNamed(name string) analysis.DateTimeParser

func (*IndexMappingImpl) DefaultSearchField 

func (im *IndexMappingImpl) DefaultSearchField() string

func (*IndexMappingImpl) FieldAnalyzer 

func (im *IndexMappingImpl) FieldAnalyzer(field string) string

FieldAnalyzer returns the name of the analyzer used on a field.

func (*IndexMappingImpl) FieldMappingForPath 

func (im *IndexMappingImpl) FieldMappingForPath(path string) FieldMapping

FieldMappingForPath returns the mapping for a specific field 'path'.

func (*IndexMappingImpl) MapDocument 

func (im *IndexMappingImpl) MapDocument(doc *document.Document, data interface{}) error

func (*IndexMappingImpl) UnmarshalJSON 

func (im *IndexMappingImpl) UnmarshalJSON(data []byte) error

UnmarshalJSON offers custom unmarshaling with optional strict validation

func (*IndexMappingImpl) Validate 

func (im *IndexMappingImpl) Validate() error

Validate will walk the entire structure ensuring the following explicitly named and default analyzers can be built

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.