mapping

package
v2.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 26, 2024 License: Apache-2.0 Imports: 18 Imported by: 113

Documentation

Index

Examples

Constants

This section is empty.

Variables

View Source
var (
	IndexDynamic     = true
	StoreDynamic     = true
	DocValuesDynamic = true // TODO revisit default?
)

control the default behavior for dynamic fields (those not explicitly mapped)

View Source
var MappingJSONStrict = false

Functions

func SetLog

func SetLog(l *log.Logger)

SetLog sets the logger used for logging by default log messages are sent to io.Discard

Types

type Classifier

type Classifier interface {
	Type() string
}

A Classifier is an interface describing any object which knows how to identify its own type. Alternatively, if a struct already has a Type field or method in conflict, one can use BleveType instead.

type DocumentMapping

type DocumentMapping struct {
	Enabled         bool                        `json:"enabled"`
	Dynamic         bool                        `json:"dynamic"`
	Properties      map[string]*DocumentMapping `json:"properties,omitempty"`
	Fields          []*FieldMapping             `json:"fields,omitempty"`
	DefaultAnalyzer string                      `json:"default_analyzer,omitempty"`

	// StructTagKey overrides "json" when looking for field names in struct tags
	StructTagKey string `json:"struct_tag_key,omitempty"`
}

A DocumentMapping describes how a type of document should be indexed. As documents can be hierarchical, named sub-sections of documents are mapped using the same structure in the Properties field. Each value inside a document can be indexed 0 or more ways. These index entries are called fields and are stored in the Fields field. Entire sections of a document can be ignored or excluded by setting Enabled to false. If not explicitly mapped, default mapping operations are used. To disable this automatic handling, set Dynamic to false.

func NewDocumentDisabledMapping

func NewDocumentDisabledMapping() *DocumentMapping

NewDocumentDisabledMapping returns a new document mapping that will not perform any indexing.

func NewDocumentMapping

func NewDocumentMapping() *DocumentMapping

NewDocumentMapping returns a new document mapping with all the default values.

func NewDocumentStaticMapping

func NewDocumentStaticMapping() *DocumentMapping

NewDocumentStaticMapping returns a new document mapping that will not automatically index parts of a document without an explicit mapping.

func (*DocumentMapping) AddFieldMapping

func (dm *DocumentMapping) AddFieldMapping(fm *FieldMapping)

AddFieldMapping adds the provided FieldMapping for this section of the document.

Example
// you can only add field mapping to those properties which already have a document mapping
documentMapping := NewDocumentMapping()
subDocumentMapping := NewDocumentMapping()
documentMapping.AddSubDocumentMapping("Property", subDocumentMapping)

fieldMapping := NewTextFieldMapping()
fieldMapping.Analyzer = "en"
subDocumentMapping.AddFieldMapping(fieldMapping)

fmt.Println(len(documentMapping.Properties["Property"].Fields))
Output:

1

func (*DocumentMapping) AddFieldMappingsAt

func (dm *DocumentMapping) AddFieldMappingsAt(property string, fms ...*FieldMapping)

AddFieldMappingsAt adds one or more FieldMappings at the named sub-document. If the named sub-document doesn't yet exist it is created for you. This is a convenience function to make most common mappings more concise. Otherwise, you would:

subMapping := NewDocumentMapping()
subMapping.AddFieldMapping(fieldMapping)
parentMapping.AddSubDocumentMapping(property, subMapping)
Example
// you can only add field mapping to those properties which already have a document mapping
documentMapping := NewDocumentMapping()
subDocumentMapping := NewDocumentMapping()
documentMapping.AddSubDocumentMapping("NestedProperty", subDocumentMapping)

fieldMapping := NewTextFieldMapping()
fieldMapping.Analyzer = "en"
documentMapping.AddFieldMappingsAt("NestedProperty", fieldMapping)

fmt.Println(len(documentMapping.Properties["NestedProperty"].Fields))
Output:

1

func (*DocumentMapping) AddSubDocumentMapping

func (dm *DocumentMapping) AddSubDocumentMapping(property string, sdm *DocumentMapping)

AddSubDocumentMapping adds the provided DocumentMapping as a sub-mapping for the specified named subsection.

Example
// adds a document mapping for a property in a document
// useful for mapping nested documents
documentMapping := NewDocumentMapping()
subDocumentMapping := NewDocumentMapping()
documentMapping.AddSubDocumentMapping("Property", subDocumentMapping)

fmt.Println(len(documentMapping.Properties))
Output:

1

func (*DocumentMapping) UnmarshalJSON

func (dm *DocumentMapping) UnmarshalJSON(data []byte) error

UnmarshalJSON offers custom unmarshaling with optional strict validation

func (*DocumentMapping) Validate

func (dm *DocumentMapping) Validate(cache *registry.Cache,
	parentName string, fieldAliasCtx map[string]*FieldMapping) error

type FieldMapping

type FieldMapping struct {
	Name string `json:"name,omitempty"`
	Type string `json:"type,omitempty"`

	// Analyzer specifies the name of the analyzer to use for this field.  If
	// Analyzer is empty, traverse the DocumentMapping tree toward the root and
	// pick the first non-empty DefaultAnalyzer found. If there is none, use
	// the IndexMapping.DefaultAnalyzer.
	Analyzer string `json:"analyzer,omitempty"`

	// Store indicates whether to store field values in the index. Stored
	// values can be retrieved from search results using SearchRequest.Fields.
	Store bool `json:"store,omitempty"`
	Index bool `json:"index,omitempty"`

	// IncludeTermVectors, if true, makes terms occurrences to be recorded for
	// this field. It includes the term position within the terms sequence and
	// the term offsets in the source document field. Term vectors are required
	// to perform phrase queries or terms highlighting in source documents.
	IncludeTermVectors bool   `json:"include_term_vectors,omitempty"`
	IncludeInAll       bool   `json:"include_in_all,omitempty"`
	DateFormat         string `json:"date_format,omitempty"`

	// DocValues, if true makes the index uninverting possible for this field
	// It is useful for faceting and sorting queries.
	DocValues bool `json:"docvalues,omitempty"`

	// SkipFreqNorm, if true, avoids the indexing of frequency and norm values
	// of the tokens for this field. This option would be useful for saving
	// the processing of freq/norm details when the default score based relevancy
	// isn't needed.
	SkipFreqNorm bool `json:"skip_freq_norm,omitempty"`

	// Dimensionality of the vector
	Dims int `json:"dims,omitempty"`

	// Similarity is the similarity algorithm used for scoring
	// vector fields.
	// See: index.DefaultSimilarityMetric & index.SupportedSimilarityMetrics
	Similarity string `json:"similarity,omitempty"`

	// Applicable to vector fields only - optimization string
	VectorIndexOptimizedFor string `json:"vector_index_optimized_for,omitempty"`
}

A FieldMapping describes how a specific item should be put into the index.

func NewBooleanFieldMapping

func NewBooleanFieldMapping() *FieldMapping

NewBooleanFieldMapping returns a default field mapping for booleans

func NewDateTimeFieldMapping

func NewDateTimeFieldMapping() *FieldMapping

NewDateTimeFieldMapping returns a default field mapping for dates

func NewGeoPointFieldMapping

func NewGeoPointFieldMapping() *FieldMapping

NewGeoPointFieldMapping returns a default field mapping for geo points

func NewGeoShapeFieldMapping added in v2.3.3

func NewGeoShapeFieldMapping() *FieldMapping

NewGeoShapeFieldMapping returns a default field mapping for geoshapes

func NewIPFieldMapping added in v2.3.0

func NewIPFieldMapping() *FieldMapping

NewIPFieldMapping returns a default field mapping for IP points

func NewKeywordFieldMapping added in v2.3.0

func NewKeywordFieldMapping() *FieldMapping

NewKeywordFieldMapping returns a default field mapping for text with analyzer "keyword".

func NewNumericFieldMapping

func NewNumericFieldMapping() *FieldMapping

NewNumericFieldMapping returns a default field mapping for numbers

func NewTextFieldMapping

func NewTextFieldMapping() *FieldMapping

NewTextFieldMapping returns a default field mapping for text

func NewVectorBase64FieldMapping added in v2.4.1

func NewVectorBase64FieldMapping() *FieldMapping

func NewVectorFieldMapping added in v2.4.0

func NewVectorFieldMapping() *FieldMapping

func (*FieldMapping) Options

func (fm *FieldMapping) Options() index.FieldIndexingOptions

Options returns the indexing options for this field.

func (*FieldMapping) UnmarshalJSON

func (fm *FieldMapping) UnmarshalJSON(data []byte) error

UnmarshalJSON offers custom unmarshaling with optional strict validation

type IndexMapping

type IndexMapping interface {
	MapDocument(doc *document.Document, data interface{}) error
	Validate() error

	DateTimeParserNamed(name string) analysis.DateTimeParser

	DefaultSearchField() string

	AnalyzerNameForPath(path string) string
	AnalyzerNamed(name string) analysis.Analyzer

	FieldMappingForPath(path string) FieldMapping
}

type IndexMappingImpl

type IndexMappingImpl struct {
	TypeMapping           map[string]*DocumentMapping `json:"types,omitempty"`
	DefaultMapping        *DocumentMapping            `json:"default_mapping"`
	TypeField             string                      `json:"type_field"`
	DefaultType           string                      `json:"default_type"`
	DefaultAnalyzer       string                      `json:"default_analyzer"`
	DefaultDateTimeParser string                      `json:"default_datetime_parser"`
	DefaultField          string                      `json:"default_field"`
	StoreDynamic          bool                        `json:"store_dynamic"`
	IndexDynamic          bool                        `json:"index_dynamic"`
	DocValuesDynamic      bool                        `json:"docvalues_dynamic"`
	CustomAnalysis        *customAnalysis             `json:"analysis,omitempty"`
	// contains filtered or unexported fields
}

An IndexMappingImpl controls how objects are placed into an index. First the type of the object is determined. Once the type is know, the appropriate DocumentMapping is selected by the type. If no mapping was determined for that type, a DefaultMapping will be used.

func NewIndexMapping

func NewIndexMapping() *IndexMappingImpl

NewIndexMapping creates a new IndexMapping that will use all the default indexing rules

func (*IndexMappingImpl) AddCustomAnalyzer

func (im *IndexMappingImpl) AddCustomAnalyzer(name string, config map[string]interface{}) error

AddCustomAnalyzer defines a custom analyzer for use in this mapping. The config map must have a "type" string entry to resolve the analyzer constructor. The constructor is invoked with the remaining entries and returned analyzer is registered in the IndexMapping.

bleve comes with predefined analyzers, like github.com/blevesearch/bleve/analysis/analyzer/custom. They are available only if their package is imported by client code. To achieve this, use their metadata to fill configuration entries:

import (
    "github.com/blevesearch/bleve/v2/analysis/analyzer/custom"
    "github.com/blevesearch/bleve/v2/analysis/char/html"
    "github.com/blevesearch/bleve/v2/analysis/token/lowercase"
    "github.com/blevesearch/bleve/v2/analysis/tokenizer/unicode"
)

m := bleve.NewIndexMapping()
err := m.AddCustomAnalyzer("html", map[string]interface{}{
    "type": custom.Name,
    "char_filters": []string{
        html.Name,
    },
    "tokenizer":     unicode.Name,
    "token_filters": []string{
        lowercase.Name,
        ...
    },
})

func (*IndexMappingImpl) AddCustomCharFilter

func (im *IndexMappingImpl) AddCustomCharFilter(name string, config map[string]interface{}) error

AddCustomCharFilter defines a custom char filter for use in this mapping

func (*IndexMappingImpl) AddCustomDateTimeParser

func (im *IndexMappingImpl) AddCustomDateTimeParser(name string, config map[string]interface{}) error

AddCustomDateTimeParser defines a custom date time parser for use in this mapping

func (*IndexMappingImpl) AddCustomTokenFilter

func (im *IndexMappingImpl) AddCustomTokenFilter(name string, config map[string]interface{}) error

AddCustomTokenFilter defines a custom token filter for use in this mapping

func (*IndexMappingImpl) AddCustomTokenMap

func (im *IndexMappingImpl) AddCustomTokenMap(name string, config map[string]interface{}) error

AddCustomTokenMap defines a custom token map for use in this mapping

func (*IndexMappingImpl) AddCustomTokenizer

func (im *IndexMappingImpl) AddCustomTokenizer(name string, config map[string]interface{}) error

AddCustomTokenizer defines a custom tokenizer for use in this mapping

func (*IndexMappingImpl) AddDocumentMapping

func (im *IndexMappingImpl) AddDocumentMapping(doctype string, dm *DocumentMapping)

AddDocumentMapping sets a custom document mapping for the specified type

func (*IndexMappingImpl) AnalyzeText

func (im *IndexMappingImpl) AnalyzeText(analyzerName string, text []byte) (analysis.TokenStream, error)

func (*IndexMappingImpl) AnalyzerNameForPath

func (im *IndexMappingImpl) AnalyzerNameForPath(path string) string

AnalyzerNameForPath attempts to find the best analyzer to use with only a field name will walk all the document types, look for field mappings at the provided path, if one exists and it has an explicit analyzer that is returned.

func (*IndexMappingImpl) AnalyzerNamed

func (im *IndexMappingImpl) AnalyzerNamed(name string) analysis.Analyzer

func (*IndexMappingImpl) DateTimeParserNamed

func (im *IndexMappingImpl) DateTimeParserNamed(name string) analysis.DateTimeParser

func (*IndexMappingImpl) DefaultSearchField

func (im *IndexMappingImpl) DefaultSearchField() string

func (*IndexMappingImpl) FieldAnalyzer

func (im *IndexMappingImpl) FieldAnalyzer(field string) string

FieldAnalyzer returns the name of the analyzer used on a field.

func (*IndexMappingImpl) FieldMappingForPath added in v2.4.0

func (im *IndexMappingImpl) FieldMappingForPath(path string) FieldMapping

FieldMappingForPath returns the mapping for a specific field 'path'.

func (*IndexMappingImpl) MapDocument

func (im *IndexMappingImpl) MapDocument(doc *document.Document, data interface{}) error

func (*IndexMappingImpl) UnmarshalJSON

func (im *IndexMappingImpl) UnmarshalJSON(data []byte) error

UnmarshalJSON offers custom unmarshaling with optional strict validation

func (*IndexMappingImpl) Validate

func (im *IndexMappingImpl) Validate() error

Validate will walk the entire structure ensuring the following explicitly named and default analyzers can be built

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL