postprocessors

package
v0.4.14-rc.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2024 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package postprocessors is basically the same as package transformers, but used at a different stage of the RAG pipeline

Index

Constants

View Source
const BM25PostprocessorName = "bm25"
View Source
const CohereRerankPostprocessorName = "cohere_rerank"
View Source
const ContentFilterPostprocessorName = "content_filter"
View Source
const ContentSubstringFilterPostprocessorName = "content_substring_filter"
View Source
const ReducePostprocessorName = "reduce"
View Source
const SimilarityPostprocessorName = "similarity"

Variables

Functions

This section is empty.

Types

type BM25Postprocessor added in v0.4.11

type BM25Postprocessor struct {
	TopN         int
	SparseWeight float64 // How to weight the BM25 scores against the similarity scores from dense vector search

	K1 float64 // K1 should be between 1.2 and 2 - controls term frequency saturation
	B  float64 // B should be around 0.75 - controls the influence of document length normalization

	CleanStopWords []string // list of stopwords to remove from the documents - if empty, no stopwords are removed, if only "auto" is present, the language is detected automatically
}

func (*BM25Postprocessor) Name added in v0.4.11

func (c *BM25Postprocessor) Name() string

func (*BM25Postprocessor) Transform added in v0.4.11

func (c *BM25Postprocessor) Transform(ctx context.Context, response *types.RetrievalResponse) error

type CohereRerankPostprocessor added in v0.1.7

type CohereRerankPostprocessor struct {
	ApiKey string `json:"apiKey" yaml:"apiKey"`
	Model  string
	TopN   int
}

func (*CohereRerankPostprocessor) Name added in v0.1.9

func (*CohereRerankPostprocessor) Transform added in v0.1.7

type ContentFilterPostprocessor added in v0.1.7

type ContentFilterPostprocessor struct {
	Question string // Question about the content, that can be answered with yes or no
	Include  bool   // Whether to include or exclude the documents for which the answer is yes
	LLM      llm.LLM
}

func (*ContentFilterPostprocessor) Name added in v0.1.9

func (*ContentFilterPostprocessor) Transform added in v0.1.7

type ContentSubstringFilterPostprocessor added in v0.1.7

type ContentSubstringFilterPostprocessor struct {
	Contains    []string
	NotContains []string
}

func (*ContentSubstringFilterPostprocessor) Name added in v0.1.9

func (*ContentSubstringFilterPostprocessor) Transform added in v0.1.7

type Postprocessor

type Postprocessor interface {
	Transform(ctx context.Context, response *types.RetrievalResponse) error
	Name() string
}

Postprocessor is similar to types.DocumentTransformer, but can take into account the retrieval query

func GetPostprocessor

func GetPostprocessor(name string) (Postprocessor, error)

type ReducePostprocessor added in v0.4.8

type ReducePostprocessor struct {
	TopK int
}

func (*ReducePostprocessor) Name added in v0.4.8

func (s *ReducePostprocessor) Name() string

func (*ReducePostprocessor) Transform added in v0.4.8

func (s *ReducePostprocessor) Transform(ctx context.Context, response *types.RetrievalResponse) error

type SimilarityPostprocessor added in v0.1.7

type SimilarityPostprocessor struct {
	Threshold float32
	KeepMin   int // KeepMin the top n documents, regardless of the threshold
}

func (*SimilarityPostprocessor) Name added in v0.1.9

func (s *SimilarityPostprocessor) Name() string

func (*SimilarityPostprocessor) Transform added in v0.1.7

func (s *SimilarityPostprocessor) Transform(ctx context.Context, response *types.RetrievalResponse) error

type TransformerWrapper added in v0.1.7

type TransformerWrapper struct {
	DocumentTransformer types.DocumentTransformer
}

func NewTransformerWrapper added in v0.1.7

func NewTransformerWrapper(transformer types.DocumentTransformer) *TransformerWrapper

func (*TransformerWrapper) Decode added in v0.4.12

func (t *TransformerWrapper) Decode(cfg map[string]any) error

func (*TransformerWrapper) Name added in v0.1.9

func (t *TransformerWrapper) Name() string

func (*TransformerWrapper) Transform added in v0.1.7

func (t *TransformerWrapper) Transform(ctx context.Context, response *types.RetrievalResponse) error

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL