embeddings

Documentation

Overview

Package embeddings contains the implementation for creating vector embeddings from text using different APIs, like OpenAI and Google PaLM (VertexAI).

The main components of this package are:

- Embedder interface: a common interface for creating vector embeddings from texts. - OpenAI: an Embedder implementation using the OpenAI API. - VertexAIPaLM: an Embedder implementation using Google PaLM (VertexAI) API. - Helper functions: utility functions for embedding, such as `batchTexts` and `maybeRemoveNewLines`.

The package provides a flexible way to handle different APIs for generating embeddings by using the Embedder interface as an abstraction.

Index

• Variables
• func BatchTexts(texts []string, batchSize int) [][]string
• func CombineVectors(vectors [][]float64, weights []int) ([]float64, error)
• func MaybeRemoveNewLines(texts []string, removeNewLines bool) []string
• type Embedder

Variables

var (
	// ErrVectorsNotSameSize is returned if the vectors returned from the
	// embeddings api have different sizes.
	ErrVectorsNotSameSize = errors.New("vectors gotten not the same size")
	// ErrAllTextsLenZero is returned if all texts to be embedded has the combined
	// length of zero.
	ErrAllTextsLenZero = errors.New("all texts have length 0")
)

Functions

func BatchTexts

func BatchTexts(texts []string, batchSize int) [][]string

BatchTexts splits strings by the length batchSize.

func CombineVectors

func CombineVectors(vectors [][]float64, weights []int) ([]float64, error)

func MaybeRemoveNewLines

func MaybeRemoveNewLines(texts []string, removeNewLines bool) []string

Types

type Embedder

type Embedder interface {
	// EmbedDocuments returns a vector for each text.
	EmbedDocuments(ctx context.Context, texts []string) ([][]float64, error)
	// EmbedQuery embeds a single text.
	EmbedQuery(ctx context.Context, text string) ([]float64, error)
}

Embedder is the interface for creating vector embeddings from texts.