chunk

Documentation

Index

• Constants
• func ExtractFilesFromZip(f multipart.File) ([]string, []string, error)
• func ExtractTextFromPDF(f multipart.File, fileSize int64) (string, error)
• func GetTextFromFile(f multipart.File) (string, error)
• type Chunk
  • func CreateChunks(fileContent string, title string) ([]Chunk, error)

Constants

const EmbeddingModel = "text-embedding-ada-002"

const MaxTokensPerChunk = 1500

MaxTokensPerChunk is the maximum number of tokens allowed in a single chunk for OpenAI embeddings MaxTokensPerChunk is the maximum number of tokens allowed in a single chunk for OpenAI embeddings

Functions

func ExtractFilesFromZip

func ExtractFilesFromZip(f multipart.File) ([]string, []string, error)

ExtractFilesFromZip extracts the text from files within a zip file

func ExtractTextFromPDF

func ExtractTextFromPDF(f multipart.File, fileSize int64) (string, error)

extract human-readable text from a given pdf with support for spaces/whitespace.

func GetTextFromFile

func GetTextFromFile(f multipart.File) (string, error)

Types

type Chunk

type Chunk struct {
	Start int
	End   int
	Title string
	Text  string
}

func CreateChunks

func CreateChunks(fileContent string, title string) ([]Chunk, error)