documentloaders

Documentation

Index

• Constants
• type AudioWithWhisper
  • func NewAudoWithWhisper(data []byte, fileName, language, output string, vadFilter bool) AudioWithWhisper
  • func (aww AudioWithWhisper) CovertToText(ctx context.Context) (string, error)
  • func (aww AudioWithWhisper) Load(ctx context.Context) ([]schema.Document, error)
  • func (aww AudioWithWhisper) LoadAndSplit(ctx context.Context, splitter textsplitter.TextSplitter) ([]schema.Document, error)
• type Option
  • func WithAnswerColumn(s string) Option
  • func WithChunkContentColumn(s string) Option
  • func WithFileNameColumn(s string) Option
  • func WithPageNumberColumn(s string) Option
  • func WithQuestionColumn(s string) Option
• type PDF
  • func NewPDF(r io.Reader, fileName string) *PDF
  • func (p *PDF) Load(ctx context.Context) ([]schema.Document, error)
  • func (p *PDF) LoadAndSplit(ctx context.Context, splitter textsplitter.TextSplitter) ([]schema.Document, error)
• type QACSV
  • func NewQACSV(r io.Reader, fileName string, opts ...Option) QACSV
  • func (c QACSV) Load(_ context.Context) ([]schema.Document, error)
  • func (c QACSV) LoadAndSplit(ctx context.Context, splitter textsplitter.TextSplitter) ([]schema.Document, error)

Constants

const (
	// QustionCol the question column, which will be embedding
	QuestionCol = "q"
	// AnswerCol the answer column, will be added to qustionCol when use qachain and knowledgebase retriever
	AnswerCol = "a"
	// FileNameCol the file name column, will show in reference
	FileNameCol = "file_name"
	// PageNumberCol the page number column, will show in reference
	PageNumberCol = "page_number"
	// ChunkContentCol the chunk content column, will show in reference
	ChunkContentCol = "chunk_content"
	// LineNumber the qafile line number column,
	LineNumber = "line_number"
	// QAFileName the qafile name
	QAFileName = "qafile_name"
)

Types

type AudioWithWhisper

type AudioWithWhisper struct {
	// contains filtered or unexported fields
}

AudioWithWhisper represents a audio using whisper document loader.

func NewAudoWithWhisper

func NewAudoWithWhisper(data []byte, fileName, language, output string, vadFilter bool) AudioWithWhisper

NewAudoWithWhisper creates a new audo loader with an io.Reader and optional column names for filtering.

func (AudioWithWhisper) CovertToText

func (aww AudioWithWhisper) CovertToText(ctx context.Context) (string, error)

input will be from the previous LLM

func (AudioWithWhisper) Load

func (aww AudioWithWhisper) Load(ctx context.Context) ([]schema.Document, error)

Load reads from the io.Reader and returns a document with the data.

func (AudioWithWhisper) LoadAndSplit

func (aww AudioWithWhisper) LoadAndSplit(ctx context.Context, splitter textsplitter.TextSplitter) ([]schema.Document, error)

LoadAndSplit reads text data from the io.Reader and splits it into multiple documents using a text splitter.

type Option

type Option func(p *QACSV)

Option is a function type that can be used to modify the client.

func WithAnswerColumn

func WithAnswerColumn(s string) Option

func WithChunkContentColumn

func WithChunkContentColumn(s string) Option

func WithFileNameColumn

func WithFileNameColumn(s string) Option

func WithPageNumberColumn

func WithPageNumberColumn(s string) Option

func WithQuestionColumn

func WithQuestionColumn(s string) Option

type PDF

type PDF struct {
	// contains filtered or unexported fields
}

func NewPDF

func NewPDF(r io.Reader, fileName string) *PDF

func (*PDF) Load

func (p *PDF) Load(ctx context.Context) ([]schema.Document, error)

func (*PDF) LoadAndSplit

func (p *PDF) LoadAndSplit(ctx context.Context, splitter textsplitter.TextSplitter) ([]schema.Document, error)

type QACSV

type QACSV struct {
	// contains filtered or unexported fields
}

QACSV represents a QA CSV document loader.

func NewQACSV

func NewQACSV(r io.Reader, fileName string, opts ...Option) QACSV

NewQACSV creates a new qa csv loader with an io.Reader and optional column names for filtering.

func (QACSV) Load

func (c QACSV) Load(_ context.Context) ([]schema.Document, error)

Load reads from the io.Reader and returns a single document with the data.

func (QACSV) LoadAndSplit

func (c QACSV) LoadAndSplit(ctx context.Context, splitter textsplitter.TextSplitter) ([]schema.Document, error)

LoadAndSplit reads text data from the io.Reader and splits it into multiple documents using a text splitter.