textsplitter

Documentation

Overview

Package textsplitter provides tools for splitting long texts into smaller chunks based on configurable rules and parameters. It aims to help in processing these chunks more efficiently when interacting with language models or other text-processing tools.

The main components of this package are:

- TextSplitter interface: a common interface for splitting texts into smaller chunks. - RecursiveCharacter: a text splitter that recursively splits texts by different characters (separators) combined with chunk size and overlap settings. - Helper functions: utility functions for creating documents out of split texts and rejoining them if necessary.

Using the TextSplitter interface, developers can implement custom splitting strategies for their specific use cases and requirements.

Index

• Variables
• func CreateDocuments(textSplitter TextSplitter, texts []string, metadatas []map[string]any) ([]schema.Document, error)
• func SplitDocuments(textSplitter TextSplitter, documents []schema.Document) ([]schema.Document, error)
• type MarkdownTextSplitter
  • func NewMarkdownTextSplitter(opts ...Option) *MarkdownTextSplitter
  • func (sp MarkdownTextSplitter) SplitText(text string) ([]string, error)
• type Option
  • func WithAllowedSpecial(allowedSpecial []string) Option
  • func WithChunkOverlap(chunkOverlap int) Option
  • func WithChunkSize(chunkSize int) Option
  • func WithDisallowedSpecial(disallowedSpecial []string) Option
  • func WithEncodingName(encodingName string) Option
  • func WithModelName(modelName string) Option
  • func WithSecondSplitter(secondSplitter TextSplitter) Option
  • func WithSeparators(separators []string) Option
• type Options
  • func DefaultOptions() Options
• type RecursiveCharacter
  • func NewRecursiveCharacter(opts ...Option) RecursiveCharacter
  • func (s RecursiveCharacter) SplitText(text string) ([]string, error)
• type TextSplitter
• type TokenSplitter
  • func NewTokenSplitter(opts ...Option) TokenSplitter
  • func (s TokenSplitter) SplitText(text string) ([]string, error)

Variables

var ErrMismatchMetadatasAndText = errors.New("number of texts and metadatas does not match")

ErrMismatchMetadatasAndText is returned when the number of texts and metadatas given to CreateDocuments does not match. The function will not error if the length of the metadatas slice is zero.

Functions

func CreateDocuments

func CreateDocuments(textSplitter TextSplitter, texts []string, metadatas []map[string]any) ([]schema.Document, error)

CreateDocuments creates documents from texts and metadatas with a text splitter. If the length of the metadatas is zero, the result documents will contain no metadata. Otherwise, the numbers of texts and metadatas must match.

func SplitDocuments

func SplitDocuments(textSplitter TextSplitter, documents []schema.Document) ([]schema.Document, error)

SplitDocuments splits documents using a textsplitter.

Types

type MarkdownTextSplitter

type MarkdownTextSplitter struct {
	ChunkSize    int
	ChunkOverlap int
	// SecondSplitter splits paragraphs
	SecondSplitter TextSplitter
}

MarkdownTextSplitter markdown header text splitter.

Now, we support H1/H2/H3/H4/H5/H6, BulletList, OrderedList, Table, Paragraph, Blockquote, other format will be ignored. If your origin document is HTML, you purify and convert to markdown, then split it.

func NewMarkdownTextSplitter

func NewMarkdownTextSplitter(opts ...Option) *MarkdownTextSplitter

NewMarkdownTextSplitter creates a new Markdown text splitter.

func (MarkdownTextSplitter) SplitText

func (sp MarkdownTextSplitter) SplitText(text string) ([]string, error)

SplitText splits a text into multiple text.

type Option

type Option func(*Options)

Option is a function that can be used to set options for a text splitter.

func WithAllowedSpecial

func WithAllowedSpecial(allowedSpecial []string) Option

WithAllowedSpecial sets the allowed special tokens for a text splitter.

func WithChunkOverlap

func WithChunkOverlap(chunkOverlap int) Option

WithChunkOverlap sets the chunk overlap for a text splitter.

func WithChunkSize

func WithChunkSize(chunkSize int) Option

WithChunkSize sets the chunk size for a text splitter.

func WithDisallowedSpecial

func WithDisallowedSpecial(disallowedSpecial []string) Option

WithDisallowedSpecial sets the disallowed special tokens for a text splitter.

func WithEncodingName

func WithEncodingName(encodingName string) Option

WithEncodingName sets the encoding name for a text splitter.

func WithModelName

func WithModelName(modelName string) Option

WithModelName sets the model name for a text splitter.

func WithSecondSplitter

func WithSecondSplitter(secondSplitter TextSplitter) Option

WithSecondSplitter sets the second splitter for a text splitter.

func WithSeparators

func WithSeparators(separators []string) Option

WithSeparators sets the separators for a text splitter.

type Options

type Options struct {
	ChunkSize         int
	ChunkOverlap      int
	Separators        []string
	ModelName         string
	EncodingName      string
	AllowedSpecial    []string
	DisallowedSpecial []string
	SecondSplitter    TextSplitter
}

Options is a struct that contains options for a text splitter.

func DefaultOptions

func DefaultOptions() Options

DefaultOptions returns the default options for all text splitter.

type RecursiveCharacter

type RecursiveCharacter struct {
	Separators   []string
	ChunkSize    int
	ChunkOverlap int
}

RecursiveCharacter is a text splitter that will split texts recursively by different characters.

func NewRecursiveCharacter

func NewRecursiveCharacter(opts ...Option) RecursiveCharacter

NewRecursiveCharacter creates a new recursive character splitter with default values. By default, the separators used are "\n\n", "\n", " " and "". The chunk size is set to 4000 and chunk overlap is set to 200.

func (RecursiveCharacter) SplitText

func (s RecursiveCharacter) SplitText(text string) ([]string, error)

SplitText splits a text into multiple text.

type TextSplitter

type TextSplitter interface {
	SplitText(text string) ([]string, error)
}

TextSplitter is the standard interface for splitting texts.

type TokenSplitter

type TokenSplitter struct {
	ChunkSize         int
	ChunkOverlap      int
	ModelName         string
	EncodingName      string
	AllowedSpecial    []string
	DisallowedSpecial []string
}

TokenSplitter is a text splitter that will split texts by tokens.

func NewTokenSplitter

func NewTokenSplitter(opts ...Option) TokenSplitter

func (TokenSplitter) SplitText

func (s TokenSplitter) SplitText(text string) ([]string, error)

SplitText splits a text into multiple text.