chunk

Documentation

Index

• func BreakSentences(content string, chunkSize int, debug bool) (result []string)
• func GetBestBreakPoint(content string, first int, last int) (int, int)
• type ContextualNode
• type Part
• type RichTextChunker
  • func NewChunkedRichText(content string, chunkSize int, debug bool) (result RichTextChunker)
  • func (c *RichTextChunker) Finish() (result string)
  • func (c *RichTextChunker) MakeChunks()
• type TextPart

Functions

func BreakSentences

func BreakSentences(content string, chunkSize int, debug bool) (result []string)

func GetBestBreakPoint

func GetBestBreakPoint(content string, first int, last int) (int, int)

If we found a good point match we just return it as the first argument. If there was no punctuation character the first return value is -1 and the 2nd return value contains the position of the first whitespace. also the 2nd return value is -1 if there was no whitespace.

Types

type ContextualNode

type ContextualNode struct {
	Text string
	// contains filtered or unexported fields
}

type Part

type Part struct {
	Text string
	// contains filtered or unexported fields
}

type RichTextChunker

type RichTextChunker struct {
	Content   string
	ChunkSize int

	TextParts []TextPart
	Debug     bool
	// contains filtered or unexported fields
}

func NewChunkedRichText

func NewChunkedRichText(content string, chunkSize int, debug bool) (result RichTextChunker)

constructor

func (*RichTextChunker) Finish

func (c *RichTextChunker) Finish() (result string)

func (*RichTextChunker) MakeChunks

func (c *RichTextChunker) MakeChunks()

type TextPart

type TextPart struct {
	ContextualNode *ContextualNode

	Text string
	// contains filtered or unexported fields
}