runedelimiterpretokenizer

Documentation

Index

• type RuneDelimiterPreTokenizer
  • func New(delimiter rune) *RuneDelimiterPreTokenizer
  • func (r *RuneDelimiterPreTokenizer) PreTokenize(pts *pretokenizedstring.PreTokenizedString) error

Types

type RuneDelimiterPreTokenizer

type RuneDelimiterPreTokenizer struct {
	// contains filtered or unexported fields
}

RuneDelimiterPreTokenizer allows the generation of pre-tokens splitting by a specific rune.

func New

func New(delimiter rune) *RuneDelimiterPreTokenizer

New returns a new RuneDelimiterPreTokenizer, setting the given rune as delimiter.

func (*RuneDelimiterPreTokenizer) PreTokenize

func (r *RuneDelimiterPreTokenizer) PreTokenize(pts *pretokenizedstring.PreTokenizedString) error

PreTokenize splits the NormalizedString by rune delimiter.