nlm

README

No Language Model

It contrast to Large Language Models (LLMs), this is a No Language Model (NLM) for generating text. I use a Markov chain to generate text in a probabalistic way based on probabilities obtained from a number of source articles.

Documentation

Index

• Constants
• func GetArticleList() ([]string, error)
• type Article
  • func GetNamedArticle(name string) (*Article, error)
  • func LoadArticle(r io.Reader) (*Article, error)
• type ArticleBuilder
  • func NewArticleBuilder(titlesize, size int) (*ArticleBuilder, error)
  • func (ab *ArticleBuilder) GenerateArticle(titleSize, articleSize int) *Article
• type CountWeightedList
  • func NewCountWeightedList[T comparable](counts map[T]int) *CountWeightedList[T]
  • func (cwl *CountWeightedList[T]) GetRandomItem() T
• type MarkovBuilder
  • func NewMarkovBuilder() *MarkovBuilder
  • func (b *MarkovBuilder) AddText(text string, size int, endCharacter rune)
  • func (b *MarkovBuilder) ConvertToSource() MarkovSource
• type MarkovSource
  • func (s MarkovSource) GenerateText(start string, maxLength int) []string
  • func (s MarkovSource) GetNextCharacter(v []rune) rune

Constants

const EndOfDocument = '\u0003'

const EndOfParagraph = '\u001d'

Functions

func GetArticleList

func GetArticleList() ([]string, error)

Types

type Article

type Article struct {
	Title      string `yaml:"title"`
	Subtitle   string `yaml:"subtitle"`
	Paragraphs []string
}

func GetNamedArticle

func GetNamedArticle(name string) (*Article, error)

func LoadArticle

func LoadArticle(r io.Reader) (*Article, error)

type ArticleBuilder

type ArticleBuilder struct {
	TitleMarkov     MarkovSource `json:"title_markov"`
	ContentMarkov   MarkovSource `json:"content_markov"`
	TitleStarters   []string     `json:"title_starters"`
	ContentStarters []string     `json:"content_starters"`
}

func NewArticleBuilder

func NewArticleBuilder(titlesize, size int) (*ArticleBuilder, error)

NewArticleBuilder creates a Markov article builder with a chain size of titlesize for titles and size for content.

func (*ArticleBuilder) GenerateArticle

func (ab *ArticleBuilder) GenerateArticle(titleSize, articleSize int) *Article

type CountWeightedList

type CountWeightedList[T comparable] struct {
	Items             []T   `json:"items"`
	CumulativeWeights []int `json:"weights"`
	Total             int   `json:"total"`
}

CountWeightedList holds a weighted list of runes and allows selection of random runes weighted by count.

func NewCountWeightedList

func NewCountWeightedList[T comparable](counts map[T]int) *CountWeightedList[T]

NewCountWeightedList returns a pointer to a CountWeightedList structure from which you can select random elements weighted by count.

func (*CountWeightedList[T]) GetRandomItem

func (cwl *CountWeightedList[T]) GetRandomItem() T

GetRandomItem returns a random rune from the weighted list.

type MarkovBuilder

type MarkovBuilder struct {
	Counts   map[string]map[rune]int
	Fragment []rune
}

MarkovBuilder is a map of strings pointing to counts of letters that follow each string. This is used while building a Markov chain structure

func NewMarkovBuilder

func NewMarkovBuilder() *MarkovBuilder

func (*MarkovBuilder) AddText

func (b *MarkovBuilder) AddText(text string, size int, endCharacter rune)

AddText adds text to the Markov Builder, creating a table that will work with a chain length of size. It also includes a defined endCharacter that indicates the end of the passage.

func (*MarkovBuilder) ConvertToSource

func (b *MarkovBuilder) ConvertToSource() MarkovSource

type MarkovSource

type MarkovSource map[string](*CountWeightedList[rune])

func (MarkovSource) GenerateText

func (s MarkovSource) GenerateText(start string, maxLength int) []string

func (MarkovSource) GetNextCharacter

func (s MarkovSource) GetNextCharacter(v []rune) rune