spell

README

spell

spell is a spell checking package, originally based on https://github.com/sajari/fuzzy

As of 7/2024, we only use a basic dictionary list of words as the input, and compute all the variants for checking on the fly after loading, to save massively on the file size.

The input dictionary file must be a simple one-word-per-line file, similar to /usr/share/dict/words

Word lists

It is somewhat difficult to find suitable lists of words, especially in different languages! Here are some resources:

• https://unix.stackexchange.com/questions/213628/where-do-the-words-in-usr-share-dict-words-come-from

• SCOWL: http://wordlist.aspell.net/ -- source of /usr/share/dict/words

• https://github.com/dwyl/english-words

• https://stackoverflow.com/questions/2213607/how-to-get-english-language-word-database

• https://web.archive.org/web/20120215032551/http://en-gb.pyxidium.co.uk/dictionary

• https://www.cs.hmc.edu/~geoff/ispell-dictionaries.html -- source with lots of different languages.

Documentation

Overview

Package spell provides functions for spell check and correction. It wraps https://github.com/sajari/fuzzy as the core spelling engine.

A single globally usable spelling dictionary is managed.

Index

• Constants
• func CheckLexLine(src []rune, tags lexer.Line) lexer.Line
• func Edits1(word string) []string
• func Levenshtein(a, b *string) int
• type Dict
  • func NewDictFromList(wl []string) Dict
  • func OpenDict(fname string) (Dict, error)
  • func OpenDictFS(fsys fs.FS, filename string) (Dict, error)
  • func (d Dict) Add(word string)
  • func (d Dict) Exists(word string) bool
  • func (d Dict) List() []string
  • func (d Dict) Save(fname string) error
• type Model
  • func NewModel() *Model
  • func (md *Model) AddWord(term string)
  • func (md *Model) Delete(term string)
  • func (md *Model) EditsMulti(term string, depth int) []string
  • func (md *Model) Init() *Model
  • func (md *Model) SetDicts(base, user Dict)
  • func (md *Model) Suggestions(input string, n int) []string
• type Pair
• type SpellData
  • func NewSpell(userFile string) *SpellData
  • func (sp *SpellData) AddWord(word string)
  • func (sp *SpellData) CheckWord(word string) ([]string, bool)
  • func (sp *SpellData) DeleteWord(word string)
  • func (sp *SpellData) IgnoreWord(word string)
  • func (sp *SpellData) OpenUser() error
  • func (sp *SpellData) OpenUserCheck() error
  • func (sp *SpellData) ResetLearnTime()
  • func (sp *SpellData) SaveUser() error
  • func (sp *SpellData) SaveUserIfLearn() error

Constants

const SaveAfterLearnIntervalSecs = 20

SaveAfterLearnIntervalSecs is number of seconds since dict file has been opened / saved above which model is saved after learning.

Functions

func CheckLexLine

func CheckLexLine(src []rune, tags lexer.Line) lexer.Line

CheckLexLine returns the Lex regions for any words that are misspelled within given line of text with existing Lex tags -- automatically excludes any Code token regions (see token.IsCode). Token is set to token.TextSpellErr on returned Lex's

func Edits1

func Edits1(word string) []string

Edits1 creates a set of terms that are 1 char delete from the input term

func Levenshtein

func Levenshtein(a, b *string) int

Calculate the Levenshtein distance between two strings

Types

type Dict

type Dict map[string]struct{}

func NewDictFromList

func NewDictFromList(wl []string) Dict

NewDictFromList makes a new dictionary from given list (slice) of words

func OpenDict

func OpenDict(fname string) (Dict, error)

OpenDict opens a dictionary list of words from a simple one-word-per-line list

func OpenDictFS

func OpenDictFS(fsys fs.FS, filename string) (Dict, error)

OpenDictFS opens a dictionary list of words from a simple one-word-per-line list, from given filesystem

func (Dict) Add

func (d Dict) Add(word string)

func (Dict) Exists

func (d Dict) Exists(word string) bool

func (Dict) List

func (d Dict) List() []string

List returns a list (slice) of words in dictionary in alpha-sorted order

func (Dict) Save

func (d Dict) Save(fname string) error

Save saves a dictionary list of words to a simple one-word-per-line list, in alpha order

type Model

type Model struct {
	// list of all words, combining Base and User dictionaries
	Dict Dict

	// user dictionary of additional words
	UserDict Dict

	// words to ignore for this session
	Ignore Dict

	// map of misspelled word to potential correct spellings
	Suggest map[string][]string

	// depth of edits to include in Suggest map (2 is only sensible value)
	Depth int

	sync.RWMutex
}

Model is the full data model

func NewModel

func NewModel() *Model

Create and initialise a new model

func (*Model) AddWord

func (md *Model) AddWord(term string)

AddWord adds a new word to user dictionary, and generates new suggestions for it

func (*Model) Delete

func (md *Model) Delete(term string)

Delete removes given word from dictionary -- undoes learning

func (*Model) EditsMulti

func (md *Model) EditsMulti(term string, depth int) []string

Edits at any depth for a given term. The depth of the model is used

func (*Model) Init

func (md *Model) Init() *Model

func (*Model) SetDicts

func (md *Model) SetDicts(base, user Dict)

func (*Model) Suggestions

func (md *Model) Suggestions(input string, n int) []string

Return the most likely corrections in order from best to worst

type Pair

type Pair struct {
	// contains filtered or unexported fields
}

type SpellData

type SpellData struct {
	// UserFile is path to user's dictionary where learned words go
	UserFile string
	// contains filtered or unexported fields
}

var Spell *SpellData

Spell is the global shared spell

func NewSpell

func NewSpell(userFile string) *SpellData

NewSpell opens spell data with given user dictionary file

func (*SpellData) AddWord

func (sp *SpellData) AddWord(word string)

AddWord adds given word to the User dictionary

func (*SpellData) CheckWord

func (sp *SpellData) CheckWord(word string) ([]string, bool)

CheckWord checks a single word and returns suggestions if word is unknown. bool is true if word is in the dictionary, false otherwise.

func (*SpellData) DeleteWord

func (sp *SpellData) DeleteWord(word string)

DeleteWord removes word from dictionary, in case accidentally added

func (*SpellData) IgnoreWord

func (sp *SpellData) IgnoreWord(word string)

IgnoreWord adds the word to the Ignore list

func (*SpellData) OpenUser

func (sp *SpellData) OpenUser() error

OpenUser opens user dictionary of words

func (*SpellData) OpenUserCheck

func (sp *SpellData) OpenUserCheck() error

OpenUserCheck checks if the current user dict file has been modified since last open time and re-opens it if so.

func (*SpellData) ResetLearnTime

func (sp *SpellData) ResetLearnTime()

func (*SpellData) SaveUser

func (sp *SpellData) SaveUser() error

SaveUser saves the user dictionary note: this will overwrite any existing file; be sure to have opened the current file before making any changes.

func (*SpellData) SaveUserIfLearn

func (sp *SpellData) SaveUserIfLearn() error

SaveUserIfLearn saves the user dictionary if learning has occurred since last save / open. If no changes also checks if file has been modified and opens it if so.