ml

Documentation

Index

• Variables
• func ExtractFeatures(config *scraper.Config, featureFile, wordsDir string) error
• func TrainModel(filename string) error
• type Features
• type Labler
  • func LoadLabler(modelName, wordListsDir string) (*Labler, error)
  • func (ll *Labler) PredictLabel(fValue ...string) (string, error)

Variables

var NonAlphaFeatureList []string = []string{
	"digit-count",
	"rune-count",
	"dict-words-count",
	"slash-count",
	"colon-count",
	"dash-count",
	"dot-count",
	"whitespace-count",
	"class",
}

NonAlphaFeatureList contains a list of strings representing the Features excluding the letter frequencies

Functions

func ExtractFeatures

func ExtractFeatures(config *scraper.Config, featureFile, wordsDir string) error

ExtractFeatures extracts features based on a given configuration and a directory containing words of different languages. Those features can then be used to train a ML model to automatically classify scraped fields for new websites.

func TrainModel

func TrainModel(filename string) error

Types

type Features

type Features struct {
	// contains filtered or unexported fields
}

Features contains all the relevant features and the class label

type Labler

type Labler struct {
	// contains filtered or unexported fields
}

func LoadLabler

func LoadLabler(modelName, wordListsDir string) (*Labler, error)

func (*Labler) PredictLabel

func (ll *Labler) PredictLabel(fValue ...string) (string, error)