languagemodeling

Documentation

Index

• Constants
• Variables
• type Interface
• type Parameters
• type Response
• type Token

Constants

const (
	// DefaultModel is a BERT pretrained model on English language using a masked language modeling (MLM) objective.
	// Model card: https://huggingface.co/bert-base-cased
	DefaultModel = "bert-base-cased"

	// DefaultItalianModel is a BERT pretrained model on Italian language using a masked language modeling (MLM) objective.
	// Model card: https://huggingface.co/dbmdz/bert-base-italian-cased
	DefaultItalianModel = "dbmdz/bert-base-italian-cased"
)

Variables

var ErrInputSequenceTooLong = errors.New("input sequence too long")

ErrInputSequenceTooLong means that pre-processing the input text produced a sequence that exceeds the maximum allowed length.

Types

type Interface

type Interface interface {
	// Predict returns the prediction of the given example.
	Predict(ctx context.Context, text string, parameters Parameters) (Response, error)
}

Interface defines the main functions for language modelling.

type Parameters

type Parameters struct {
	// K is the number of returned predictions per token
	K int
}

Parameters contains the parameters for language modeling.

type Response

type Response struct {
	Tokens []Token
}

Response contains the response from language modelling..

type Token

type Token struct {
	Start  int
	End    int
	Words  []string
	Scores []float64
}

Token is a labeled text token.