generationutils

README

This package is a derivative work from Hugging Face Transformers generation utils .

Copyright 2020 The Google AI Language Team Authors, Facebook AI Research Authors, and The HuggingFace Inc. team. Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an " AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Documentation

Overview

Package generationutils implements a decoding search algorithm for conditional generation.

Index

• type BeamSearchDecoder
  • func (b *BeamSearchDecoder) Decode(ctx context.Context) ([][]int, []float64)
• type Config
• type DecodingStrategyFunc
• type PredictNextFunc
• type ScoreProcessor
  • func ProcessScores(processors ...ScoreProcessor) ScoreProcessor
  • func TemperatureProcessor(temperature float64) ScoreProcessor
  • func TopKProcessor(topK int, filterValue float64) ScoreProcessor
  • func TopPProcessor[T float.DType](topP, filterValue T, minSize int) ScoreProcessor
• type ScoredToken
  • func SelectNextMultinomial(tokensScores []mat.Matrix, resultSize int) []*ScoredToken
  • func SelectNextTopK(tokensScores []mat.Matrix, resultSize int) []*ScoredToken

Types

type BeamSearchDecoder

type BeamSearchDecoder struct {
	// Config is the configuration of the beam decoder.
	Config Config
	// PredictNext is a function that predicts the next tokens given the current tokens.
	PredictNext PredictNextFunc
	// SelectNext is a function that selects the next tokens given the current tokens.
	SelectNext DecodingStrategyFunc
}

BeamSearchDecoder is an implementations of a decoding search algorithm for conditional generation.

func (*BeamSearchDecoder) Decode

func (b *BeamSearchDecoder) Decode(ctx context.Context) ([][]int, []float64)

Decode generates sequences for model with a language modeling head, using beam-search decoding.

type Config

type Config struct {
	// NumBeams is the number of beams for decoding search.
	NumBeams int
	// MinLength is the minimum length of the sequence to be generated.
	MinLength int
	// MaxLength is the maximum length of the sequence to be generated.
	MaxLength int
	// IsEncoderDecoder reports whether the model is used as an encoder/decoder.
	IsEncoderDecoder bool
	// BOSTokenID is the ID of the Beginning-Of-Sequence token.
	BOSTokenID int
	// EOSTokenID is the ID of the End-Of-Sequence token.
	EOSTokenID int
	// PadTokenID is the id of the padding token.
	PadTokenID int
	// VocabSize is the size of the vocabulary.
	VocabSize int
	// DecoderStartTokenID is the ID of the start token for the decoder of an
	// encoder-decoder model.
	DecoderStartTokenID int
	// LengthPenalty is the exponential penalty to the length.
	// 1.0 means no penalty. Set to values < 1.0 in order to encourage the
	// model to generate shorter sequences, to a value > 1.0 in order to
	// encourage the model to produce longer sequences.
	LengthPenalty float64
	// EarlyStopping reports whether to stop the decoding search when at least
	// NumBeams sentences are finished per batch or not.
	EarlyStopping bool
	// BadWordsIDs is a list of token IDs that are not allowed to be generated.
	BadWordsIDs [][]int
	// When set to a positive value, generated n-grams of this size will
	// only occur once.
	NoRepeatNGramSize int
}

Config provides configuration options for the decoding search algorithm.

type DecodingStrategyFunc

type DecodingStrategyFunc func(tokensScores []mat.Matrix, resultSize int) []*ScoredToken

DecodingStrategyFunc returns the next tokens to be generated.

type PredictNextFunc

type PredictNextFunc func(decodingInputIDs [][]int, lastBeamIndices []int) []mat.Matrix

PredictNextFunc is a function that predicts the next token scores for a given input.

type ScoreProcessor

type ScoreProcessor func(scores mat.Matrix) mat.Matrix

ScoreProcessor is a function that takes a matrix of scores and returns an altered matrix of scores.

func ProcessScores

func ProcessScores(processors ...ScoreProcessor) ScoreProcessor

ProcessScores applies a list of score processors to a matrix of scores.

func TemperatureProcessor

func TemperatureProcessor(temperature float64) ScoreProcessor

TemperatureProcessor applies a temperature to a matrix of scores.

func TopKProcessor

func TopKProcessor(topK int, filterValue float64) ScoreProcessor

TopKProcessor applies a top-k filter to a matrix of scores.

func TopPProcessor

func TopPProcessor[T float.DType](topP, filterValue T, minSize int) ScoreProcessor

TopPProcessor applies a top-p filter to a matrix of scores. Note that when using beam decoding (with beam > 1) then minSize must be at least 2.

type ScoredToken

type ScoredToken struct {
	BeamIndex  int
	TokenIndex int
	Score      float64
}

ScoredToken associates a score to a token identified by its (beam-index, token-index) position.

func SelectNextMultinomial

func SelectNextMultinomial(tokensScores []mat.Matrix, resultSize int) []*ScoredToken

SelectNextMultinomial returns the next tokens to be generated.

func SelectNextTopK

func SelectNextTopK(tokensScores []mat.Matrix, resultSize int) []*ScoredToken

SelectNextTopK returns the next tokens to be generated.