pipelines

package
v0.3.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 26, 2025 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func WithHypothesisTemplate added in v0.1.4 

func WithHypothesisTemplate(hypothesisTemplate string) pipelineBackends.PipelineOption[*ZeroShotClassificationPipeline]

WithHypothesisTemplate can be used to set the hypothesis template for classification.

func WithIgnoreLabels 

func WithIgnoreLabels(ignoreLabels []string) pipelineBackends.PipelineOption[*TokenClassificationPipeline]

func WithLabels added in v0.1.4 

func WithLabels(labels []string) pipelineBackends.PipelineOption[*ZeroShotClassificationPipeline]

WithLabels can be used to set the labels to classify the examples.

func WithMultiLabel added in v0.0.9 

func WithMultiLabel() pipelineBackends.PipelineOption[*TextClassificationPipeline]

func WithMultilabel added in v0.1.4 

func WithMultilabel(multilabel bool) pipelineBackends.PipelineOption[*ZeroShotClassificationPipeline]

WithMultilabel can be used to set whether the pipeline is multilabel.

func WithNormalization added in v0.1.1 

func WithNormalization() pipelineBackends.PipelineOption[*FeatureExtractionPipeline]

WithNormalization applies normalization to the mean pooled output of the feature pipeline.

func WithOutputName added in v0.1.4 

func WithOutputName(outputName string) pipelineBackends.PipelineOption[*FeatureExtractionPipeline]

WithOutputName if there are multiple outputs from the underlying model, which output should be returned. If not passed, the first output from the feature pipeline is returned.

func WithSigmoid added in v0.0.9 

func WithSigmoid() pipelineBackends.PipelineOption[*TextClassificationPipeline]

func WithSimpleAggregation 

func WithSimpleAggregation() pipelineBackends.PipelineOption[*TokenClassificationPipeline]

WithSimpleAggregation sets the aggregation strategy for the token labels to simple It reproduces simple aggregation from the huggingface implementation.

func WithSingleLabel added in v0.0.9 

func WithSingleLabel() pipelineBackends.PipelineOption[*TextClassificationPipeline]

func WithSoftmax added in v0.0.9 

func WithSoftmax() pipelineBackends.PipelineOption[*TextClassificationPipeline]

func WithoutAggregation 

func WithoutAggregation() pipelineBackends.PipelineOption[*TokenClassificationPipeline]

WithoutAggregation returns the token labels.

Types

type ClassificationOutput 

type ClassificationOutput struct {
	Label string
	Score float32
}

type Entity added in v0.0.5 

type Entity struct {
	Entity    string
	Score     float32
	Scores    []float32
	Index     int
	Word      string
	TokenID   uint32
	Start     uint
	End       uint
	IsSubword bool
}

type FeatureExtractionOutput 

type FeatureExtractionOutput struct {
	Embeddings [][]float32
}

func (*FeatureExtractionOutput) GetOutput added in v0.0.5 

func (t *FeatureExtractionOutput) GetOutput() []any

type FeatureExtractionPipeline 

type FeatureExtractionPipeline struct {
	*pipelineBackends.BasePipeline
	Normalization bool
	OutputName    string
	Output        pipelineBackends.InputOutputInfo
}

FeatureExtractionPipeline A feature extraction pipeline is a go version of https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/feature_extraction.py

func NewFeatureExtractionPipeline 

func NewFeatureExtractionPipeline(config pipelineBackends.PipelineConfig[*FeatureExtractionPipeline], s *options.Options, model *pipelineBackends.Model) (*FeatureExtractionPipeline, error)

NewFeatureExtractionPipeline init a feature extraction pipeline.

func (*FeatureExtractionPipeline) Forward added in v0.1.4 

func (p *FeatureExtractionPipeline) Forward(batch *pipelineBackends.PipelineBatch) error

Forward performs the forward inference of the feature extraction pipeline.

func (*FeatureExtractionPipeline) GetMetadata added in v0.1.4 

func (p *FeatureExtractionPipeline) GetMetadata() pipelineBackends.PipelineMetadata

GetMetadata returns metadata information about the pipeline, in particular: OutputInfo: names and dimensions of the output layer.

func (*FeatureExtractionPipeline) GetModel added in v0.3.0 

func (p *FeatureExtractionPipeline) GetModel() *pipelineBackends.Model

func (*FeatureExtractionPipeline) GetStats added in v0.1.4 

func (p *FeatureExtractionPipeline) GetStats() []string

GetStats returns the runtime statistics for the pipeline.

func (*FeatureExtractionPipeline) Postprocess 

func (p *FeatureExtractionPipeline) Postprocess(batch *pipelineBackends.PipelineBatch) (*FeatureExtractionOutput, error)

Postprocess parses the first output from the network similar to the transformers' implementation.

func (*FeatureExtractionPipeline) Preprocess added in v0.1.4 

func (p *FeatureExtractionPipeline) Preprocess(batch *pipelineBackends.PipelineBatch, inputs []string) error

Preprocess tokenizes the input strings.

func (*FeatureExtractionPipeline) Run 

func (p *FeatureExtractionPipeline) Run(inputs []string) (pipelineBackends.PipelineBatchOutput, error)

Run the pipeline on a batch of strings.

func (*FeatureExtractionPipeline) RunPipeline added in v0.0.6 

func (p *FeatureExtractionPipeline) RunPipeline(inputs []string) (*FeatureExtractionOutput, error)

RunPipeline is like Run, but returns the concrete feature extraction output type rather than the interface.

func (*FeatureExtractionPipeline) Validate added in v0.0.5 

func (p *FeatureExtractionPipeline) Validate() error

Validate checks that the pipeline is valid.

type TextClassificationOutput 

type TextClassificationOutput struct {
	ClassificationOutputs [][]ClassificationOutput
}

func (*TextClassificationOutput) GetOutput added in v0.0.5 

func (t *TextClassificationOutput) GetOutput() []any

type TextClassificationPipeline 

type TextClassificationPipeline struct {
	*pipelineBackends.BasePipeline
	IDLabelMap              map[int]string
	AggregationFunctionName string
	ProblemType             string
}

func NewTextClassificationPipeline 

func NewTextClassificationPipeline(config pipelineBackends.PipelineConfig[*TextClassificationPipeline], s *options.Options, model *pipelineBackends.Model) (*TextClassificationPipeline, error)

NewTextClassificationPipeline initializes a new text classification pipeline.

func (*TextClassificationPipeline) Forward 

func (p *TextClassificationPipeline) Forward(batch *pipelineBackends.PipelineBatch) error

func (*TextClassificationPipeline) GetMetadata added in v0.1.4 

func (p *TextClassificationPipeline) GetMetadata() pipelineBackends.PipelineMetadata

GetMetadata returns metadata information about the pipeline, in particular: OutputInfo: names and dimensions of the output layer used for text classification.

func (*TextClassificationPipeline) GetModel added in v0.3.0 

func (p *TextClassificationPipeline) GetModel() *pipelineBackends.Model

func (*TextClassificationPipeline) GetStats added in v0.1.4 

func (p *TextClassificationPipeline) GetStats() []string

GetStats returns the runtime statistics for the pipeline.

func (*TextClassificationPipeline) Postprocess 

func (p *TextClassificationPipeline) Postprocess(batch *pipelineBackends.PipelineBatch) (*TextClassificationOutput, error)

func (*TextClassificationPipeline) Preprocess added in v0.1.4 

func (p *TextClassificationPipeline) Preprocess(batch *pipelineBackends.PipelineBatch, inputs []string) error

Preprocess tokenizes the input strings.

func (*TextClassificationPipeline) Run 

func (p *TextClassificationPipeline) Run(inputs []string) (pipelineBackends.PipelineBatchOutput, error)

Run the pipeline on a string batch.

func (*TextClassificationPipeline) RunPipeline added in v0.0.6 

func (p *TextClassificationPipeline) RunPipeline(inputs []string) (*TextClassificationOutput, error)

func (*TextClassificationPipeline) Validate added in v0.0.5 

func (p *TextClassificationPipeline) Validate() error

Validate checks that the pipeline is valid.

type TextClassificationPipelineConfig 

type TextClassificationPipelineConfig struct {
	IDLabelMap map[int]string `json:"id2label"`
}

type TokenClassificationOutput 

type TokenClassificationOutput struct {
	Entities [][]Entity
}

func (*TokenClassificationOutput) GetOutput added in v0.0.5 

func (t *TokenClassificationOutput) GetOutput() []any

type TokenClassificationPipeline 

type TokenClassificationPipeline struct {
	*pipelineBackends.BasePipeline
	IDLabelMap          map[int]string
	AggregationStrategy string
	IgnoreLabels        []string
}

TokenClassificationPipeline is a go version of huggingface tokenClassificationPipeline. https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/token_classification.py

func NewTokenClassificationPipeline 

func NewTokenClassificationPipeline(config pipelineBackends.PipelineConfig[*TokenClassificationPipeline], s *options.Options, model *pipelineBackends.Model) (*TokenClassificationPipeline, error)

NewTokenClassificationPipeline Initializes a feature extraction pipeline.

func (*TokenClassificationPipeline) Aggregate 

func (p *TokenClassificationPipeline) Aggregate(input pipelineBackends.TokenizedInput, preEntities []Entity) ([]Entity, error)

func (*TokenClassificationPipeline) Forward added in v0.1.4 

func (p *TokenClassificationPipeline) Forward(batch *pipelineBackends.PipelineBatch) error

Forward performs the forward inference of the pipeline.

func (*TokenClassificationPipeline) GatherPreEntities 

func (p *TokenClassificationPipeline) GatherPreEntities(input pipelineBackends.TokenizedInput, output [][]float32) []Entity

GatherPreEntities from batch of logits to list of pre-aggregated outputs.

func (*TokenClassificationPipeline) GetMetadata added in v0.1.4 

func (p *TokenClassificationPipeline) GetMetadata() pipelineBackends.PipelineMetadata

GetMetadata returns metadata information about the pipeline, in particular: OutputInfo: names and dimensions of the output layer used for token classification.

func (*TokenClassificationPipeline) GetModel added in v0.3.0 

func (p *TokenClassificationPipeline) GetModel() *pipelineBackends.Model

func (*TokenClassificationPipeline) GetStats added in v0.1.4 

func (p *TokenClassificationPipeline) GetStats() []string

GetStats returns the runtime statistics for the pipeline.

func (*TokenClassificationPipeline) GroupEntities 

func (p *TokenClassificationPipeline) GroupEntities(entities []Entity) ([]Entity, error)

GroupEntities group together adjacent tokens with the same entity predicted.

func (*TokenClassificationPipeline) Postprocess 

func (p *TokenClassificationPipeline) Postprocess(batch *pipelineBackends.PipelineBatch) (*TokenClassificationOutput, error)

Postprocess function for a token classification pipeline.

func (*TokenClassificationPipeline) Preprocess added in v0.1.4 

func (p *TokenClassificationPipeline) Preprocess(batch *pipelineBackends.PipelineBatch, inputs []string) error

Preprocess tokenizes the input strings.

func (*TokenClassificationPipeline) Run 

func (p *TokenClassificationPipeline) Run(inputs []string) (pipelineBackends.PipelineBatchOutput, error)

Run the pipeline on a string batch.

func (*TokenClassificationPipeline) RunPipeline added in v0.0.6 

func (p *TokenClassificationPipeline) RunPipeline(inputs []string) (*TokenClassificationOutput, error)

RunPipeline is like Run but returns the concrete type rather than the interface.

func (*TokenClassificationPipeline) Validate added in v0.0.5 

func (p *TokenClassificationPipeline) Validate() error

Validate checks that the pipeline is valid.

type TokenClassificationPipelineConfig 

type TokenClassificationPipelineConfig struct {
	IDLabelMap map[int]string `json:"id2label"`
}

type ZeroShotClassificationOutput added in v0.1.4 

type ZeroShotClassificationOutput struct {
	Sequence     string
	SortedValues []struct {
		Key   string
		Value float64
	}
}

type ZeroShotClassificationPipeline added in v0.1.4 

type ZeroShotClassificationPipeline struct {
	*pipelineBackends.BasePipeline
	IDLabelMap         map[int]string
	Sequences          []string
	Labels             []string
	HypothesisTemplate string
	Multilabel         bool
	// contains filtered or unexported fields
}

func NewZeroShotClassificationPipeline added in v0.1.4 

func NewZeroShotClassificationPipeline(config pipelineBackends.PipelineConfig[*ZeroShotClassificationPipeline], s *options.Options, model *pipelineBackends.Model) (*ZeroShotClassificationPipeline, error)

NewZeroShotClassificationPipeline create new Zero Shot Classification Pipeline.

func (*ZeroShotClassificationPipeline) Forward added in v0.1.4 

func (p *ZeroShotClassificationPipeline) Forward(batch *pipelineBackends.PipelineBatch) error

func (*ZeroShotClassificationPipeline) GetMetadata added in v0.1.4 

func (p *ZeroShotClassificationPipeline) GetMetadata() pipelineBackends.PipelineMetadata

func (*ZeroShotClassificationPipeline) GetModel added in v0.3.0 

func (p *ZeroShotClassificationPipeline) GetModel() *pipelineBackends.Model

func (*ZeroShotClassificationPipeline) GetStats added in v0.1.4 

func (p *ZeroShotClassificationPipeline) GetStats() []string

func (*ZeroShotClassificationPipeline) Postprocess added in v0.1.4 

func (p *ZeroShotClassificationPipeline) Postprocess(outputTensors [][][]float32, labels []string, sequences []string) (*ZeroShotOutput, error)

func (*ZeroShotClassificationPipeline) Preprocess added in v0.1.4 

func (p *ZeroShotClassificationPipeline) Preprocess(batch *pipelineBackends.PipelineBatch, inputs []string) error

func (*ZeroShotClassificationPipeline) Run added in v0.1.4 

func (p *ZeroShotClassificationPipeline) Run(inputs []string) (pipelineBackends.PipelineBatchOutput, error)

func (*ZeroShotClassificationPipeline) RunPipeline added in v0.1.4 

func (p *ZeroShotClassificationPipeline) RunPipeline(inputs []string) (*ZeroShotOutput, error)

func (*ZeroShotClassificationPipeline) Validate added in v0.1.4 

func (p *ZeroShotClassificationPipeline) Validate() error

type ZeroShotClassificationPipelineConfig added in v0.1.4 

type ZeroShotClassificationPipelineConfig struct {
	IDLabelMap map[int]string `json:"id2label"`
}

type ZeroShotOutput added in v0.1.4 

type ZeroShotOutput struct {
	ClassificationOutputs []ZeroShotClassificationOutput
}

func (*ZeroShotOutput) GetOutput added in v0.1.4 

func (t *ZeroShotOutput) GetOutput() []any

GetOutput converts raw output to readable output.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.