base

Documentation

Index

• type Base
  • func (llm *Base) AudioTranscription(*pb.TranscriptRequest) (schema.Result, error)
  • func (llm *Base) Busy() bool
  • func (llm *Base) Embeddings(opts *pb.PredictOptions) ([]float32, error)
  • func (llm *Base) GenerateImage(*pb.GenerateImageRequest) error
  • func (llm *Base) Load(opts *pb.ModelOptions) error
  • func (llm *Base) Lock()
  • func (llm *Base) Locking() bool
  • func (llm *Base) Predict(opts *pb.PredictOptions) (string, error)
  • func (llm *Base) PredictStream(opts *pb.PredictOptions, results chan string) error
  • func (llm *Base) Status() (pb.StatusResponse, error)
  • func (llm *Base) TTS(*pb.TTSRequest) error
  • func (llm *Base) TokenizeString(opts *pb.PredictOptions) (pb.TokenizationResponse, error)
  • func (llm *Base) Unlock()
• type SingleThread
  • func (llm *SingleThread) Busy() bool
  • func (llm *SingleThread) Lock()
  • func (llm *SingleThread) Locking() bool
  • func (llm *SingleThread) Status() (pb.StatusResponse, error)
  • func (llm *SingleThread) Unlock()

Types

type Base

type Base struct {
}

Base is a base class for all backends to implement Note: the backends that does not support multiple requests should use SingleThread instead

func (*Base) AudioTranscription

func (llm *Base) AudioTranscription(*pb.TranscriptRequest) (schema.Result, error)

func (*Base) Busy

func (llm *Base) Busy() bool

func (*Base) Embeddings

func (llm *Base) Embeddings(opts *pb.PredictOptions) ([]float32, error)

func (*Base) GenerateImage

func (llm *Base) GenerateImage(*pb.GenerateImageRequest) error

func (*Base) Load

func (llm *Base) Load(opts *pb.ModelOptions) error

func (*Base) Lock

func (llm *Base) Lock()

func (*Base) Locking

func (llm *Base) Locking() bool

func (*Base) Predict

func (llm *Base) Predict(opts *pb.PredictOptions) (string, error)

func (*Base) PredictStream

func (llm *Base) PredictStream(opts *pb.PredictOptions, results chan string) error

func (*Base) Status

func (llm *Base) Status() (pb.StatusResponse, error)

backends may wish to call this to capture the gopsutil info, then enhance with additional memory usage details?

func (*Base) TTS

func (llm *Base) TTS(*pb.TTSRequest) error

func (*Base) TokenizeString

func (llm *Base) TokenizeString(opts *pb.PredictOptions) (pb.TokenizationResponse, error)

func (*Base) Unlock

func (llm *Base) Unlock()

type SingleThread

type SingleThread struct {
	Base
	// contains filtered or unexported fields
}

SingleThread are backends that does not support multiple requests. There will be only one request being served at the time. This is useful for models that are not thread safe and cannot run multiple requests at the same time.

func (*SingleThread) Busy

func (llm *SingleThread) Busy() bool

func (*SingleThread) Lock

func (llm *SingleThread) Lock()

func (*SingleThread) Locking

func (llm *SingleThread) Locking() bool

Locking returns true if the backend needs to lock resources

func (*SingleThread) Status

func (llm *SingleThread) Status() (pb.StatusResponse, error)

backends may wish to call this to capture the gopsutil info, then enhance with additional memory usage details?

func (*SingleThread) Unlock

func (llm *SingleThread) Unlock()