vllm

Documentation

Index

• type Manager
  • func New(modelDir string, s3Client s3Client) *Manager
  • func (m *Manager) CreateNewModelOfGGUF(modelName string, spec *ollama.ModelSpec) error
  • func (m *Manager) DownloadAndCreateNewModel(ctx context.Context, modelName string, resp *mv1.GetBaseModelPathResponse) error
  • func (m *Manager) UpdateModelTemplateToLatest(modelName string) error

Types

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages the Ollama service.

TODO(kenji): Refactor this class once we completely switch to the one-odel-per-pod implementation where inference-manager-engine doesn't directly run vLLM or Ollama.

func New

func New(modelDir string, s3Client s3Client) *Manager

New returns a new Manager.

func (*Manager) CreateNewModelOfGGUF

func (m *Manager) CreateNewModelOfGGUF(modelName string, spec *ollama.ModelSpec) error

CreateNewModelOfGGUF creates a new model with the given name and spec that uses a GGUF model file.

func (*Manager) DownloadAndCreateNewModel

func (m *Manager) DownloadAndCreateNewModel(ctx context.Context, modelName string, resp *mv1.GetBaseModelPathResponse) error

DownloadAndCreateNewModel downloads the model from the given path and creates a new model.

func (*Manager) UpdateModelTemplateToLatest

func (m *Manager) UpdateModelTemplateToLatest(modelName string) error

UpdateModelTemplateToLatest updates the model template to the latest.