ollama

README

---
title: "Ollama"
lang: "en-US"
draft: false
description: "Learn about how to set up a VDP Ollama component https://github.com/instill-ai/instill-core"
---

The Ollama component is an AI component that allows users to connect the AI models served with the Ollama library.
It can carry out the following tasks:

- [Text Generation Chat](#text-generation-chat)
- [Text Embeddings](#text-embeddings)



## Release Stage

`Alpha`



## Configuration

The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/ollama/v0/config/definition.json).




## Setup


| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Endpoint (required) | `endpoint` | string | Fill in your Ollama hosting endpoint. ### WARNING ###: As of 2024-07-26, the Ollama component does not support authentication methods. To prevent unauthorized access to your Ollama serving resources, please implement additional security measures such as IP whitelisting. |
| Model Auto-Pull (required) | `auto-pull` | boolean | Automatically pull the requested models from the Ollama server if the model is not found in the local cache. |




## Supported Tasks

### Text Generation Chat

Provide text outputs in response to text/image inputs.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_TEXT_GENERATION_CHAT` |
| Model Name (required) | `model` | string | The OSS model to be used, check https://ollama.com/library for list of models available |
| Prompt (required) | `prompt` | string | The prompt text |
| System message | `system-message` | string | The system message helps set the behavior of the assistant. For example, you can modify the personality of the assistant or provide specific instructions about how it should behave throughout the conversation. By default, the model’s behavior is set using a generic message as "You are a helpful assistant." |
| Prompt Images | `prompt-images` | array[string] | The prompt images |
| Chat history | `chat-history` | array[object] | Incorporate external chat history, specifically previous messages within the conversation. Please note that System Message will be ignored and will not have any effect when this field is populated. Each message should adhere to the format: : \{"role": "The message role, i.e. 'system', 'user' or 'assistant'", "content": "message content"\}. |
| Seed | `seed` | integer | The seed |
| Temperature | `temperature` | number | The temperature for sampling |
| Top K | `top-k` | integer | Top k for sampling |
| Max new tokens | `max-new-tokens` | integer | The maximum number of tokens for model to generate |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Text | `text` | string | Model Output |






### Text Embeddings

Turn text into a vector of numbers that capture its meaning, unlocking use cases like semantic search.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_TEXT_EMBEDDINGS` |
| Model Name (required) | `model` | string | The OSS model to be used, check https://ollama.com/library for list of models available |
| Text (required) | `text` | string | The text |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Embedding | `embedding` | array[number] | Embedding of the input text |

Documentation

Index

• Constants
• func Init(bc base.Component) *component
• type ChatMessage
• type ChatRequest
• type ChatResponse
• type EmbedRequest
• type EmbedResponse
• type ListLocalModelsRequest
• type ListLocalModelsResponse
• type MultiModalContent
• type OllamaChatMessage
• type OllamaClient
  • func NewClient(endpoint string, autoPull bool, logger *zap.Logger) *OllamaClient
  • func (c *OllamaClient) Chat(request ChatRequest) (ChatResponse, error)
  • func (c *OllamaClient) CheckModelAvailability(modelName string) bool
  • func (c *OllamaClient) Embed(request EmbedRequest) (EmbedResponse, error)
  • func (c *OllamaClient) IsAutoPull() bool
  • func (c *OllamaClient) Pull(modelName string) error
• type OllamaClientInterface
• type OllamaModelInfo
• type OllamaOptions
• type OllamaSetup
• type PullModelRequest
• type PullModelResponse
• type TaskTextEmbeddingsInput
• type TaskTextEmbeddingsOutput
• type TaskTextGenerationChatInput
• type TaskTextGenerationChatOuput
• type URL

Constants

const (
	TaskTextGenerationChat = "TASK_TEXT_GENERATION_CHAT"
	TaskTextEmbeddings     = "TASK_TEXT_EMBEDDINGS"
)

Functions

func Init

func Init(bc base.Component) *component

Types

type ChatMessage

type ChatMessage struct {
	Role    string              `json:"role"`
	Content []MultiModalContent `json:"content"`
}

type ChatRequest

type ChatRequest struct {
	Model    string              `json:"model"`
	Messages []OllamaChatMessage `json:"messages"`
	Stream   bool                `json:"stream"`
	Options  OllamaOptions       `json:"options"`
}

type ChatResponse

type ChatResponse struct {
	Model              string            `json:"model"`
	CreatedAt          string            `json:"created_at"`
	Message            OllamaChatMessage `json:"message"`
	Done               bool              `json:"done"`
	DoneReason         string            `json:"done_reason"`
	TotalDuration      int               `json:"total_duration"`
	LoadDuration       int               `json:"load_duration"`
	PromptEvalCount    int               `json:"prompt_eval_count"`
	PromptEvalDuration int               `json:"prompt_eval_duration"`
	EvalCount          int               `json:"eval_count"`
	EvalDuration       int               `json:"eval_duration"`
}

type EmbedRequest

type EmbedRequest struct {
	Model  string `json:"model"`
	Prompt string `json:"prompt"`
}

type EmbedResponse

type EmbedResponse struct {
	Embedding []float32 `json:"embedding"`
}

type ListLocalModelsRequest

type ListLocalModelsRequest struct {
}

type ListLocalModelsResponse

type ListLocalModelsResponse struct {
	Models []OllamaModelInfo `json:"models"`
}

type MultiModalContent

type MultiModalContent struct {
	ImageURL URL    `json:"image-url"`
	Text     string `json:"text"`
	Type     string `json:"type"`
}

type OllamaChatMessage

type OllamaChatMessage struct {
	Role    string   `json:"role"`
	Content string   `json:"content"`
	Images  []string `json:"images,omitempty"`
}

type OllamaClient

type OllamaClient struct {
	// contains filtered or unexported fields
}

func NewClient

func NewClient(endpoint string, autoPull bool, logger *zap.Logger) *OllamaClient

func (*OllamaClient) Chat

func (c *OllamaClient) Chat(request ChatRequest) (ChatResponse, error)

func (*OllamaClient) CheckModelAvailability

func (c *OllamaClient) CheckModelAvailability(modelName string) bool

func (*OllamaClient) Embed

func (c *OllamaClient) Embed(request EmbedRequest) (EmbedResponse, error)

func (*OllamaClient) IsAutoPull

func (c *OllamaClient) IsAutoPull() bool

func (*OllamaClient) Pull

func (c *OllamaClient) Pull(modelName string) error

type OllamaClientInterface

type OllamaClientInterface interface {
	Chat(ChatRequest) (ChatResponse, error)
	Embed(EmbedRequest) (EmbedResponse, error)
	IsAutoPull() bool
}

type OllamaModelInfo

type OllamaModelInfo struct {
	Name       string `json:"name"`
	ModifiedAt string `json:"modified_at"`
	Size       int    `json:"size"`
	Dijest     string `json:"digest"`
	Details    struct {
		Format            string `json:"format"`
		Family            string `json:"family"`
		Families          string `json:"families"`
		ParameterSize     string `json:"parameter_size"`
		QuantizationLevel string `json:"quantization_level"`
	} `json:"details"`
}

type OllamaOptions

type OllamaOptions struct {
	Temperature float32 `json:"temperature,omitempty"`
	TopK        int     `json:"top_k,omitempty"`
	Seed        int     `json:"seed,omitempty"`
}

type OllamaSetup

type OllamaSetup struct {
	AutoPull bool   `json:"auto-pull"`
	Endpoint string `json:"endpoint"`
}

type PullModelRequest

type PullModelRequest struct {
	Name   string `json:"name"`
	Stream bool   `json:"stream"`
}

type PullModelResponse

type PullModelResponse struct {
}

type TaskTextEmbeddingsInput

type TaskTextEmbeddingsInput struct {
	Text  string `json:"text"`
	Model string `json:"model"`
}

type TaskTextEmbeddingsOutput

type TaskTextEmbeddingsOutput struct {
	Embedding []float32 `json:"embedding"`
}

type TaskTextGenerationChatInput

type TaskTextGenerationChatInput struct {
	ChatHistory  []ChatMessage `json:"chat-history"`
	MaxNewTokens int           `json:"max-new-tokens"`
	Model        string        `json:"model"`
	Prompt       string        `json:"prompt"`
	PromptImages []string      `json:"prompt-images"`
	Seed         int           `json:"seed"`
	SystemMsg    string        `json:"system-message"`
	Temperature  float32       `json:"temperature"`
	TopK         int           `json:"top-k"`
}

type TaskTextGenerationChatOuput

type TaskTextGenerationChatOuput struct {
	Text string `json:"text"`
}

type URL

type URL struct {
	URL string `json:"url"`
}