instill

README

---
title: "Instill Model"
lang: "en-US"
draft: false
description: "Learn about how to set up a VDP Instill Model component https://github.com/instill-ai/instill-core"
---

The Instill Model component is an AI component that allows users to connect the AI models served on the Instill Model Platform.
It can carry out the following tasks:

- [Classification](#classification)
- [Instance Segmentation](#instance-segmentation)
- [Keypoint](#keypoint)
- [Detection](#detection)
- [Ocr](#ocr)
- [Semantic Segmentation](#semantic-segmentation)
- [Text Generation](#text-generation)
- [Text Generation Chat](#text-generation-chat)
- [Text To Image](#text-to-image)
- [Visual Question Answering](#visual-question-answering)
- [Chat](#chat)



## Release Stage

`Alpha`



## Configuration

The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/ai/instill/v0/config/definition.json).





## Supported Tasks

### Classification

Classify images into predefined categories.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CLASSIFICATION` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Image (required) | `image-base64` | string | Image base64 |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Category | `category` | string | The predicted category of the input. |
| Score | `score` | number | The confidence score of the predicted category of the input. |






### Instance Segmentation

Detect, localize and delineate multiple objects in images.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_INSTANCE_SEGMENTATION` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Image (required) | `image-base64` | string | Image base64 |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Objects | `objects` | array[object] | A list of detected instance bounding boxes. |






### Keypoint

Detect and localize multiple keypoints of objects in images.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_KEYPOINT` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Image (required) | `image-base64` | string | Image base64 |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Objects | `objects` | array[object] | A list of keypoint objects, a keypoint object includes all the pre-defined keypoints of a detected object. |






### Detection

Detect and localize multiple objects in images.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DETECTION` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Image (required) | `image-base64` | string | Image base64 |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Objects | `objects` | array[object] | A list of detected objects. |






### Ocr

Detect and recognize text in images.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_OCR` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Image (required) | `image-base64` | string | Image base64 |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Objects | `objects` | array[object] | A list of detected bounding boxes. |






### Semantic Segmentation

Classify image pixels into predefined categories.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_SEMANTIC_SEGMENTATION` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Image (required) | `image-base64` | string | Image base64 |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Stuffs | `stuffs` | array[object] | A list of RLE binary masks. |






### Text Generation

Generate texts from input text prompts.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_TEXT_GENERATION` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Prompt (required) | `prompt` | string | The prompt text |
| System message | `system-message` | string | The system message helps set the behavior of the assistant. For example, you can modify the personality of the assistant or provide specific instructions about how it should behave throughout the conversation. By default, the model’s behavior is using a generic message as "You are a helpful assistant." |
| Seed | `seed` | integer | The seed |
| Temperature | `temperature` | number | The temperature for sampling |
| Max new tokens | `max-new-tokens` | integer | The maximum number of tokens for model to generate |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Text | `text` | string | Text |






### Text Generation Chat

Generate texts from input text prompts and chat history.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_TEXT_GENERATION_CHAT` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Prompt (required) | `prompt` | string | The prompt text |
| System message | `system-message` | string | The system message helps set the behavior of the assistant. For example, you can modify the personality of the assistant or provide specific instructions about how it should behave throughout the conversation. By default, the model’s behavior is using a generic message as "You are a helpful assistant." |
| Prompt Images | `prompt-images` | array[string] | The prompt images |
| Chat history | `chat-history` | array[object] | Incorporate external chat history, specifically previous messages within the conversation. Please note that System Message will be ignored and will not have any effect when this field is populated. Each message should adhere to the format: : \{"role": "The message role, i.e. 'system', 'user' or 'assistant'", "content": "message content"\}. |
| Seed | `seed` | integer | The seed |
| Temperature | `temperature` | number | The temperature for sampling |
| Max new tokens | `max-new-tokens` | integer | The maximum number of tokens for model to generate |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Text | `text` | string | Text |






### Text To Image

Generate images from input text prompts.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_TEXT_TO_IMAGE` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Prompt (required) | `prompt` | string | The prompt text |
| Samples | `samples` | integer | The number of generated samples, default is 1 |
| Seed | `seed` | integer | The seed, default is 0 |
| Aspect ratio | `negative-prompt` | string | Keywords of what you do not wish to see in the output image. |
| Aspect ratio | `aspect-ratio` | string | Controls the aspect ratio of the generated image. Defaults to 1:1. |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Images | `images` | array[string] | Images |






### Visual Question Answering

Answer questions based on a prompt and an image.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_VISUAL_QUESTION_ANSWERING` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Prompt (required) | `prompt` | string | The prompt text |
| System message | `system-message` | string | The system message helps set the behavior of the assistant. For example, you can modify the personality of the assistant or provide specific instructions about how it should behave throughout the conversation. By default, the model’s behavior is using a generic message as "You are a helpful assistant." |
| Prompt Images | `prompt-images` | array[string] | The prompt images |
| Chat history | `chat-history` | array[object] | Incorporate external chat history, specifically previous messages within the conversation. Please note that System Message will be ignored and will not have any effect when this field is populated. Each message should adhere to the format: : \{"role": "The message role, i.e. 'system', 'user' or 'assistant'", "content": "message content"\}. |
| Seed | `seed` | integer | The seed |
| Temperature | `temperature` | number | The temperature for sampling |
| Max new tokens | `max-new-tokens` | integer | The maximum number of tokens for model to generate |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Text | `text` | string | Text |






### Chat

Generate texts from input text prompts and chat history.


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CHAT` |
| Model Name (required) | `model-name` | string | The Instill Model model to be used. |
| Prompt (required) | `prompt` | string | The prompt text |
| System message | `system-message` | string | The system message helps set the behavior of the assistant. For example, you can modify the personality of the assistant or provide specific instructions about how it should behave throughout the conversation. By default, the model’s behavior is using a generic message as "You are a helpful assistant." |
| Prompt Images | `prompt-images` | array[string] | The prompt images |
| Chat history | `chat-history` | array[object] | Incorporate external chat history, specifically previous messages within the conversation. Please note that System Message will be ignored and will not have any effect when this field is populated. Each message should adhere to the format: : \{"role": "The message role, i.e. 'system', 'user' or 'assistant'", "content": "message content"\}. |
| Seed | `seed` | integer | The seed |
| Temperature | `temperature` | number | The temperature for sampling |
| Max new tokens | `max-new-tokens` | integer | The maximum number of tokens for model to generate |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Text | `text` | string | Text |

Documentation

Index

• func Init(bc base.Component) *component
• type ChatParameter
• type ChatRequestData
• type Content
• type Message
• type ModelsResp
• type RequestWrapper
• type TextCompletionRequestData
• type TextCompletionRequestParameter
• type TextGenerationInput
• type TextToImageRequestData
• type TextToImageRequestParameter
• type VisionRequestData

Functions

func Init

func Init(bc base.Component) *component

Types

type ChatParameter

type ChatParameter struct {
	MaxTokens   int     `json:"max-tokens,omitempty"`
	Seed        int     `json:"seed,omitempty"`
	N           int     `json:"n,omitempty"`
	Temperature float32 `json:"temperature,omitempty"`
	TopP        int     `json:"top-p,omitempty"`
}

type ChatRequestData

type ChatRequestData struct {
	Messages []Message `json:"messages,omitempty"`
}

type Content

type Content struct {
	Text        string `json:"text,omitempty"`
	ImageBase64 string `json:"image-base64,omitempty"`
	Type        string `json:"type,omitempty"`
}

type Message

type Message struct {
	Content []Content `json:"content,omitempty"`
	Role    string    `json:"role,omitempty"`
}

type ModelsResp

type ModelsResp struct {
	Models []struct {
		Name string `json:"name"`
		Task string `json:"task"`
	} `json:"models"`
}

type RequestWrapper

type RequestWrapper struct {
	Data      any `json:"data,omitempty"`
	Parameter any `json:"parameter,omitempty"`
}

type TextCompletionRequestData

type TextCompletionRequestData struct {
	Prompt        string `json:"prompt"`
	SystemMessage string `json:"system-message,omitempty"`
}

type TextCompletionRequestParameter

type TextCompletionRequestParameter struct {
	MaxTokens   int     `json:"max-tokens,omitempty"`
	Seed        int     `json:"seed,omitempty"`
	N           int     `json:"n,omitempty"`
	Temperature float32 `json:"temperature,omitempty"`
	TopP        int     `json:"top-p,omitempty"`
}

type TextGenerationInput

type TextGenerationInput struct {
	Prompt        string   `json:"prompt"`
	SystemMessage *string  `json:"system-message,omitempty"`
	PromptImages  []string `json:"prompt-images,omitempty"`

	// Note: We're currently sharing the same struct in the OpenAI component,
	// but this will be moved to the standardized format later.
	ChatHistory []*openai.TextMessage `json:"chat-history,omitempty"`
}

type TextToImageRequestData

type TextToImageRequestData struct {
	Prompt string `json:"prompt"`
}

type TextToImageRequestParameter

type TextToImageRequestParameter struct {
	AspectRatio    string `json:"aspect-ratio,omitempty"`
	NegativePrompt string `json:"negative-prompt,omitempty"`
	N              int    `json:"n,omitempty"`
	Seed           int    `json:"seed,omitempty"`
}

type VisionRequestData

type VisionRequestData struct {
	ImageBase64 string `json:"image-base64"`
	Type        string `json:"type"`
}