milvus

README

---
title: "Milvus"
lang: "en-US"
draft: false
description: "Learn about how to set up a VDP Milvus component https://github.com/instill-ai/instill-core"
---

The Milvus component is a data component that allows users to build and search vector datasets.
It can carry out the following tasks:

- [Vector Search](#vector-search)
- [Upsert](#upsert)
- [Batch Upsert](#batch-upsert)
- [Delete](#delete)
- [Create Collection](#create-collection)
- [Drop Collection](#drop-collection)
- [Create Partition](#create-partition)
- [Drop Partition](#drop-partition)
- [Create Index](#create-index)
- [Drop Index](#drop-index)



## Release Stage

`Alpha`



## Configuration

The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/milvus/v0/config/definition.json).




## Setup




In order to communicate with Milvus, the following connection details need to be
provided. You may specify them directly in a pipeline recipe as key-value pairs
withing the component's `setup` block, or you can create a **Connection** from
the [**Integration Settings**](https://www.instill.tech/docs/vdp/integration)
page and reference the whole `setup` as `setup:
${connection.<my-connection-id>}`.

| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Milvus URL Endpoint (required) | `url` | string | Fill in your Milvus public URL endpoint with port number, e.g http://3.25.202.142:19530 |
| Milvus Username (required) | `username` | string | Fill in your Milvus username |
| Milvus Password (required) | `password` | string | Fill in your Milvus password |




## Supported Tasks

### Vector Search

Perform a vector search on a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_VECTOR_SEARCH` |
| Collection Name (required) | `collection-name` | string | The name of the collection to perform vector search on |
| Partition Name | `partition-name` | string | The name of the partition to vector search the data from |
| Vector (required) | `vector` | array[number] | An array of dimensions for the vector search |
| Vector Field (required) | `vector-field` | string | The name of the field to perform vector search on |
| Fields | `fields` | array[string] | The fields to return in the data. If empty then all fields will be returned |
| Limit (required) | `limit` | integer | The limit of the data to return |
| Filter | `filter` | string | The properties filter to be applied to the data with milvus scalar filter, please refer to [filter-search](https://milvus.io/docs/single-vector-search.md#Filtered-search) |
| Offset | `offset` | integer | The offset of the data to return |
| Grouping Field | `grouping-field` | string | The name of the field to group the data by, please refer to [Grouping-search](https://milvus.io/docs/single-vector-search.md#Grouping-search) |
| Search Parameters | `search-params` | object | The search parameters to be applied to the data with milvus search parameters, please refer to [Search-parameters](https://milvus.io/docs/single-vector-search.md#Search-parameters) |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Result | `result` | object | Result of the vector search operation |
| Status | `status` | string | Vector search status |






### Upsert

Insert a vector data into a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_UPSERT` |
| Collection Name (required) | `collection-name` | string | The name of the collection to upsert the data into |
| Partition Name | `partition-name` | string | The name of the partition to upsert the data from. If empty then default partition will be used |
| Data (required) | `data` | object | The data |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Upsert status |






### Batch Upsert

Insert a batch of vector data into a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_BATCH_UPSERT` |
| Collection Name (required) | `collection-name` | string | The name of the collection to upsert the data into |
| Partition Name | `partition-name` | string | The name of the partition to upsert the data from. If empty then default partition will be used |
| Array Data (required) | `array-data` | array[object] | The data |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Batch upsert status |






### Delete

Delete vector data from a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DELETE` |
| Collection Name (required) | `collection-name` | string | The name of the collection to delete the data from |
| Partition Name | `partition-name` | string | The name of the partition to delete the data from. If empty then default partition will be used |
| Filter | `filter` | string | The properties filter to be applied to the data with milvus scalar filter, please refer to [Filtered-search](https://milvus.io/docs/single-vector-search.md#Filtered-search) |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Delete status |






### Create Collection

Create a collectio, please refer to [Create-Collection](https://milvus.io/docs/manage-collections.md#Create-Collection)


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CREATE_COLLECTION` |
| Collection Name (required) | `collection-name` | string | The name of the collection to create |
| Dimension (required) | `dimension` | integer | The dimension of the collection |
| ID Type | `id-type` | string | The type of the id |
| Schema | `schema` | object | The schema of the collection |
| Auto ID | `auto-id` | boolean | Whether to auto generate id |
| Metric Type | `metric-type` | string | The metric type of the collection |
| Index Parameters | `index-params` | array[object] | The index parameters to be applied to the collection with milvus index parameters, please refer to [Create](https://milvus.io/api-reference/restful/v2.4.x/v2/Collection%20(v2)/Create.md) |
| Parameters | `params` | object | The parameters to be applied to the collection with milvus parameters, please refer to [Parameters](https://milvus.io/docs/single-vector-search.md#Parameters) |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Create collection status |






### Drop Collection

Drop a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DROP_COLLECTION` |
| Collection Name (required) | `collection-name` | string | The name of the collection to drop |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Drop collection status |






### Create Partition

Create a partition in a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CREATE_PARTITION` |
| Collection Name (required) | `collection-name` | string | The name of the collection to create the partition in |
| Partition Name (required) | `partition-name` | string | The name of the partition to create |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Create partition status |






### Drop Partition

Drop a partition from a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DROP_PARTITION` |
| Collection Name (required) | `collection-name` | string | The name of the collection to drop the partition from |
| Partition Name (required) | `partition-name` | string | The name of the partition to drop |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Drop partition status |






### Create Index

Create an index in a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CREATE_INDEX` |
| Collection Name (required) | `collection-name` | string | The name of the collection to create the index in |
| Index Parameters (required) | `index-params` | object | The index parameters to be applied to the collection with milvus index parameters, please refer to [Create](https://milvus.io/api-reference/restful/v2.4.x/v2/Index%20(v2)/Create.md) |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Create index status |






### Drop Index

Drop an index from a collection


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DROP_INDEX` |
| Collection Name (required) | `collection-name` | string | The name of the collection to drop the index from |
| Index Name (required) | `index-name` | string | The name of the index to drop |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Drop index status |

Documentation

Index

• Constants
• func Init(bc base.Component) *component
• type BatchUpsertInput
• type BatchUpsertOutput
• type CreateCollectionInput
• type CreateCollectionOutput
• type CreateCollectionReq
• type CreateCollectionResp
• type CreateIndexInput
• type CreateIndexOutput
• type CreateIndexReq
• type CreateIndexResp
• type CreatePartitionInput
• type CreatePartitionOutput
• type CreatePartitionReq
• type CreatePartitionResp
• type Data
• type DataDescribe
• type DeleteInput
• type DeleteOutput
• type DeleteReq
• type DeleteResp
• type DescribeCollection
• type DescribeCollectionReq
• type DescribeCollectionResp
• type DropCollectionInput
• type DropCollectionOutput
• type DropCollectionReq
• type DropCollectionResp
• type DropIndexInput
• type DropIndexOutput
• type DropIndexReq
• type DropIndexResp
• type DropPartitionInput
• type DropPartitionOutput
• type DropPartitionReq
• type DropPartitionResp
• type Field
• type LoadCollectionReq
• type LoadCollectionResp
• type ReleaseCollectionReq
• type ReleaseCollectionResp
• type Result
• type SearchInput
• type SearchOutput
• type SearchReq
• type SearchResp
• type UpsertInput
• type UpsertOutput
• type UpsertReq
• type UpsertResp

Constants

const (
	TaskVectorSearch     = "TASK_VECTOR_SEARCH"
	TaskUpsert           = "TASK_UPSERT"
	TaskBatchUpsert      = "TASK_BATCH_UPSERT"
	TaskDelete           = "TASK_DELETE"
	TaskCreateCollection = "TASK_CREATE_COLLECTION"
	TaskDropCollection   = "TASK_DROP_COLLECTION"
	TaskCreatePartition  = "TASK_CREATE_PARTITION"
	TaskDropPartition    = "TASK_DROP_PARTITION"
	TaskCreateIndex      = "TASK_CREATE_INDEX"
	TaskDropIndex        = "TASK_DROP_INDEX"
)

Functions

func Init

func Init(bc base.Component) *component

Types

type BatchUpsertInput

type BatchUpsertInput struct {
	CollectionName string           `json:"collection-name"`
	PartitionName  string           `json:"partition-name"`
	ArrayData      []map[string]any `json:"array-data"`
}

type BatchUpsertOutput

type BatchUpsertOutput struct {
	Status string `json:"status"`
}

type CreateCollectionInput

type CreateCollectionInput struct {
	CollectionName   string         `json:"collection-name"`
	Dimension        int            `json:"dimension"`
	MetricType       string         `json:"metric-type"`
	IDType           string         `json:"id-type"`
	AutoID           bool           `json:"auto-id"`
	PrimaryFieldName string         `json:"primary-field-name"`
	VectorFieldName  string         `json:"vector-field-name"`
	Schema           map[string]any `json:"schema"`
	IndexParams      map[string]any `json:"index-params"`
	Params           map[string]any `json:"params"`
}

type CreateCollectionOutput

type CreateCollectionOutput struct {
	Status string `json:"status"`
}

type CreateCollectionReq

type CreateCollectionReq struct {
	CollectionNameReq   string         `json:"collectionName"`
	DimensionReq        int            `json:"dimension"`
	MetricTypeReq       string         `json:"metricType"`
	IDTypeReq           string         `json:"idType"`
	AutoIDReq           bool           `json:"autoID"`
	PrimaryFieldNameReq string         `json:"primaryFieldName"`
	VectorFieldNameReq  string         `json:"vectorFieldName"`
	SchemaReq           map[string]any `json:"schema"`
	IndexParamsReq      map[string]any `json:"indexParams"`
	ParamsReq           map[string]any `json:"params"`
}

type CreateCollectionResp

type CreateCollectionResp struct {
	Code    int            `json:"code"`
	Message string         `json:"message"`
	Data    map[string]any `json:"data"`
}

type CreateIndexInput

type CreateIndexInput struct {
	CollectionName string         `json:"collection-name"`
	IndexParams    map[string]any `json:"index-params"`
}

type CreateIndexOutput

type CreateIndexOutput struct {
	Status string `json:"status"`
}

type CreateIndexReq

type CreateIndexReq struct {
	CollectionName string           `json:"collectionName"`
	IndexParams    []map[string]any `json:"indexParams"`
}

type CreateIndexResp

type CreateIndexResp struct {
	Code    int    `json:"code"`
	Message string `json:"message"`
}

type CreatePartitionInput

type CreatePartitionInput struct {
	CollectionName string `json:"collection-name"`
	PartitionName  string `json:"partition-name"`
}

type CreatePartitionOutput

type CreatePartitionOutput struct {
	Status string `json:"status"`
}

type CreatePartitionReq

type CreatePartitionReq struct {
	CollectionNameReq string `json:"collectionName"`
	PartitionNameReq  string `json:"partitionName"`
}

type CreatePartitionResp

type CreatePartitionResp struct {
	Code    int            `json:"code"`
	Message string         `json:"message"`
	Data    map[string]any `json:"data"`
}

type Data

type Data struct {
	UpsertCount int      `json:"upsertCount"`
	UpsertIDs   []string `json:"upsertIds"`
}

type DataDescribe

type DataDescribe struct {
	Fields []Field `json:"fields"`
}

type DeleteInput

type DeleteInput struct {
	CollectionName string `json:"collection-name"`
	PartitionName  string `json:"partition-name"`
	Filter         string `json:"filter"`
}

type DeleteOutput

type DeleteOutput struct {
	Status string `json:"status"`
}

type DeleteReq

type DeleteReq struct {
	CollectionNameReq string `json:"collectionName"`
	PartitionNameReq  string `json:"partitionName"`
	FilterReq         string `json:"filter"`
}

type DeleteResp

type DeleteResp struct {
	Code    int            `json:"code"`
	Message string         `json:"message"`
	Data    map[string]any `json:"data"`
}

type DescribeCollection

type DescribeCollection struct {
	CollectionName string `json:"collection-name"`
}

type DescribeCollectionReq

type DescribeCollectionReq struct {
	CollectionNameReq string `json:"collectionName"`
}

type DescribeCollectionResp

type DescribeCollectionResp struct {
	Code    int          `json:"code"`
	Message string       `json:"message"`
	Data    DataDescribe `json:"data"`
}

type DropCollectionInput

type DropCollectionInput struct {
	CollectionName string `json:"collection-name"`
}

type DropCollectionOutput

type DropCollectionOutput struct {
	Status string `json:"status"`
}

type DropCollectionReq

type DropCollectionReq struct {
	CollectionNameReq string `json:"collectionName"`
}

type DropCollectionResp

type DropCollectionResp struct {
	Code    int            `json:"code"`
	Message string         `json:"message"`
	Data    map[string]any `json:"data"`
}

type DropIndexInput

type DropIndexInput struct {
	CollectionName string `json:"collection-name"`
	IndexName      string `json:"index-name"`
}

type DropIndexOutput

type DropIndexOutput struct {
	Status string `json:"status"`
}

type DropIndexReq

type DropIndexReq struct {
	CollectionNameReq string `json:"collectionName"`
	IndexNameReq      string `json:"indexName"`
}

type DropIndexResp

type DropIndexResp struct {
	Code    int    `json:"code"`
	Message string `json:"message"`
}

type DropPartitionInput

type DropPartitionInput struct {
	CollectionName string `json:"collection-name"`
	PartitionName  string `json:"partition-name"`
}

type DropPartitionOutput

type DropPartitionOutput struct {
	Status string `json:"status"`
}

type DropPartitionReq

type DropPartitionReq struct {
	CollectionNameReq string `json:"collectionName"`
	PartitionNameReq  string `json:"partitionName"`
}

type DropPartitionResp

type DropPartitionResp struct {
	Code    int            `json:"code"`
	Message string         `json:"message"`
	Data    map[string]any `json:"data"`
}

type Field

type Field struct {
	Name       string `json:"name"`
	PrimaryKey bool   `json:"primaryKey"`
}

type LoadCollectionReq

type LoadCollectionReq struct {
	CollectionNameReq string `json:"collectionName"`
}

type LoadCollectionResp

type LoadCollectionResp struct {
	Code    int    `json:"code"`
	Message string `json:"message"`
}

type ReleaseCollectionReq

type ReleaseCollectionReq struct {
	CollectionNameReq string `json:"collectionName"`
}

type ReleaseCollectionResp

type ReleaseCollectionResp struct {
	Code    int    `json:"code"`
	Message string `json:"message"`
}

type Result

type Result struct {
	Ids      []string         `json:"ids"`
	Data     []map[string]any `json:"data"`
	Vectors  [][]float32      `json:"vectors"`
	Metadata []map[string]any `json:"metadata"`
}

type SearchInput

type SearchInput struct {
	CollectionName string         `json:"collection-name"`
	PartitionName  string         `json:"partition-name"`
	Vector         []float32      `json:"vector"`
	Filter         string         `json:"filter"`
	Limit          int            `json:"limit"`
	VectorField    string         `json:"vector-field"`
	Offset         int            `json:"offset"`
	GroupingField  string         `json:"grouping-field"`
	Fields         []string       `json:"fields"`
	SearchParams   map[string]any `json:"search-params"`
}

type SearchOutput

type SearchOutput struct {
	Status string `json:"status"`
	Result Result `json:"result"`
}

type SearchReq

type SearchReq struct {
	CollectionName string         `json:"collectionName"`
	PartitionName  string         `json:"partitionName"`
	Data           [][]float32    `json:"data"`
	Filter         string         `json:"filter"`
	Limit          int            `json:"limit"`
	AnnsField      string         `json:"annsField"`
	Offset         int            `json:"offset"`
	GroupingField  string         `json:"groupingField"`
	OutputFields   []string       `json:"outputFields"`
	SearchParams   map[string]any `json:"searchParams"`
}

type SearchResp

type SearchResp struct {
	Code    int              `json:"code"`
	Message string           `json:"message"`
	Data    []map[string]any `json:"data"`
}

type UpsertInput

type UpsertInput struct {
	CollectionName string         `json:"collection-name"`
	PartitionName  string         `json:"partition-name"`
	Data           map[string]any `json:"data"`
}

type UpsertOutput

type UpsertOutput struct {
	Status string `json:"status"`
}

type UpsertReq

type UpsertReq struct {
	CollectionNameReq string           `json:"collectionName"`
	PartitionNameReq  string           `json:"partitionName,omitempty"`
	DataReq           []map[string]any `json:"data"`
}

type UpsertResp

type UpsertResp struct {
	Code    int    `json:"code"`
	Message string `json:"message"`
	Data    Data   `json:"data"`
}