Milvus Private Preview
Milvus is an open-source vector database built for GenAI applications. It provides high-performance and scalable vector search for use cases like semantic search, Retrieval-Augmented Generation, and multi-modal search. Fivetran supports Milvus and Zilliz Cloud (fully-managed Milvus) as a destination.
The quickest way to get up and running with Milvus is to create a new cluster in Zilliz Cloud.
NOTE: Only the following connectors are supported as sources for this destination:
Learn how to sync data from sources not directly supported by Milvus.
Optimal architecture
The following diagram demonstrates the optimal architecture for syncing source data to Milvus destination using Fivetran.
Setup guide
Follow our step-by-step Milvus setup guide to connect your Milvus vector database with Fivetran.
NOTE: This destination is partner-built. For any questions related to Milvus destination and its documentation, raise a GitHub issue.
Type transformation mapping
The data types in your Milvus vector database are mapped from Fivetran's standard data type storage.
We use the following data type conversions:
| Fivetran Data Type | Destination Data Type |
|---|---|
| BOOLEAN | BOOLEAN |
| SHORT | INT16 |
| INT | INT32 |
| LONG | INT64 |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| DECIMAL | VARCHAR |
| LOCALDATE | VARCHAR |
| LOCALDATETIME | VARCHAR |
| INSTANT | VARCHAR |
| STRING | VARCHAR |
| JSON | JSON |
| BINARY | VARCHAR |
Limitations
The Milvus destination has the following known limitations:
- Only the following sources are supported*:
- The following are not supported:
- Schema changes
- Truncates
- Partial row updates
- History mode
- Only OpenAI's embedding models are supported.
*NOTE: Learn how to sync data from sources not directly supported by Milvus.
Syncing data from sources not directly supported by Milvus
You can sync data from other sources not directly supported by Milvus by doing the following:
- Create or use an existing Databricks, Snowflake, or BigQuery destination.
- Set up your connectors that are not directly supported by Milvus to sync relevant data to your Databricks, Snowflake, or BigQuery destination.
- Create a Milvus destination.
- Set up transformations in your Databricks, Snowflake, or BigQuery destination so that the relevant data you want to sync to your Milvus destination is stored in one or multiple table's column named 'original_text' of type STRING.
- Set up your intermediate destination as a Databricks, Snowflake, or BigQuery source connector to sync the 'original_text' columns to your Milvus destination.