1

We are using Milvus2.0 for vector search. And Milvus is positioned as a database. So it should provide an efficient way to export data from it.

After reading Milvus Website, I only find an API named "query" to obtain vectors from Milvus. Then I test this API. But I found it is too slow to use it.

How do I quickly export large quantities of data from Milvus?

Chris Galbraith
  • 53
  • 4

1 Answers1

1

Query is not designed to pull massive amount of data from Milvus for the following reasons...

  • 2 Gb rpc transfer limit per call and pagination is not implemented yet
  • performance is slower than pulling directly from drive

There is a tool called Milvus Data Migration (MilvusDM) tool, however, it does not support Milvus 2.x yet. The community is working on this and it will be available in future release. This is the MEP document: https://wiki.lfaidata.foundation/display/MIL/MEP+24+--+Support+bulk+load

Stan Shen
  • 66
  • 3