deployment milvus cluster based on openstack-vm

[cherrylzhao]

hi, everyone.
currently we only have openstack environment, is it possible to deploy Milvus-cluster based on vm instead of docker, and could i remove the mq (pulsar) module in distributed mode? since I only found the etcd & minio in standalone mode.

thanks

[yhmo]

We don't have experience with openstack. Milvus cluster is deployed by k8s(by helm, operator), so it is possible if you can install the fundmental infrastructures on your openstack.
Pulsar is a must-have dependency for milvus cluster, cannot be removed from milvus cluster. Milvus standalone is a single process, so it can use a local rocksdb to replace the pulsar.

Another possible solution is zilliz cloud: https://zilliz.com/ which is a cloud service based on milvus.

[yhmo]

Session level is for a single client process.

Let's say we use python sdk to run a client program to do insert/search, the program uses "Session" level for search, this level ensure that "the data inserted by this program is visible to the search/query request sent by this program".

[yhmo]

Yes, the other programs might not see the latest data inserted by that program.

[cherrylzhao]

Milvus manages messages by time synchronization machinery: https://milvus.io/docs/time_sync.md#Time-Synchronization Each dml request is processed as one or multiple messages to pulsar/kafka, there is a timestamp in each message. Query nodes and data nodes subscribe these messages from pulsar/kafka. For each 200ms, the rootcoord node will send a time-tick message to pulsar/kafka, which is like a heart-beat. Query nodes and data nodes also receive these time-tick messages for each 200ms. Once a time-tick message is consumed by all query nodes, then we say "the data is consistent at this time point".

thanks again, I think each dml request corresponding messages should be consumed by one data nodes, how we avoid to duplicate message consumes by others.

[yhmo]

No need to care which message is consumed by which node. Milvus manages the messages by itself, messages always go to the right node.

[cherrylzhao]

through the roadmap, milvus will simplify the architecture in the future, it seems that pulsar will be replaced by stream node, is there any docs that I can follow, thanks.