avg-LoadCollection-milvus-proxy request latency meet a Sudden increase

[AnyangAngus]

Environment

• Milvus version: 2.4.1
• Deployment mode(standalone or cluster): cluster
• MQ type(rocksmq, pulsar or kafka): pulsar
• SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.2.12
• OS(Ubuntu or CentOS): Ubuntu
• CPU/Memory:
• GPU: null
• Others:
  Current Behavior
  avg-LoadCollection-milvus-proxy request latency meet a Sudden increase from 147 to 3.45K.
  even I load an empty collection need wait a lot of time.
  

[yhmo]

Observe the "Query Node" panel: Segment Load latency, Collection Load Num, Partition Load Num, Segment Load Num

[AnyangAngus]

Ideally, if all the collections are already loaded, you don't need to call collection.load() repeatly. Maybe you have new collection created for each minute, most of load actions are done very quickly, which means the resource is sufficient. Only one load action jumps to 3.45k milliseconds. It could be a system lag, maybe a lag of task schedulers, maybe a lag of the storage service(s3/minio). From the monitor ui, the "Load Segment Latency" shows at this time point the latency jumps from 200ms to 500ms, might be a storage lag, since this latency mainly depends on IO read.

@yhmo This db has been running for half a year. My Business Scenario is that when user need create a collection, miluvs server will build a collection and load it then never unload forever, So you can see the db loaded collection number is 2k now. Recently when create and load a new collection, the latency increase compared to before significantly.
so I can not delete the history collections owned users, but I can unload history collections, will I decrease the number of loaded collection can help? That is why I ask whether 2k loaded collections will affect latency or 2k collections regardless of whether they are loaded or not.

[yhmo]

Normally, if the memory usage of query nodes is not a high level, you don't need to unload collections. However, if some historical collections are no longer used, it is better to release them.
Theoretically, with collection number increased, the time cost to check collection loaded state is also increased. Increase slowly, not suddenly.
2.4.1 is a pretty old version, there are lots of fixes and enhancements for "load" in the 2.4.x: https://milvus.io/docs/v2.4.x/release_notes.md.

[AnyangAngus]

Normally, if the memory usage of query nodes is not a high level, you don't need to unload collections. However, if some historical collections are no longer used, it is better to release them. Theoretically, with collection number increased, the time cost to check collection loaded state is also increased. Increase slowly, not suddenly. 2.4.1 is a pretty old version, there are lots of fixes and enhancements for "load" in the 2.4.x: https://milvus.io/docs/v2.4.x/release_notes.md.

@yhmo Got it, Thank you!

If I still need to serve 2k and 3k collections in the future, which step can I do to maintain the load collection latency?

• Up the version to 2.4.20
• increase the node query pod, ?
• release unused loaded collection and keep loaded collection number maintain 1k around?

[yhmo]

Upgrade to v2.4.20+
Increase query node resource(memory or node number) since more collections are loaded, more memory is required.
No need to unload collections if you have sufficient memory.
Release unused collections if you have insufficient memory.

[AnyangAngus]

Upgrade to v2.4.20+ Increase query node resource(memory or node number) since more collections are loaded, more memory is required. No need to unload collections if you have sufficient memory. Release unused collections if you have insufficient memory.

@yhmo
As for increasing the query node resource, I check the querynode memory info which only consum a quarter of the pod memory. Is it necessary to increase the number of replicas in this situation？

On the other hand, Are there any IO issue which may effect the load latency? maybe some etcd or minio metrics and clean some catch may help?