Can loading a collection, support an LRU cache in memory?

[lovepeacemz]

I have many collections. When loading a collection, can it support an LRU (Least Recently Used) cache in memory? For example, when I first load Collection A and then load Collection B, can Collection A be automatically released due to insufficient memory?

[yhmo]

In my opinion, it is better to do this in the upper applications. A milvus server might serve multiple clients, automatically releasing a collection will block the service of the other clients.

[lovepeacemz]

I think, The upstream service and the Milvus server are essentially similar, as upstream also requires global control of memory management. If each client can only control its own memory, and the current required collection cannot be loaded due to insufficient memory, it may also necessitate the release of other collections. This could block queries from other clients and would incur additional maintenance costs. In contrast, from the server's perspective, managing memory globally is more cost-effective and user-friendly.

[yhmo]

Wait @xiaofan-luan to comment. I don't think it is a good idea.

[lovepeacemz]

My scenario is as follows: we need to handle hundreds or even thousands of collections, each of which may be accessed very frequently. Do you have any good management solutions to suggest? Please share briefly. Thank you!

[yhmo]

"thousands of collections be accessed very frequently" ------------- it could run into a state of "always load/release collections", each search latency is poor, QPS is poor. The bottleneck is storage IO, especially for remote storage.

To reduce memory usage, we can choose DISKANN/IVF_PQ/IVF_SQ8 index, enable MMAP.

[lovepeacemz]

Thank you, but my provider's current machine configuration does not support disk and mmap. I apologize for not noticing this error 'very frequently'earlier, correct is 'thousands of collections may be accessed'

[xiaofan-luan]

@lovepeacemz
if each of the collection might be accessed very frequently, why do they need to be load and releaed?

We do working the automatically LRU feature but I guess under you case you need to load every collection onto disk use mmap.
This give you reason cost but great performance for hot data.

I would also recommend the zilliz cloud capacity optimized instance. It should give you a great balance between cost and performance

[lovepeacemz]

I apologize for the confusion in my previous statement. What I meant to say is that each collection might be accessed.
Additionally, I would like to ask, if all collections are preloaded, can this issue be resolved by horizontally scaling the number of query nodes?