Paralel processing of /explain/

[xsuchy]

Today, we are able to process only one request to /explain in a time. This does not scale.

We should investigate how to parallelize this.

In https://github.com/fedora-copr/logdetective/pull/106/files I discovered that:

I found that llama_cpp.server should have --parallel 2.See https://www.reddit.com/r/LocalLLaMA/comments/1be845y/multiple_concurrent_generations_with_llamacpp/
Is this what we need for parallel access?

But later @tt discovered that:

unfortunately, we are using llama-cpp-python's server and not the one in llama.cpp :// we are using this one and it doesn't know the parallel flag - I saw only threading being configurable https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#openai-compatible-web-server

We should either replace our server or find another way how to parallelize this.

[jpodivin]

Replacement with the server from llama.cpp would be the most straightforward.