Which message broker to choose

[ldmi3i]

I propose to discuss which message broker we may choose for the async actions.

On the terrastruct V1 scheme ( https://app.terrastruct.com/diagrams/1204758278/edit#layer=877011438 ) I assumed using of priority queue ( request from API gateway to "data scraping service") to process requests from users in higher priority than updating already existing API definitions (and higher priority than adding new API refs from the crawler in future) ).
So I'm keeping in mind idea to use RabbitMQ cause it supports priority for the queue and fits the architecture fine.

But we may emulate priorities from the client side and just use different "topics" for different priorities.
So we may use NATS for instance which much more lightweight and has some interesting features. Or maybe some others brokers.

Please share your opinion, which broker should we choose?

[syedalisait]

Could you please elaborate on the scale we are talking about here?
how many requests are we expecting per second?
How long are we expecting the data scraping service takes for parsing the content for each message from the queue?

Assuming if we will be running multiple instances of the data scraping service wihich can be horizontally scaled easily, do we have to implement priority queue?

[ldmi3i]

I don't think we can predict the scale cause we don't have business model, and I'm just a developer who just finds resolving of problem of creating API hub interesting. Maybe no one will use this application, I don't know.
In my opinion prioritisation of sources scraping is the right thing to do and it will not cause problems with the big amount of user waiting time in future (and also in present). But avoiding prioritisation with some probability (or depending on the circumstances) may cause incorrect behaviour - and hence to unstable system.

Also this part with data scraping service may become under load If we reach the stage of automatically collecting data from the entire Internet. So it's difficult to predict load currently.

[ldmi3i]

It turns out that this service can come:

1. Requests from the user to add new API. (in the case of the second stage, this may be scanning the entire website with links following)
2. Update existing API definition (probably at certain intervals from the storage and update service).
3. In the case of entire Internet scanning, this will be a constant stream of domains for a complete scan.

Those. in the final form, it may be a constant load, and user requests should be processed as soon as possible.

[pallasite99]

I would be inclined to go with RabbitMQ because it is proven to handle requests at a large scale.

Even the last company I worked with had this used for almost every communication in K8s. And they handled infrastructure at a very large scale.

It is easy and intuitive to use afaik, although I haven’t directly worked on it at any stage

[syedalisait]

Thank you for the explanation.
In that case, could we consider Apache Kafka also into the discussion

https://www.confluent.io/blog/kafka-fastest-messaging-system/

[ldmi3i]

Yes, sure. Kafka is great and I used to take a part in project with it about two years ago. But if I'm not mistaken, Kafka doesn't have default docker image and is not easy to setup.
Also Kafka written in Java + Scala and consume a lot of resources. Don't know about RabbitMQ, but NATS written in Ruby + Go and very lightweight (but NATS has no persistence).
I think we may collect all propositions discuss and vote.

I'm just a little worried about resources cause I don't know where we can deploy project. Go-services take several Mb each. But just Scala lib alone is bigger than all the services combined.

[syedalisait]

Understood. Thank you for your view