Getting "512 worker_connections are not enough" on High Load

[maieredv-manuel]

Hey!

were using NPMPlus for 23 Proxy and 11 Stream Hosts.
Now we have the problem when our internal users are accessing the internal applications we are getting the "512 worker_connections are not enough" Message.

I found that there is an setting in the ngnix.conf file, but i think when ive edit this file, on the next update it will be changed to default!? Also there is an option that nginx can handle the worker_connection itselfs.

Can he have an ENV option for Settings? Or any otherway?

[Zoey2936]

Also there is an option that nginx can handle the worker_connection itselfs.

which exactly?

Can he have an ENV option for Settings? Or any otherway?

if nginx can do this automatic then I would do this, if not then it depends if increasing this value has any downsides, if not then I can jsut increase the default

[maieredv-manuel]

https://www.sabatino.dev/misconfigured-nginx-worker_connections-caused-downtime/

I went to the nginx docs to read up about this parameter, and by default this was configured to 512.

Because of our many simultaneous websocket connections, we were effectively hitting the 512 limit, and NGINX was not configured to increase the worker_processes, so when our application server would hit this limit, it stopped accepting any new incoming requests.

I resolved this by increasing our worker_connections to 768 and I changed the worker_processes configuration from 1 to auto, so nginx would add extra workers if needed.

worker_processes auto;
events {
	worker_connections 768;
}

[Zoey2936]

the default for one of the values should already be auto:

NPMplus/rootfs/usr/local/nginx/conf/nginx.conf

Line 5 in e5ecf35

worker_processes auto;

[maieredv-manuel]

Ah okay, thats the setting for allocating all CPU Cores - thats good.
Now we need the "worker_connections", i didnt found that setting in the nginx.conf !? that means when nothing is set in the config file, the default is set!?

[Zoey2936]

It should use the value written in nginx docs

[maieredv-manuel]

So can we change the value via "/data/nginx_custom/events.conf" or gets that file also an "reset" when updating?

[maieredv-manuel]

Ive changed the "/data/nginx_custom/events.conf" with "worker_connections 768;" and restartet the container, actually it started, but i dont know if that really changed anything ... do you know a way to check the current loaded config or get the actual worker_connections?

[Zoey2936]

docker exec npmplus nginx -T should output the full loaded config

[maieredv-manuel]

thats saadly only the output of all configs loaded, so only the "include /data/nginx_custom/events.conf;" is in the output, not if the process really uses the changed "worker_connections" value :-/

But okay, ill test it after the holidays, when next time all users are in the office :-) ill report again!

Thanks for your help and merry christmas!

[Zoey2936]

Do you think increasing worker_connections by default is a bad idea?
And how many cpu cores has this system? Since worker_procces should be identical to the number of cpu threads.

Any many thanks!

[maieredv-manuel]

Puuuh, i dont know if its needed for all installations.
When it can be done by editing that one file, its not that big deal.

My machine has only 1 CPU Core, but that wasnt the problem, cpu load is at ~5-10% at this time when ive got the "512 worker_connections are not enough" errors.
I think auto is a good setting for "worker_processes" so nginx can scale up when its needed.

[Zoey2936]

The total connections are calculated by processes*connection, so doubling the cpu cores would double the total connections

[maieredv-manuel]

Aha, okay ...
That makes sense - but the CPU load isnt the problem, like i said, we have around 10% Load at this time ...

But i think for the most installations with more then one CPU Core it shouldt be an problem.

[Zoey2936]

Yes thats true, but the cpu cores definde the number of total processes

[maieredv-manuel]

Sure, but when we can adjust the worker_processes with the "worker_connections" Setting in "/data/nginx_custom/events.conf" its not a big deal to do that :-) Why should i change my Cloud VM to something bigger with more costs when i can change the behave of nginx with a simple setting? :-)

I think the Setting in that file is also peristent for updates because its an file in the "Data" Folder that is accessible for all, so there is also no need for changing anything from your side.

[Zoey2936]

yes this file is persistent

[maieredv-manuel]

My Chef said we update the VM anyways :-D So all its all good, but also good to know that there is an option to change it, without the need of upgrading :-)

[Zoey2936]

I will convert this to a discussion