Crowdsec and Immich

[MaximumFish]

Hi,

I recently migrated from a working nginx-proxy-manager + crowdsec (+ cloudflare) install to NPMplus + crowdsec. I thought everything was working nicely but it turns out that local backups to Immich are now failing, which I'm pretty sure is due to Crowdsec, even when uploading on the LAN. Stupidly, I deleted the old stack with all its config so I'm not entirely sure what I'm missing (and I didn't take any notes!) but I think that when it was working it was thanks to the suggestions in this thread: immich-app/immich#3243

Alas it doesn't seem to help with npmplus and now I'm at a bit of a loss. Before I get too far into the weeds trying to bodge something together, has anybody got a working example that I can crib from?

Thanks in advance...

[Zoey2936]

any docker logs?

[Strana-Mechty]

If you open a console in portainer/use docker exec on the crowdsec container, what does cscli decisions list say?

If you're sure crowdsec is causing it you can always whitelist the local IP (something I recommend you always do to prevent embarassing lock-out situations or strange APIs that work over intranet anyway), should it disappear then it's confirmed, you might wanna see in the decisions list what rule got tripped and remove it from the crowdsec list.

https://doc.crowdsec.net/u/getting_started/post_installation/whitelists/

[Zoey2936]

?

[Strana-Mechty]

It sounds to me like crowdsec might be detecting a violation as a false-positive when using immitch. I'm experiencing the same thing but with modsec-CRS and services like gitea ("looks like you're opening a .git/.env file on that webserver 😄" ).

I don't have immitch however so it's a big fat guess to me.

[Zoey2936]

yes that's why logs are important, if it is a false positive then it would be visible in the log. But I must say, I never say crowdsec to overblock yeet

[MaximumFish]

My apologies, I am seeing these comments but it's been one of those days and I've not found a moment to investigate further. Thanks for your replies, I'll get back to you as soon as I can!

[MaximumFish]

OK, here we go:

2024/11/19 17:20:06 [warn] 80110#80110: *51203 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000050 while reading request body, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 send() failed (32: Broken pipe), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [error] 80110#80110: *51203 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/19 17:20:06 [alert] 80110#80110: *51203 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '10.42.0.11' with 'ban' (by appsec), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"

Yes, I'm using a very non-standard local subnet but it falls within the 10.0.0.0/8 range that's (supposed to be) whitelisted in Crowdsec by default.

As mentioned above I added the suggested whitelistings from this post (immich-app/immich#3243 (reply in thread)) which are like so:

name: crowdsecurity/immich-whitelists
description: "Whitelist false positive from Immich-api"
filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']"
whitelist:
  reason: "Whitelist false positive from Immich-api"
  expression:
   - evt.Meta.http_verb == 'POST' && evt.Meta.http_status == '403' && evt.Parsed.request contains '/asset/upload'
   - evt.Meta.http_verb == 'GET' && evt.Meta.http_status == '429' && evt.Parsed.request contains '/api/asset/thumbnail/'
   - evt.Meta.http_verb == 'GET' && evt.Meta.http_status == '200' && evt.Parsed.request contains '/api/asset/thumbnail/'
   - evt.Meta.http_verb == 'GET' && evt.Meta.http_status == '304' && evt.Parsed.request contains '/api/asset/thumbnail/'
   - evt.Meta.http_verb == 'GET' && evt.Meta.http_status == '404' && evt.Parsed.request contains '/api/asset/thumbnail/'

Though looking at the logs, maybe the API endpoint has changed from /api/asset/<function> to just /api/assets? Though I can't find any supporting comments about it in the Immich changelog. Or perhaps I just need to add that endpoint to the whitelist and keep the others there...

Thanks for your help, both. It's massively appreciated.

[Zoey2936]

whitelisting will not help.
can you please disable appsec in your crowdsec config file. just comment the lines mentioning appsec please and retry (also disable modsec)

[MaximumFish]

I don't know what I've managed to do but now Crowdsec is completely broken. I set the file in acquis.d to read:

filenames:
  - /opt/npm/nginx/access.log
labels:
  type: npmplus
---
source: docker
container_name:
 - npmplus
labels:
  type: npmplus
#---
#source: docker
#container_name:
# - npmplus
#labels:
#  type: modsecurity
#---
#listen_addr: 0.0.0.0:7422
#appsec_config: crowdsecurity/appsec-default
#name: appsec
#source: appsec
#labels:
#  type: appsec

But after doing that it tries to load but bombs out with messages like:

level=fatal msg="crowdsec init: while loading scenarios: scenario loading failed: unable to load alert context: compilation of 'match.uri' context value failed: unknown name match (1:1)\n | match.uri\n | ^

and

level=fatal msg="crowdsec init: while loading scenarios: scenario loading failed: unable to load alert context: compilation of 'match.method' context value failed: unknown name match (1:1)\n | match.method\n | ^

I tried to undo the changes, completely removed the container, deleted all the local config files, even logged into the CS dashboard and removed the engine from there. Went right back to square one and tried to get it reinstalled from scratch (just crowsec, not npmplus itself) but it's still doing the same thing.

Fake edit: Deploying a new container with the - COLLECTIONS=ZoeyVid/npmplus line removed from the compose file allows crowdsec to load, but I guess it's not really doing anything useful like that. I just don't know what's going on at this point!

[Zoey2936]

crowdsec config, not npmplus.yaml, you edited the wrong file

[MaximumFish]

OK, I've fixed it though I've got no idea how it got so incredibly broken to the point that rolling back changes didn't help at all. I ended up doing:

• Stopped and deleted the Crowdsec container, all local files (/opt/crowdsec) and deleted the engine from the Crowdsec dashboard.
• Stopped and deleted the npmplus container.
• Renamed the /opt/npmplus directory.
• Reinstalled npmplus from scratch, taking it as far as retrieving a new LE certificate.
• Stopped the container, copied in the original proxy hosts from the backup to /opt/npm/nginx/proxy_host and the database to /opt/npm/etc/npm
• Reinstalled Crowdsec following the original instructions but when editing /opt/npm/etc/crowdsec/crowdsec.conf I changed the line APPSEC_FAILURE_ACTION=deny to APPSEC_FAILURE_ACTION=passthrough

After doing all of that it appears to be working again, and Immich backups are now being processed. FWIW, the logs are now producing the following warnings when doing a backup:

2024/11/20 18:31:20 [warn] 1570#1570: *555 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000002 while reading request body, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/20 18:31:20 [error] 1570#1570: *555 send() failed (32: Broken pipe), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/20 18:31:20 [error] 1570#1570: *555 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/20 18:31:20 [error] 1570#1570: *555 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"

Thanks

[Zoey2936]

if you use passthrough you can also disable appsec... if the requests fails the entire protection is gone

[Zoey2936]

To conclude the current state:

• the issue comes from AppSec or Immich and should therefore be reported to Immich (not done I think) or Crowdsec (already done: send() failed (32: Broken pipe) when performing HTTP POST to Immich crowdsecurity/lua-cs-bouncer#77)
• Whitelisting will not fix anything here (at least for the appsec Issue)
• It can be fixed by either disabling appsec or setting APPSEC_FAILURE_ACTION to passthrough (both are bad solutions) in /opt/npm/etc/crowdsec/crowdsec.conf
• having modsec enabled can cause similar issues

If you have any further useful information you can also leave it here: crowdsecurity/lua-cs-bouncer#77

[MaximumFish]

Yes disabling appsec has fixed the issue. I assume that doesn't make Crowdsec entirely useless? Some protection is better than none...

Anyway, to be clear, I specifically raised this as a discussion rather than a bug report because I agree that it's not an issue with NPMPlus itself. It's just that I had it working fine in one of the other random forks of nginx-proxy-manager with just a bit of whitelisting (I think LePresidente but I didn't document any of it). I switched to npmplus because the other solution felt unwieldy and overly complicated and I like that this one tries to keep things simple.

I was just hoping that someone had run into the same thing and had a solution. :)

[Zoey2936]

It worked with LePresidente because they don't enable appsec in their fork.
crowdsec has three parts:

• appsec: NPMplus redirects the request to appsec and ask if the request is "bad" => ban while requesting
• log parsing: reads logs and sees if an ip does "bad" things => ban after request
• bouncers: asks the local crowdsec api if an ip which made an request is banned, if yes, then NPMplus will block the request

[Zoey2936]

the issue you currently have/had is because of appsec - it seems like the connections between NPMplus and appsec disconnects or something like this (maybe timeout, maybe file to big, maybe something else) → issue of immich (requests that break appsec) or crowdsec (can't parse immich request)

the other issue you have/had is because of log parsing and can fix with whitelisting => the request is logged and crowdsec detects it as bad and blocks it

[MaximumFish]

Thanks for that explanation. My background is IT but I find Crowdsec super confusing and this has certainly helped! I've subscribed to the ticket you linked.

[yurividal]

@Zoey2936
I recently moved back to /lepresidente/nginxproxymanager

That image is based on official npm, and adds crowdsec-openresty-bouncer and appsec.

Immich works perfectly fine with it, even with appsec enabled.

[Zoey2936]

interesting, I have an Idea which then could be the issue. Is it possible that I create a special NPMplus build and you test if it fixes it?

[MaximumFish]

Happy to help with that. 👍

[Zoey2936]

So my thought is that modsec + crowsed in combination cause this error, so I created a NPMplus build without modsec.
So if you want to test:

1. make a backup (since all modsec related things will be removed with this build)
2. enable appsec again and also let it fail if it needs
3. change the tag to ghcr.io/zoeyvid/npmplus:pr-1248
4. docker compose up -d
5. test
6. report back if it works now or not

[MaximumFish]

Didn't make a difference for me. With appsec re-enabled and the new tag I get slightly different error codes, but the result is the same:

2024/11/21 16:01:41 [warn] 1436#1436: *100 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000001 while reading request body, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/21 16:01:41 [error] 1436#1436: *100 send() failed (32: Broken pipe), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/21 16:01:41 [error] 1436#1436: *100 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/21 16:01:41 [error] 1436#1436: *100 [lua] crowdsec.lua:651: Allow(): AppSec check: broken pipe, client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"
2024/11/21 16:01:41 [alert] 1436#1436: *100 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '10.42.0.11' with 'ban' (by appsec), client: 10.42.0.11, server: immich.example.com, request: "POST /api/assets HTTP/1.1", host: "immich.example.com"

[LaurenceJJones]

Also if you have chance on the linked issue from Zoey, if you can provide as much details as possible such as:

Is the deployment local or remote (VPS)?
Is the domain being proxied by a CDN like cloudflare?
Does it happen on upload like the OP or when you said when immich makes a backup?

We have the same issue as Zoey no one in crowdsec team runs immich so when we develop a test environment to try to replicate it, we need enough information so we are able to see it ourselves. As currently there is too many unknowns.

[MFYDev]

Will appreciate so much if Zoey or anyone else can provide a solution for this. Just switched to NPMplus and this issue is preventing me directly from editing my backend blog post, super frustrating already

[Zoey2936]

you can either temporary disable appsec or change APPSEC_FAILURE_ACTION to passthrough as written in the marked awnser

[yurividal]

Ok, so i have a theory as to why lepresidente/nginx-proxy-manager works, and npm-plus doesn't work.
I think, by default, that image uses APPSEC_FAILURE_ACTION passtrhough.

Here are my most recent nginx logs from my immich instance.
As you can see, appsec also randomy times out.

This is the default appsec configs for that image:

APPSEC_URL=<ip>:<port>
APPSEC_FAILURE_ACTION=passthrough  
APPSEC_CONNECT_TIMEOUT=300        
APPSEC_SEND_TIMEOUT=300            
APPSEC_PROCESS_TIMEOUT=1500        
ALWAYS_SEND_TO_APPSEC=false        
SSL_VERIFY=false

So, i think if we manage to find out why the communication issue between nginx and appsec is failing, we will have solved this issue.

[Zoey2936]

That makes sense, if you want please report a way to reproduce this to crowdsecurity/lua-cs-bouncer#77

[yurividal]

Initially, the containers were on different docker networks, and communicating via ports exposed to the host.
I then moved the 2 containers to the same docker network. Now the error is a little different. connection reset by peer, instead of broken pipe.

error 10.10.10.20 -> photos.****.com : Allow(): AppSec check: connection reset by peer - POST /api/assets HTTP/1.1 - 
error 10.10.10.20 -> photos.****.com : AppSecCheck(): Fallback because of err: connection reset by peer - POST /api/assets HTTP/1.1 - 

Still, seems like something is making the connection break between the proxy and appsec component. Not sure why it only affects immich. also, it only happens when uploading a file.