ARC Scaling Issue - Pod Persists After Running a Workflow

[djabalbuena]

This is somewhat similar to other issues posted here but the difference is that the pod is in Running status
#3840
#2656

Controller Version

gha-runner-scale-set-controller-0.9.3

Deployment Method

Helm

Issue:

I'm encountering an intermittent issue with the actions-runner-controller (ARC) where some runner pods do not terminate as expected after completing their workflows. Instead, these pods persist and continue to run for extended periods. This behavior is inconsistent and sometimes prevents new pods from spinning up due to the existing pods.

Here are some relevant log entries:

https://gist.github.com/djabalbuena/e2f9079f4cf5229a85d09eab93da115e
https://gist.github.com/djabalbuena/db1917a1e623209384741d6a7c9685dd

Events:
Type Reason Age From Message
Normal Killing 70s kubelet Stopping container runner
Normal Killing 70s kubelet Stopping container dind

Steps Taken:

1. Verified runner configuration for one-time use mode.
2. Monitored controller logs during pod deletion attempts but I am not getting any useful logs on the controller pod.
3. Checked for finalizers and owner references on the pod.
4. Ensured network stability and sufficient resource limits.

Despite these efforts, the pods still do not terminate as expected. The only workaround is to manually delete the pod, which is not ideal. Any insights or suggestions on how to resolve this issue would be greatly appreciated.