Can't open nfl.com website on Heroku but works fine locally

[robertr2112]

I've got a webscraper app that is scraping the game info from NFL.com. I have it working fine on my local debian system but it fails with 2 alternating errors when running it on Heroku. These errors alternate as to which will come up, however the first error listed below is the most common. It appears to me as if I may have a configuration issue, but I'm not familiar enough with Selenium/Webdriver/Undetected-chromedriver to know.

Here is the code I'm running:

from bs4 import BeautifulSoup
import undetected_chromedriver as uc
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

uc_options = uc.ChromeOptions()
uc_options.add_argument('--headless')
driver = uc.Chrome(options=uc_options, use_subprocess=False)

path = f"https://www.nfl.com/schedules/{year}/REG{week}"
driver.get(path)

### Wait for the javascript content to load ###
wait = WebDriverWait(driver, timeout=25)
wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'nfl-o-matchup-group')))

doc = BeautifulSoup(driver.page_source, 'html.parser')

And here is the Heroku setup information:
Buildpacks:

heroku/python
heroku/ruby
https://github.com/heroku/heroku-buildpack-google-chrome
https://github.com/heroku/heroku-buildpack-chromedriver

Requirements.txt

selenium==4.9
undetected-chromedriver==3.5.3
bs4==0.0.1

Here is the first (and most common) error I'm receiving:

Traceback (most recent call last):
  File "/app/lib/assets/python/nfl-test.py", line 149, in <module>
    driver.get(path)
  File "/app/.heroku/python/lib/python3.11/site-packages/undetected_chromedriver/__init__.py", line 629, in get_wrapped
    return orig_get(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/site-packages/undetected_chromedriver/__init__.py", line 665, in get
    return super().get(url)
           ^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 449, in get
    self.execute(Command.GET, {"url": url})
  File "/app/.heroku/python/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
    self.error_handler.check_response(response)
  File "/app/.heroku/python/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed
  (Session info: chrome=117.0.5938.132)
Stacktrace:
#0 0x562f8b6b36b3 <unknown>
#1 0x562f8b389036 <unknown>
#2 0x562f8b37283b <unknown>
#3 0x562f8b372081 <unknown>
#4 0x562f8b37106c <unknown>
#5 0x562f8b370fde <unknown>
#6 0x562f8b36fbe4 <unknown>
#7 0x562f8b3701df <unknown>
#8 0x562f8b37fbce <unknown>
#9 0x562f8b393ef1 <unknown>
#10 0x562f8b3988ab <unknown>
#11 0x562f8b370901 <unknown>
#12 0x562f8b393b59 <unknown>
#13 0x562f8b40b51c <unknown>
#14 0x562f8b3f1e93 <unknown>
#15 0x562f8b3c4934 <unknown>
#16 0x562f8b3c571e <unknown>
#17 0x562f8b678cb8 <unknown>
#18 0x562f8b67cbf0 <unknown>
#19 0x562f8b68719c <unknown>
#20 0x562f8b67d808 <unknown>
#21 0x562f8b64a27f <unknown>
#22 0x562f8b6a1e88 <unknown>
#23 0x562f8b6a2059 <unknown>
#24 0x562f8b6b2843 <unknown>
#25 0x7fd676325609 start_thread

Here is the other error I receive(although less frequently):

Traceback (most recent call last):
  File "/app/lib/assets/python/nfl-test.py", line 154, in <module>
    wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'nfl-o-matchup-group')))
  File "/app/.heroku/python/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 86, in until
    value = method(self._driver)
               ^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/site-packages/selenium/webdriver/support/expected_conditions.py", line 139, in 
_predicate
    return _element_if_visible(driver.find_element(*locator))
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 831, in find_element
    return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
    self.error_handler.check_response(response)
  File "/app/.heroku/python/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools
  (failed to check if window was closed: disconnected: not connected to DevTools)
  (Session info: chrome=117.0.5938.132)
Stacktrace:
#0 0x55f4778756b3 <unknown>
#1 0x55f47754b1e7 <unknown>
#2 0x55f47753273d <unknown>
#3 0x55f4775325c3 <unknown>
#4 0x55f47754d2e2 <unknown>
#5 0x55f4775ccea0 <unknown>
#6 0x55f4775b3e93 <unknown>
#7 0x55f477586934 <unknown>
#8 0x55f47758771e <unknown>
#9 0x55f47783acb8 <unknown>
#10 0x55f47783ebf0 <unknown>
#11 0x55f47784919c <unknown>
#12 0x55f47783f808 <unknown>
#13 0x55f47780c27f <unknown>
#14 0x55f477863e88 <unknown>
#15 0x55f477864059 <unknown>
#16 0x55f477874843 <unknown>
#17 0x7f47c577d609 start_thread

[robertr2112]

So, I was able to fix these issues.

To fix the first problem I needed to add the following to my chrome_options:

chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

And to fix the second problem, I needed to ensure that the version of chrome running was the same as the chromedriver so on heroku I had to add the following command:

heroku config:set CHROMEDRIVER_VERSION=117.0.5938.132 --app  <appname>