Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prefect agent loses connection #7990

Closed
4 tasks done
johnkangw opened this issue Dec 27, 2022 · 4 comments
Closed
4 tasks done

Prefect agent loses connection #7990

johnkangw opened this issue Dec 27, 2022 · 4 comments
Labels
bug Something isn't working

Comments

@johnkangw
Copy link

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar issue and didn't find it.
  • I searched the Prefect documentation for this issue.
  • I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

I upgraded from Prefect 2.7.2 to 2.7.4 on a windows virtual machine that runs Prefect agents. Unfortunately, one of the agents running on the device has connection errors, so the agent no longer processes flows.

Reproduction

Please take a look at the error below. This started when I upgraded from 2.7.2 to 2.7.4.

Error

13:16:47.143 | INFO    | Flow run 'sociable-curassow' - Finished in state Completed()

Failed the last 3 attempts.  Please check your environment and configuration.
Examples of recent errors:

Traceback (most recent call last):
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\h2\connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\http2.py", line 108, in
handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\http2.py", line 205, in
_send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\h2\connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\h2\connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_transports\default.py", line 60, in
map_httpcore_exceptions
    yield
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_transports\default.py", line 353, in
handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in
handle_async_request
    raise exc
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in
handle_async_request
    response = await connection.handle_async_request(request)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\connection.py", line 90, in
handle_async_request
    return await self._connection.handle_async_request(request)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\http2.py", line 144, in
handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\utilities\services.py", line 46, in
critical_service_loop
    await workload()
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\agent.py", line 163, in
get_and_submit_flow_runs
    async for work_queue in self.get_work_queues():
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\agent.py", line 124, in get_work_queues
    work_queue = await self.client.read_work_queue_by_name(name)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\client\orion.py", line 713, in
read_work_queue_by_name
    response = await self._client.get(f"/work_queues/name/{name}")
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1751, in get
    return await self.request(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\client\base.py", line 229, in send
    response = await self._send_with_retry(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\client\base.py", line 187, in
_send_with_retry
    response = await request()
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1642, in
_send_handling_auth
    response = await self._send_handling_redirects(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1679, in
_send_handling_redirects
    response = await self._send_single_request(request)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1716, in
_send_single_request
    response = await transport.handle_async_request(request)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_transports\default.py", line 353, in
handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_transports\default.py", line 77, in
map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED



Failed the last 3 attempts.  Please check your environment and configuration.
Examples of recent errors:

Traceback (most recent call last):
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\h2\connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\http2.py", line 108, in
handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\http2.py", line 205, in
_send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\h2\connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\h2\connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_transports\default.py", line 60, in
map_httpcore_exceptions
    yield
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_transports\default.py", line 353, in
handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in
handle_async_request
    raise exc
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in
handle_async_request
    response = await connection.handle_async_request(request)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\connection.py", line 90, in
handle_async_request
    return await self._connection.handle_async_request(request)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpcore\_async\http2.py", line 144, in
handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\utilities\services.py", line 46, in
critical_service_loop
    await workload()
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\agent.py", line 222, in
check_for_cancelled_flow_runs
    cancelling_flow_runs = await self.client.read_flow_runs(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\client\orion.py", line 1540, in
read_flow_runs
    response = await self._client.post(f"/flow_runs/filter", json=body)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1842, in post
    return await self.request(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\client\base.py", line 229, in send
    response = await self._send_with_retry(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\prefect\client\base.py", line 187, in
_send_with_retry
    response = await request()
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1642, in
_send_handling_auth
    response = await self._send_handling_redirects(
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1679, in
_send_handling_redirects
    response = await self._send_single_request(request)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_client.py", line 1716, in
_send_single_request
    response = await transport.handle_async_request(request)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_transports\default.py", line 353, in
handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\jkang1\.conda\envs\Capacity_venv\lib\site-packages\httpx\_transports\default.py", line 77, in
map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

Versions

Version:             2.7.4
API version:         0.8.4
Python version:      3.8.13
Git commit:          30db76e7
Built:               Thu, Dec 22, 2022 2:55 PM
OS/Arch:             win32/AMD64
Profile:             remote-desktop
Server type:         cloud

Additional context

No response
@johnkangw johnkangw added bug Something isn't working status:triage labels Dec 27, 2022
@zanieb
Copy link
Contributor

zanieb commented Dec 28, 2022 •
edited
Loading

Please see #7442 (note this is unrelated to Windows so I've removed that from the title for clarity)

@zanieb zanieb changed the title Prefect agent loses connection on Windows Terminal Prefect agent loses connection Dec 28, 2022
@johnkangw
Copy link
Author

@madkinsz Is this the same issue? It is occurring with 2.7.4.

@zanieb
Copy link
Contributor

zanieb commented Dec 28, 2022

Yes it looks like the same issue. It's an issue in an upstream HTTP library that is unresolved but we've added retries and the ability to disable http2 to work around the problem for you.

@johnkangw
Copy link
Author

@madkinsz Appreciate the heads up. I'll look at disabling http2 if this issue persists.

@zanieb zanieb closed this as completed Dec 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zanieb @johnkangw