Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aiormq 6.2.3 fails to clean up connection if first frame_receiver.get_frame() fails #138

Open
TBBle opened this issue Apr 11, 2022 · 0 comments

Comments

@TBBle
Copy link

TBBle commented Apr 11, 2022

We have a test in our test suite using aio-pika 7.1.2 and hence aiormq 6.2.3, which feeds an expired SSL certificate into the connect request.

The trace for the (expected) exception is:

ERROR    handler.client_service:client_service.py:664 [SSL: SSLV3_ALERT_CERTIFICATE_EXPIRED] sslv3 alert certificate expired (_ssl.c:2633)
Traceback (most recent call last):
  File "C:\Projects\PS\services\projects\handler\src\handler\client_service.py", line 648, in serve_forever
    async with self.message_handler as message_source:
  File "C:\Projects\PS\services\projects\handler\src\handler\video_generation_message_source\rabbitmq.py", line 54, in __aenter__
    await self.__listener.connect()
  File "C:\Projects\PS\services\lib\shared\src\ps\shared\rabbitmq.py", line 95, in connect
    self.connection = await aio_pika.connect_robust(rmq_url)
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aio_pika\robust_connection.py", line 325, in connect_robust
    await connection.connect(
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aio_pika\robust_connection.py", line 133, in connect
    await super().connect(
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aio_pika\connection.py", line 135, in connect
    await self._make_connection(timeout=timeout, **kwargs)
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aio_pika\connection.py", line 114, in _make_connection
    connection: aiormq.abc.AbstractConnection = await asyncio.wait_for(
  File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 442, in wait_for
    return await fut
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\connection.py", line 731, in connect
    await connection.connect(client_properties or {})
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\base.py", line 162, in wrap
    return await self.create_task(func(self, *args, **kwargs))
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\abc.py", line 40, in __inner
    return await self.task
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\connection.py", line 397, in connect
    _, _, frame = await frame_receiver.get_frame()
  File "C:\Users\paulh\AppData\Local\pypoetry\Cache\virtualenvs\handler-vLGyY0no-py3.9\lib\site-packages\aiormq\connection.py", line 143, in get_frame
    frame_header = await countdown(self.reader.readexactly(1))
  File "C:\Program Files\Python39\lib\asyncio\tasks.py", line 479, in wait_for
    return fut.result()
  File "C:\Program Files\Python39\lib\asyncio\streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "C:\Program Files\Python39\lib\asyncio\streams.py", line 517, in _wait_for_data
    await self._waiter
  File "C:\Program Files\Python39\lib\asyncio\sslproto.py", line 528, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "C:\Program Files\Python39\lib\asyncio\sslproto.py", line 199, in feed_ssldata
    chunk = self._sslobj.read(self.max_size)
  File "C:\Program Files\Python39\lib\ssl.py", line 888, in read
    v = self._sslobj.read(len)
ssl.SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_EXPIRED] sslv3 alert certificate expired (_ssl.c:2633)

However, observed on Python 3.9 on Windows, the exception raised at aiormq\connection.py line 397 occurs after the asyncio.open_connection call on line 380, but outside the big try-catch on lines 408-449 that ensures that any exception raised will close the sockets, and hence leaks the socket without ever putting the connection somewhere we can close it (because aio_pika has wrapped this call in a factory function, so even if the best solution was to call close even on a failed connection, aio-pika would have to do that manually (as would any other user of this library))

The relevant caller in aio-pika, for reference:

    async def _make_connection(
        self, *, timeout: TimeoutType = None, **kwargs: Any
    ) -> aiormq.abc.AbstractConnection:
        connection: aiormq.abc.AbstractConnection = await asyncio.wait_for(
            aiormq.connect(self.url, **kwargs), timeout=timeout,
        )
        connection.closing.add_done_callback(
            partial(self._on_connection_close, connection),
        )
        await connection.ready()
        return connection

The symptom for us is that a later test fails with

E               ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x0000022DB33E7FA0>

because the leaked SSLProtocolTransport is not detected until tear-down of a later test (due to how pytest-aiohttp cleans up its loops, I suspect).

I found the trivial fix was to move all the code after the await asyncio.open_connection, i.e. lines 391 to 407 down inside the big try-catch, i.e. indented once and moved after line 408.

That way a failure in that block will still trigger the clean-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant