Packet sequence number wrong

[houmie]

Hello,

There are some serious instability with ASGI server. The server runs for a few days without any issues, then it starts crashing for no particular reason.

crud_server.py

db = databases.Database(settings.SQLALCHEMY_DATABASE_URI)

async def get_servers(is_maintenance: bool) -> List[Server]:
    query = ServerTable.select().where(ServerTable.c.is_maintenance == is_maintenance)
    if not db.is_connected:
        await db.connect()
    return await db.fetch_all(query)

ServerTable.py

metadata = MetaData(schema=settings.DBNAME_MAIN)

ServerTable = sqlalchemy.Table(
    "server",
    metadata,
    Column("id", String(15), primary_key=True),
    Column("name", String(100), nullable=False),
    Column("country", String(100), nullable=False),
    Column("group_master", String(15), nullable=True),
    Column("group_slave", String(15), nullable=True, index=True),
    Column("ip", String(length=45), nullable=False),
    Column("max_connections", Integer(), nullable=False),
    Column(
        "is_maintenance",
        Boolean(),
        nullable=False,
        server_default=text("0"),
        default=text("0"),
        index=True,
    ),
    Column("dow_maintenance", String(3), nullable=True),
    Column("hour_maintenance", Integer(), nullable=True),
    Column("minutes_maintenance", Integer(), nullable=True),
    Column("datetime_maintenance", TIMESTAMP(), nullable=True),
)

Error:

  File "/home/admin/tg/app/routers/server_status.py", line 25, in server_status
    servers = await crud_server.get_servers(is_maintenance=False)
  File "/home/admin/tg/app/database/crud_server.py", line 15, in get_servers
    return await db.fetch_all(query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/core.py", line 140, in fetch_all
    return await connection.fetch_all(query, values)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/core.py", line 239, in fetch_all
    return await self._connection.fetch_all(built_query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/backends/mysql.py", line 108, in fetch_all
    await cursor.execute(query, args)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/cursors.py", line 239, in execute
    await self._query(query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/cursors.py", line 457, in _query
    await conn.query(q)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 428, in query
    await self._read_query_result(unbuffered=unbuffered)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 622, in _read_query_result
    await result.read()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 1105, in read
    first_packet = await self.connection._read_packet()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 574, in _read_packet
    raise InternalError(
pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1

I would really appreciate a hint, what I'm doing wrong.
Thank you so much

[houmie]

Hello,

Unfortunately, I see the issue was closed and moved here for further discussion. The ASGI server seems unstable.

I have restarted the ASGI server seven days ago and it ran alright for four days. Then it crashed again and remained in that state:

A simple versions = await db.fetch_all(query) causes an pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1

Please be so kind and advice what I should do next to get to the bottom of this,
Thanks,
Houman

More details below:

[2021-05-16 06:06:00 +0000] [799] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/middleware/sessions.py", line 75, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette_wtf/csrf.py", line 140, in dispatch
    return await call_next(request)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/fastapi/routing.py", line 201, in app
    raw_response = await run_endpoint_function(
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/fastapi/routing.py", line 148, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/admin/tegant/app/main.py", line 98, in check_version
    versions = await db.fetch_all(query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/core.py", line 140, in fetch_all
    return await connection.fetch_all(query, values)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/core.py", line 239, in fetch_all
    return await self._connection.fetch_all(built_query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/backends/mysql.py", line 108, in fetch_all
    await cursor.execute(query, args)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/cursors.py", line 239, in execute
    await self._query(query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/cursors.py", line 457, in _query
    await conn.query(q)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 428, in query
    await self._read_query_result(unbuffered=unbuffered)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 622, in _read_query_result
    await result.read()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 1105, in read
    first_packet = await self.connection._read_packet()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 574, in _read_packet
    raise InternalError(
pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1

[leon1114]

Hello,

I'm having the exact same problem.
Have you solved the problem? I've been digging on this for a few days but I have no clue..

Thanks

[houmie]

Yes, I have solved it in a radical way.

I had further discussions with Fast-API maintainers and they said that mysql async drivers are too old and not getting maintained. They convinced me to switch to Postgres and use asyncpg instead.

I have made the jump and had never any problems anymore.

[aminalaee]

@leon1114 As @houmie explained this isn't an issue in databases but the driver aiomysql itself.

[leon1114]

@houmie @aminalaee Thank you all for the replies. Maybe I should also migrate to postgres or use mysql sync driver in an async web app 😢

[houmie]

@aminalaee I just wished someone here from the maintainer had told me that before. ;-) When I raised the issue back in May, the issue was converted to discussion and completely ignored. This is why I don't like the discussion feature on Github.

[aminalaee]

@houmie You're right, the maintainer team has been a little inactive, but I hope that can be changed.
If you can comment your final solution/workaround then we can mark it as answer and close this.

[aminalaee]

aiomysql is not up-to-date and has open issues, as suggested you should switch the database driver.