python - 获取调用中的 aiohttp ClientSession 错误：看到的标头字节太多；返回检测到的溢出

Question

描述错误 在aiohttp ClientSession 中发送调用时，too many header bytes seen; overflow detected返回错误。

重现 没有要重现的确切代码，因为它使用专有资源。编码：

import asyncio
import aiohttp

loop = asyncio.get_event_loop()

url = "https://url.com"
bodies = [{"body": 1}, {"body": 2}, {"body": 3}]

async def request(body, session):
    async with session.post(url, json=body) as resp:
        return await resp.json()
    #return await es_client.search(index=es_index, doc_type=es_doc_type, body=query)

async def msearch(bodies):
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.ensure_future(request(body, session)) for body in bodies]
        return await asyncio.gather(*tasks)

results = loop.run_until_complete(msearch(bodies))

预期行为 此代码应返回 json 响应。

它适用于大多数电话，但如果结果？/标题？/...？一个电话越来越大，这个消息被抛出？

日志/回溯

Traceback (most recent call last):
  File "/Users/stijngeuens/.virtualenvs/findme/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 847, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/Users/stijngeuens/.virtualenvs/findme/lib/python3.7/site-packages/aiohttp/streams.py", line 591, in read
    await self._waiter
  File "/Users/stijngeuens/.virtualenvs/findme/lib/python3.7/site-packages/aiohttp/client_proto.py", line 201, in data_received
    messages, upgraded, tail = self._parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 523, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message='too many header bytes seen; overflow detected'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Users/stijngeuens/.pyenv/versions/3.7.3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-25-8dd725416572>", line 11, in <module>
    results = loop.run_until_complete(msearch(bodies))
  File "/Users/stijngeuens/.pyenv/versions/3.7.3/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "<ipython-input-25-8dd725416572>", line 9, in msearch
    return await asyncio.gather(*tasks)
  File "<ipython-input-25-8dd725416572>", line 2, in request
    async with session.post(url, json=body) as resp:
  File "/Users/stijngeuens/.virtualenvs/findme/lib/python3.7/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/Users/stijngeuens/.virtualenvs/findme/lib/python3.7/site-packages/aiohttp/client.py", line 504, in _request
    await resp.start(conn)
  File "/Users/stijngeuens/.virtualenvs/findme/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 852, in start
    message=exc.message, headers=exc.headers) from exc
aiohttp.client_exceptions.ClientResponseError: 400, message='too many header bytes seen; overflow detected', url=URL('https://url.com')

你的 Python 版本

Python 3.7.3 (default, Oct 18 2019, 10:37:04) 

您的 aiohttp/yarl/multidict 发行版版本

aiohttp==3.6.2
yarl==1.3.0
multidict==4.5.2

附加上下文 在 Mac OS 上运行。Pycharm 和 Docker 中的错误 Call is going to a local ES instance for development 使用客户端库 (ClientSession)

标签： pythonpython-3.xaiohttp

Answer 1

aiohttp.client_exceptions.ClientResponseError: 400, message='too many header bytes seen; overflow detected', url=URL('https://url.com')

这意味着服务器正在回复400 Bad Request错误，并且服务器抱怨标头格式不正确。

根据该消息，服务器很可能是以期望一定数量的标头的方式进行编程的。当前请求使用以下标头发送：

{
    "Host": "example.com",
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "User-Agent": "Python/3.8 aiohttp/3.7.4.post0",
    "Content-Length": "11",
    "Content-Type": "application/json",
}

您可以通过以下方式检查：

async def request(body, session):
    async with session.post(url, json=body) as resp:
        print(dict(resp.request_info.headers))
        return await resp.json()

我会检查服务器的文档，看看他们的端点到底post期望什么。如果您正在对某个端点进行逆向工程，我建议您还复制原始请求发送到服务器的标头。这有望解决您的问题。

我无法进一步深入研究这一点，因为我假设url.com这只是您放置的一个“临时”域来替代原始域（可以理解，如果它是出于隐私目的）。