python - Requests/aiohttp:关闭响应对象
问题描述
我对在和.close()
中都需要响应对象感到有些困惑。(请注意,这是一个单独的实例方法,而不是——我说的是响应对象本身。)requests
aiohttp
session.close()
Response
(requests
) 或ClientResponse
( )是否aiohttp
需要显式调用.close()
?- 如果不是,使用响应本身作为上下文管理器的目的是什么?(
async with session.request('GET', 'https://www.pastebin.com'
) 下面。)如果它被隐式关闭,为什么要为此定义两个 dunder 方法,如下所示?
一些简单的测试(如下)似乎暗示响应在会话上下文管理器中定义时会自动关闭。 (它本身调用or 。但这是 Session 的关闭,而不是 Response 对象。self.close()
)__exit__
__aexit__
例子 -requests
>>> import requests
>>>
>>> with requests.Session() as s:
... resp = s.request('GET', 'https://www.pastebin.com')
... resp.raise_for_status()
... print(resp.raw.closed) # `raw` is urllib3.response.HTTPResponse object
... print(resp.raw._pool)
... print(resp.raw._connection)
... c = resp.text
...
True
HTTPSConnectionPool(host='pastebin.com', port=443)
None
>>>
>>> while 1:
... print(resp.raw.closed)
... print(resp.raw._pool)
... print(resp.raw._connection)
... break
...
True
HTTPSConnectionPool(host='pastebin.com', port=443)
None
例子 -aiohttp
>>> import asyncio
>>> import aiohttp
>>>
>>> async def get():
... async with aiohttp.ClientSession() as s:
... # The response is already closed after this `with` block.
... # Why would it need to be used as a context manager?
... resp = await s.request('GET', 'https://www.pastebin.com')
... print(resp._closed)
... print(resp._connection)
... print(resp._released)
... c = await resp.text()
... print()
... print(resp._closed)
... print(resp._connection)
... print(resp._released)
... return c
...
>>> c = asyncio.run(get()) # Python 3.7 +
False
Connection<ConnectionKey(host='pastebin.com', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)>
False
True
None
False
这是 的来源requests.models.Response
。“通常不需要显式调用”是什么意思?有哪些例外?
def close(self):
"""Releases the connection back to the pool. Once this method has been
called the underlying ``raw`` object must not be accessed again.
*Note: Should not normally need to be called explicitly.*
"""
if not self._content_consumed:
self.raw.close()
release_conn = getattr(self.raw, 'release_conn', None)
if release_conn is not None:
release_conn()
解决方案
Requests
: 你不需要显式调用close()
. 完成后请求将自动关闭,因为它基于 urlopen(这就是为什么resp.raw.closed
是 True),这是我观看后的简化session.py
代码adapters.py
:
from urllib3 import PoolManager
import time
manager = PoolManager(10)
conn = manager.connection_from_host('host1.example.com')
conn2 = manager.connection_from_host('host2.example.com')
res = conn.urlopen(url="http://host1.example.com/",method="get")
print(len(manager.pools))
manager.clear()
print(len(manager.pools))
print(res.closed)
#2
#0
#True
然后做了__exit__
什么?它用于清除PoolManager(self.poolmanager=PoolManager(...))
和proxy
。
# session.py
def __exit__(self, *args): #line 423
self.close()
def close(self): #line 733
for v in self.adapters.values():
v.close()
# adapters.py
# v.close()
def close(self): #line 307
self.poolmanager.clear()
for proxy in self.proxy_manager.values():
proxy.clear()
所以你什么时候需要使用close()
,正如注释所说的释放连接回池,因为DEFAULT_POOLSIZE = 10
(http/https是独立的)。这意味着如果您想通过一个会话访问超过 10 个网站,您可以选择关闭一些您不需要的网站,否则当您有一个会话时,管理员将关闭从第一个到最新的连接。但实际上你不需要关心这个,你可以指定池大小,它不会浪费太多时间来重建连接。
aiohttp
aiohttp.ClientSession() 对所有请求使用一个 TCPConnector。当它触发时__aexit__
, self._connector
将被关闭。
编辑:s.request()
设置了来自主机的连接,但没有得到响应。await resp.text()
只有在得到响应后才能执行,如果您没有执行此步骤(等待响应),您将退出而没有响应。
if connector is None: #line 132
connector = TCPConnector(loop=loop)
...
self._connector = connector #line 151
# connection timeout
try:
with CeilTimeout(real_timeout.connect,loop=self._loop):
assert self._connector is not None
conn = await self._connector.connect(
req,
traces=traces,
timeout=real_timeout
)
...
async def close(self) -> None:
if not self.closed:
if self._connector is not None and self._connector_owner:
self._connector.close()
self._connector = None
...
async def __aexit__(self,
...) -> None:
await self.close()
这是显示我所说的代码
import asyncio
import aiohttp
import time
async def get():
async with aiohttp.ClientSession() as s:
# The response is already closed after this `with` block.
# Why would it need to be used as a context manager?
resp = await s.request('GET', 'https://www.stackoverflow.com')
resp2 = await s.request('GET', 'https://www.github.com')
print("resp:",resp._closed)
print("resp:",resp._connection)
print("resp2:",resp2._closed)
print("resp2:",resp2._connection)
s.close()
print(s.closed)
c = await resp.text()
d = await resp2.text()
print()
print(s._connector)
print("resp:",resp._closed)
print("resp:",resp._connection)
print("resp2:",resp2._closed)
print("resp2:",resp2._connection)
loop = asyncio.get_event_loop()
loop.run_until_complete(get()) # Python 3.5 +
#dead loop