python - 处理来自 python 请求的异常
问题描述
我的函数是用线程执行的:
def getdata(self, page, ...):
tries = 10
for n in range(tries):
try:
...
datarALL = []
url = 'http://website/...'.format(...)
responsedata = requests.get(url, data=data, headers=self.hed, verify=False)
responsedata.raise_for_status()
if responsedata.status_code == 200: # 200 for successful call
...
if ...
break
except (ChunkedEncodingError, requests.exceptions.HTTPError) as e:
print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
if n == tries - 1:
print ("page {0} could not be imported. Max retried reached.".format(page))
os._exit(1) #One thread max retried - close all threads and
return datarALL
如下:
with ThreadPoolExecutor(max_workers=num_of_workers) as executor:
futh = [(executor.submit(self.getdata, page,...)) for page in pages]
for data in as_completed(futh):
datarALL.extend(data.result())
print ("Finished generateing data.")
return datarALL
有时我会遇到意想不到的异常ConnectionResetError: [Errno 104] Connection reset by peer
,例如:关闭我的程序。我想更改我的代码,以便无论发生哪种异常,线程都会重试直到if n == tries - 1:
满足。我不希望我的线程由于随机异常而关闭。
我阅读了请求异常信息页面,但我不知道如何在不手动列出所有异常的情况下捕获所有异常。有没有通用的方法来做到这一点?
基本上我想要类似的东西:
except (ALL EXCEPTIONS from Requests) as e:
print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
if n == tries - 1:
print ("page {0} could not be imported. Max retried reached.".format(page))
os._exit(1) #One thread max retried - close all threads and
return datarALL
我怎样才能做到这一点?
编辑: 使用
except Exception as e:
print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
if n == tries - 1:
print ("page {0} could not be imported. Max retried reached.".format(page))
os._exit(1) #One thread max retried - close all threads and
return datarALL
不抓。它给了我这个:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 331, in _error_catcher
yield
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 640, in read_chunked
chunk = self._handle_chunk(amt)
File "/home/ubuntu/.local/lib/python3.5/site-packages/urllib3/response.py", line 595, in _handle_chunk
returned_chunk = self._fp._safe_read(self.chunk_left)
File "/usr/lib/python3.5/http/client.py", line 607, in _safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
循环不会重试。运行终止...
编辑2:
except requests.exceptions.RequestException as e:
print ("page #{0} run #{1} failed. Returned status code {2}. Reason: {3}. Msg: {4}. Retry.".format(page, n, responsedata.status_code, responsedata.reason, sys.exc_info()[0]))
if n == tries - 1:
print ("page {0} could not be imported. Max retried reached.".format(page))
os._exit(1) #One thread max retried - close all threads and
return datarALL
也没有赶上ConnectionResetError: [Errno 104] Connection reset by peer
上面列出的。
解决方案
捕获所有异常通常被认为是一种不好的做法,因为它可能会隐藏一些问题。
也就是说,Python 异常受益于继承,并且捕获基本异常将捕获从该基本异常继承的每个异常。
有关详细信息,请参阅Python 标准异常层次结构。
您可以看到根异常是BaseException
,但永远不应该捕获这个异常,因为它会捕获Ctrl+C
中断和生成器退出。如果你想捕捉每一个异常类型,你可以捕捉Exception
.
您也可能只想从requests
. 在这种情况下,根据doc,可以通过捕获requests
模块的基本异常来完成:RequestException
如果你想同时捕获requests
异常和ConnectionResetError
(这是一个 Python 标准异常),你必须在你的except
子句中同时指定:
except (requests.exceptions.RequestException,
ConnectionResetError) as err:
# some code
或者,如果您想不那么具体并捕获所有可能的连接错误,您可以使用ConnectionError
而不是ConnectionResetError
. (请参阅异常层次结构)
最后,您可能希望对每种异常类型做出不同的反应。在这种情况下,您可以这样做:
try:
# something
except ConnectionError as err:
# manage connection errors
except requests.exceptions.RequestException as err:
# manage requests errors
推荐阅读
- php - PHP存储过程执行不完整
- java - 更改系统键盘语言的 Java 应用程序
- html - 如何正确设置 getElementById 对象?
- xcode - XCode Instruments:Time Profiler 在录制后立即停止
- c++ - C++初始化继承的模板基类
- python - Django 重定向 - 不是有效的视图函数或模式名称错误
- javascript - 从多个/不同文件输入访问文件到 JS Var
- ios - iOS RxSwift 4.2 how to implement Reactive extension of CBCentralManager using DelegateProxy?
- matlab - 用符号变量计算矩阵
- python-3.x - 尝试单击元素时出现“ElementNotVisibleException”和“InvalidSelectorException:无效选择器:不允许复合类名”