python - Python 抓取:错误 54“对等方重置连接”
问题描述
我编写了简单的脚本来从多个网站获取 html。虽然直到昨天我对脚本没有任何问题。它突然开始抛出异常。
Traceback (most recent call last):
File "crowling.py", line 45, in <module>
result = requests.get(url)
File "/Users/gen/.pyenv/versions/3.7.1/lib/python3.7/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/Users/gen/.pyenv/versions/3.7.1/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/gen/.pyenv/versions/3.7.1/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/Users/gen/.pyenv/versions/3.7.1/lib/python3.7/site-packages/requests/sessions.py", line 685, in send
r.content
File "/Users/gen/.pyenv/versions/3.7.1/lib/python3.7/site-packages/requests/models.py", line 829, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/Users/gen/.pyenv/versions/3.7.1/lib/python3.7/site-packages/requests/models.py", line 754, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(54, 'Connection reset by peer')", ConnectionResetError(54, 'Connection reset by peer'))
脚本的主要部分是这个。
c = 0
#urls is the list of urls as strings
for url in urls:
result = requests.get(url)
c += 1
with open('htmls/p{}.html'.format(c),'w',encoding='UTF-8') as f:
f.write(result.text)
列表 url 是由我的其他代码生成的,我检查了 url 是否正确。此外,异常发生的时间也不是恒定的。有时它会在抓取第 20 个 html 时停止,有时它会持续到第 80 个然后停止。由于在没有更改代码的情况下突然出现异常,我猜测该异常是由于 Internet 连接造成的。但是,我想确保脚本稳定运行。有没有可能导致此错误的原因?
解决方案
如果您确定 URL 正确并且是间歇性连接问题,您可以在失败后重试连接:
c = 0
#urls is the list of urls as strings
for url in urls:
trycnt = 3 # max try cnt
while trycnt > 0:
try:
result = requests.get(url)
c += 1
with open('htmls/p{}.html'.format(c),'w',encoding='UTF-8') as f:
f.write(result.text)
trycnt = 0 # success
except ChunkedEncodingError as ex:
if trycnt <= 0: print("Failed to retrieve: " + url + "\n" + str(ex)) # done retrying
else: trycnt -= 1 # retry
time.sleep(0.5) # wait 1/2 second then retry
# go to next URL
推荐阅读
- c# - Instagram 获取电子邮件(2019 修复?)
- javascript - 如何使用 JavaScript 从字符串中删除所有 2019 表情符号字符?
- python - Numpy 3d数组从每一行删除第一个条目
- awk - 使用 sed/awk 在目录之间添加栏
- assembly - 使用汇编语言写入视频内存时无效的段覆盖
- python - Tkinter:将滚动条与画布相结合
- ios - 如何在IOS swift中将Double转换为Int
- javascript - v-for 和 :src 的问题指向 Vuejs 中的反应变量
- jquery - JavaScript openCallBack的意义和目的:函数contextMenu
- laravel - 在 laravel 中写错误的好方法是什么