python - 为什么我无法使用 BeautifulSoup 从网站获取数据?我收到超时错误
问题描述
我试图从以下网站获取数据,但出现如下所示的错误。PFB 的代码相同。
from urllib2 import urlopen
import bs4 as bs
response = urlopen('http://www.mec.ac.in/mec/stats2018.php')
html = response.read()
soup = bs.BeautifulSoup(response,'lxml')
print soup.title
PFB错误:
Traceback (most recent call last):
File "et.py", line 3, in <module>
response = urlopen('http://www.mec.ac.in/mec/stats2018.php')
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
恢复此错误后如何检索数据?
解决方案
服务器专门“阻止”带有User-Agent
包含Python-urllib
字符串的标头的请求(默认情况下urllib2
/urllib
发送):
In [1]: import requests
In [2]: url = "http://www.mec.ac.in/mec/stats2018.php"
In [3]: requests.get(url, headers={'User-Agent': 'Python-urllib/2.6'})
Out[3]: <Response [403]>
In [4]: requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'})
Out[4]: <Response [200]>
推荐阅读
- ffmpeg - 使用 FFmpeg 录制屏幕
- log4j2 - 如何将 log4j.property 设置为 .jar 位置
- model-view-controller - stimulsoft mvc core如何动态报告图像大小
- c - 传递给函数的值已更改
- javascript - 将数组前一个对象分配给下一个元素并更新最后一个
- haskell - 从脚本本身中获取 Haskell Stack & Turtle 脚本的源目录
- ruby-on-rails - 在 ruby 中使用正则表达式拆分数据库查询字符串
- c - 获得不太理想的解决方案 TSP
- php - 如何在 laravel 中使用 createmany 插入 2 个数组
- c# - 有没有更好的方法来定位辅助窗口相对于主窗口的启动位置?