python - 仅在 AWS 实例上阻止使用 Python 进行抓取

问题描述

我已经构建了一个脚本来抓取www.tesco.com杂货店结果页面，示例链接：

https://www.tesco.com/groceries/en-GB/search?query=kitkat

不幸的是，我的 Python 脚本被服务器阻止（常规获取请求）。我什至尝试在我的机器上使用 CURL 进行故障排除：

curl htpps://www.tesco.com

但我得到以下回应：

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
 
You don't have permission to access "http&#58;&#47;&#47;dce&#45;homepage&#46;tesco&#46;com&#47;" on this server.<P>
Reference&#32;&#35;18&#46;496cd417&#46;1592645071&#46;44e961c
</BODY>
</HTML>

当尝试使用带有标准标题的 Postman 时，我得到 200 OK 响应。在我的脚本中，我尝试使用与 Postman 相同的标题并且我得到 200 OK，但前提是我在本地 PC 上使用它。当我在 AWS 上启动一个新的实例时——Ubuntu 18.04 或类似的免费层——CURL 得到 404 如上所述。理想情况下，我希望我的脚本可以在 AWS 上运行。运行时，脚本不起作用 - 只是挂起。当我打断它时，我得到以下信息：

^CTraceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ttest.py", line 18, in <module>
    results = requests.get(url, headers = headers)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 874, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
KeyboardInterrupt

也许 tesco.com 已经禁止所有 AWS 实例抓取他们的网站？

这是在我的 PC 上运行但不在 AWS 实例上运行的代码。

'编辑' - 我已经尝试在标题中使用我们的 cookie - 仍然没有运气。


import requests
headers = {'User-Agent': 'PostmanRuntime/7.25.0',
'Accept': '*/*',
'Cache-Control': 'no-cache',
'Host': 'www.tesco.com',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
'Cookie': 'bm_sz=04919BE521C5C4D8ADF4617D5250A484~YAAQrpxkX+b8IYVyAQAA/VQr0QgTg5gDEXUmuUfa0qqtHv0QHHZjtL4gcSJ9RA7hoaEXJOTp1DYPb9xCrGwP37BrvtUY2kCKB7PqvVLXAXnfrt9F0ZiEPj10SiSVXZRZj8klW46ZA7Ho/0XtWlsO2aFX1MPkmD2/C10cDH6E1PgeO9EUNkZi9uPu109p4DE=; _abck=5621BD87FE69A39458BD0AB267BB9A81~-1~YAAQrpxkX+f8IYVyAQAA/VQr0QTSvxcBlxnRsND9THtPksH0EbfK/A3XkW0xT9oCk0Bj1ewbVDXr3PqtBjR7hHO6h6IXMvC2XID5RrAk0gVEKGwm9RDyBWyvp6hnPzicHMH6tTUIZdYLmssjIBAJ2WnpBkKUuF0YbX45V4H8d3m6u8FOhyqZewFyT1+Yvh14NDHwmDw4Yb4hQkLPglrkzt8LV39SpfSjjGkWMjyX4l967aCe+SHK5hjcTIz9bjSAoOQNqFWR5ATMnfBDSLOfaAQ4Dic=~-1~-1~-1; atrc=48693e75-78d9-4fce-85d0-9a0a50232644; _csrf=2wH2UKiamS-tjvd4hERekcG2',
'Referer': 'http://www.tesco.com/'

}

url = 'https://www.tesco.com/groceries/en-GB/search?query=kitkat'
results = requests.get(url, headers = headers)

print(results.status_code)

www.tesco.com robots.txt不禁止抓取：


Sitemap: https://www.tesco.com/UK.sitemap.xml
 
User-agent: *
Disallow: *reviews/submission/*
Disallow: *&sortBy*
Disallow: *promotion=*
Disallow: *currentModal*
Disallow: *active-tab*
Disallow: *include-children*
Disallow: *new&new*
Disallow: /groceries/*reviews/submission

编辑：

我已将无头 chrome webbrowser 下载到 AWS 上的 ubuntu 服务器实例，并尝试截取 tesco.com 的屏幕截图。我得到以下错误：

为了澄清起见，我尝试浏览 https 地址 - 这不重要，因为我确定它具有 https 重定向。

标签： pythonweb-scrapingpython-requestspostman

python - 仅在 AWS 实例上阻止使用 Python 进行抓取 - 阻止请求

问题描述

解决方案

推荐阅读