时间戳

首页 > 解决方案 > 无法从 Expedia.com 抓取数据

python-3.x - 无法从 Expedia.com 抓取数据

问题描述

我正在使用 spyder 从 Expedia 抓取数据，它现在正在我的本地系统上运行。最初，它显示与 expedia.com 相同的问题，然后我切换到 expedia.co.in。它显示此问题

========================================

Python== 3.6
Django==2.0.2
django-import-export==1.0.0
lxml==4.1.1
requests==2.9.1
Scrapy==1.5.0

===============================================

错误日志是：

[15/Jun/2018 07:04:37] "GET /myapp/expedia HTTP/1.0" 200 512
:0: UserWarning: You do not have a working installation of the service_identity module: 'cannot import name 'opentype''.  Please install it from <https://pypi.python.org/pypi/service_identity> and make sure all of its dependencies are satisfied.  Without the service_identity module, Twisted can perform only rudimentary TLS client hostname verification.  Many valid certificate/hostname mappings may be rejected.
2018-06-15 07:04:46 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-06-15 07:04:46 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.4.0, Python 3.5.2 (default, Nov 23 2017, 16:37:01) - [GCC 5.4.0 20160609], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptography 2.2.2, Platform Linux-4.4.0-1052-aws-x86_64-with-Ubuntu-16.04-xenial
2018-06-15 07:04:46 [scrapy.crawler] INFO: Overridden settings: {'FEED_URI': 'data.csv', 'SPIDER_LOADER_WARN_ONLY': True, 'USER_AGENT': 'Safari/537.36', 'FEED_FORMAT': 'csv'}
2018-06-15 07:04:46 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.memusage.MemoryUsage']
2018-06-15 07:04:46 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-06-15 07:04:46 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-06-15 07:04:46 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-06-15 07:04:46 [scrapy.core.engine] INFO: Spider opened
2018-06-15 07:04:46 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-06-15 07:04:46 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-06-15 07:04:46 [scrapy.core.downloader.tls] WARNING: Remote certificate is not valid for hostname "www.expedia.co.in"; 'www.expedia.com'!='www.expedia.co.in'
2018-06-15 07:04:46 [scrapy.core.engine] DEBUG: Crawled (429) <GET https://www.expedia.co.in/Austin-Hotels-InterContinental-Stephen-F-Austin.h450973.Hotel-Information?chkin=11%2F06%2F2018&chkout=12%2F06%2F2018&rm1=a2&hwrqCacheKey=813c7608-e4c5-44e4-bd49-d7f0dad48bacHWRQ1528694186089&cancellable=false&regionId=6059085&vip=false&c=01dd4254-f5cf-4a9e-b3f4-1ffa4747252e&&exp_dp=16473.05&exp_ts=1528694186049&exp_curr=INR&swpToggleOn=false> (referer: None)
2018-06-15 07:04:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.expedia.co.in/Austin-Hotels-InterContinental-Stephen-F-Austin.h450973.Hotel-Information?chkin=11%2F06%2F2018&chkout=12%2F06%2F2018&rm1=a2&hwrqCacheKey=813c7608-e4c5-44e4-bd49-d7f0dad48bacHWRQ1528694186089&cancellable=false&regionId=6059085&vip=false&c=01dd4254-f5cf-4a9e-b3f4-1ffa4747252e&&exp_dp=16473.05&exp_ts=1528694186049&exp_curr=INR&swpToggleOn=false>: HTTP status code is not handled or not allowed
2018-06-15 07:04:46 [scrapy.core.engine] INFO: Closing spider (finished)
2018-06-15 07:04:46 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 538,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 18177,
 'downloader/response_count': 1,
 'downloader/response_status_count/429': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 6, 15, 7, 4, 46, 611257),
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/429': 1,
 'log_count/DEBUG': 2,
 'log_count/INFO': 8,
 'log_count/WARNING': 1,
 'memusage/max': 50315264,
 'memusage/startup': 50315264,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2018, 6, 15, 7, 4, 46, 396958)}
2018-06-15 07:04:46 [scrapy.core.engine] INFO: Spider closed (finished)
[15/Jun/2018 07:04:46] "POST /myapp/expedia HTTP/1.0" 200 44

标签： python-3.xweb-scrapingscrapy

解决方案

正如日志所说，你得到了HTTP 429 - Too Many Requests，这意味着他们阻止了你。

请阅读他们Terms of Use的内容，以免违反他们的规则。

推荐阅读