时间戳

首页 > 解决方案 > Scrapy 不会产生 DNS 查找失败网站的网址

python - Scrapy 不会产生 DNS 查找失败网站的网址

问题描述

我有一个 url 列表，这些 url 被重定向到文本文件中的另一个 url。我想获取所有重定向的 url。所以我做了一个从文本文件打开 url 的蜘蛛。现在我收到一些错误，例如“DNS 查找失败”或“无路由”。我直接在浏览器上检查了这些 url，发现这些 url 得到“ip address not found error”。但是，无论错误如何，我都希望对每个重定向的 url 进行抓取。任何解决方案来实现这一目标？

这是我跑的蜘蛛

import scrapy

class AwesomeSpiderSpider(scrapy.Spider):
    name = 'web_uk'
# opening the list of urls that gets redirected.
    f = open("urls.txt")
    start_urls = [url.strip() for url in f.readlines()]
    f.close()


    def parse(self,response):

        item = {}
        item['Web Address'] = response.url
        yield item

这是输出

2019-07-04 03:02:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://gatesidedevelopments.com/> (referer: None) ['cached']
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://honka.com/gb/en/>
{'Web Address': 'https://honka.com/gb/en/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.dapconstruction.co.uk/>
{'Web Address': 'https://www.dapconstruction.co.uk/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.virtueprojects.com>
{'Web Address': 'http://www.virtueprojects.com'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://waynemoore.com/>
{'Web Address': 'https://waynemoore.com/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.avenuenorth.co.uk/>
{'Web Address': 'http://www.avenuenorth.co.uk/'}
2019-07-04 03:02:03 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 1 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:03 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.mic.uk.com>
{'Web Address': 'http://www.mic.uk.com'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.vlconstruction.co.uk/>
{'Web Address': 'https://www.vlconstruction.co.uk/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.whitehalloflondon.co.uk>
{'Web Address': 'http://www.whitehalloflondon.co.uk'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.vandthomes.com>
{'Web Address': 'http://www.vandthomes.com'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.atlanticdwellings.com/>
{'Web Address': 'https://www.atlanticdwellings.com/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.rgfhomeimprovements.co.uk>
{'Web Address': 'http://www.rgfhomeimprovements.co.uk'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://leonoc.co.uk/>
{'Web Address': 'https://leonoc.co.uk/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.home-refurbishments.co.uk/>
{'Web Address': 'http://www.home-refurbishments.co.uk/'}
2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://gatesidedevelopments.com/>
{'Web Address': 'https://gatesidedevelopments.com/'}
2019-07-04 03:02:07 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 1 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:07 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
2019-07-04 03:02:11 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 2 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.45 Safari/537.36
2019-07-04 03:02:12 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 2 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:12 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
2019-07-04 03:02:17 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:17 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://www.galkivconstruction.co.uk/robots.txt>: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
Traceback (most recent call last):
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 975, in startConnectionAttempts
    "no results for hostname lookup: {}".format(self._hostStr)
DNSLookupError: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:17 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36
2019-07-04 03:02:19 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 3 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:19 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://www.rrawlins.co.uk/robots.txt>: An error occurred while connecting: 113: No route to host.
Traceback (most recent call last):
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
ConnectError: An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:19 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
2019-07-04 03:02:22 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/> (failed 1 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:22 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36
2019-07-04 03:02:23 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk> (failed 1 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:23 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31
2019-07-04 03:02:29 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk> (failed 2 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:29 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
2019-07-04 03:02:29 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/> (failed 2 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:29 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
2019-07-04 03:02:34 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.galkivconstruction.co.uk> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:34 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.galkivconstruction.co.uk>
Traceback (most recent call last):
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 975, in startConnectionAttempts
    "no results for hostname lookup: {}".format(self._hostStr)
DNSLookupError: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk.
2019-07-04 03:02:36 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.rrawlins.co.uk/> (failed 3 times): An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:36 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.rrawlins.co.uk/>
Traceback (most recent call last):
  File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
ConnectError: An error occurred while connecting: 113: No route to host.
2019-07-04 03:02:36 [scrapy.core.engine] INFO: Closing spider (finished)

标签： pythonscrapy

解决方案

您需要使用errback（请参阅Using errbacks to catch exceptions in request processing）。

推荐阅读