时间戳

首页 > 解决方案 > Scrapy Crawl ValueError

python-3.x - Scrapy Crawl ValueError

问题描述

我是python和scrapy的新手。我按照教程进行了爬网引文.toscrape.com。

我在代码中输入了它在教程中的样子，但是ValueError: invalid hostname当我运行scrapy crawl quotes时，我不断得到一个 : 。我在电脑上做Pycharm这个Mac。

URL我尝试在in部分周围加上单引号和双引号，start_urls = []但这并没有解决错误。

这是代码的样子：

import scrapy

class QuoteSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = [
        'http: // quotes.toscrape.com /'
    ]

    def parse(self, response):
        title = response.css('title').extract()
        yield {'titletext':title}

它应该是为了标题而抓取网站。

这是错误的样子：

2019-11-08 12:52:42 [scrapy.core.engine] INFO: Spider opened
2019-11-08 12:52:42 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-11-08 12:52:42 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-11-08 12:52:42 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http:///robots.txt>: invalid hostname: 
Traceback (most recent call last):
  File "/Users/newuser/PycharmProjects/ScrapyTutorial/venv/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
    defer.returnValue((yield download_func(request=request, spider=spider)))
ValueError: invalid hostname: 
2019-11-08 12:52:42 [scrapy.core.scraper] ERROR: Error downloading <GET http:///%20//%20quotes.toscrape.com%20/>
Traceback (most recent call last):
  File "/Users/newuser/PycharmProjects/ScrapyTutorial/venv/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
    defer.returnValue((yield download_func(request=request, spider=spider)))
ValueError: invalid hostname: 
2019-11-08 12:52:42 [scrapy.core.engine] INFO: Closing spider (finished)

标签： python-3.xscrapypycharm

解决方案

不要在 URL 中使用空格！

start_urls = [
    'http://quotes.toscrape.com/'
]

推荐阅读