python-3.x - ValueError:请求 url Scrapy 中缺少方案
问题描述
我正在尝试https://www.skynewsarabia.com/
使用Scrapy
并遇到此错误,ValueError: Missing scheme in request url:
我尝试了在 stackoverflow 上找到的每一个解决方案,但没有一个对我有用。这是我的蜘蛛:
name = 'skynews'
allowed_domains = ['www.skynewsarabia.com']
start_urls = ['https://www.skynewsarabia.com/sport/latest-news-%D8%A2%D8%AE%D8%B1-%D8%A7%D9%84%D8%A3%D8%AE%D8%A8%D8%A7%D8%B1']
}
def parse(self, response):
link = "https://www.skynewsarabia.com"
# get the urls of each article
urls = response.css("a.item-wrapper::attr(href)").extract()
# for each article make a request to get the text of that article
for url in urls:
# get the info of that article using the parse_details function
yield scrapy.Request(url=link +url, callback=self.parse_details)
# go and get the link for the next article
next_article = response.css("a.item-wrapper::attr(href)").extract_first()
if next_article:
# keep repeating the process until the bot visits all the links in the website!
yield scrapy.Request(url=next_article, callback=self.parse) # keep calling yourself!
这是整个错误:
2019-01-30 11:49:34 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-01-30 11:49:34 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2019-01-30 11:49:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.skynewsarabia.com/robots.txt> (referer: None)
2019-01-30 11:49:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.skynewsarabia.com/sport/latest-news-%D8%A2%D8%AE%D8%B1-%D8%A7%D9%84%D8%A3%D8%AE%D8%A8%D8%A7%D8%B1> (referer: None)
2019-01-30 11:49:35 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.skynewsarabia.com/sport/latest-news-%D8%A2%D8%AE%D8%B1-%D8%A7%D9%84%D8%A3%D8%AE%D8%A8%D8%A7%D8%B1> (referer: None)
Traceback (most recent call last):
File "c:\users\hozrifai\desktop\scraping\venv\lib\site-
packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
File "c:\users\hozrifai\desktop\scraping\venv\lib\site-
packages\scrapy\spidermiddlewares\offsite.py", line 30, in
process_spider_output
for x in result:
File "c:\users\hozrifai\desktop\scraping\venv\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
File "c:\users\hozrifai\desktop\scraping\venv\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "c:\users\hozrifai\desktop\scraping\venv\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "C:\Users\HozRifai\Desktop\scraping\articles\articles\spiders\skynews.py", line 28, in parse
yield scrapy.Request(url=next_article, callback=self.parse) # keep calling yourself!
File "c:\users\hozrifai\desktop\scraping\venv\lib\site-packages\scrapy\http\request\__init__.py", line 25, in __init__
self._set_url(url)
File "c:\users\hozrifai\desktop\scraping\venv\lib\site-packages\scrapy\http\request\__init__.py", line 62, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: /sport/1222754-%D8%A8%D9%8A%D8%B1%D9%86%D9%84%D9%8A-%D9%8A%D8%B6%D8%B9-%D8%AD%D8%AF%D8%A7-%D9%84%D8%B3%D9%84%D8%B3%D9%84%D8%A9-%D8%A7%D9%86%D8%AA%D8%B5%D8%A7%D8%B1%D8%A7%D8%AA-%D8%B3%D9%88%D9%84%D8%B4%D8%A7%D8%B1
2019-01-30 11:49:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.skynewsarabia.com/sport/1222754-%D8%A8%D9%8A%D8%B1%D9%86%D9%84%D9%8A-%D9%8A%D8%B6%D8%B9-%D8%AD%D8%AF%D8%A7-%D9%84%D8%B3%D9%84%D8%B3%D9%84%D8%A9-%D8%A7%D9%86%D8%AA%D8%B5%D8%A7%D8%B1%D8%A7%D8%AA-%D8%B3%D9%88%D9%84%D8%B4%D8%A7%D8%B1> (referer: https://www.skynewsarabia.com/sport/latest-news-%D8%A2%D8%AE%D8%B1-%D8%A7%D9%84%D8%A3%D8%AE%D8%A8%D8%A7%D8%B1)
提前致谢
解决方案
你有next_article
没有方案的网址。尝试:
next_article = response.css("a.item-wrapper::attr(href)").get()
if next_article:
yield scrapy.Request(response.urljoin(next_article))
推荐阅读
- javascript - Windows 10 上的 Svelte '_ajax is not defined' 错误,而不是 Ubuntu,bundle.js 不同
- python - luma.oled SPI ssd1306 数据到 GPIO 引脚问题使用包/模块
- android - 如何在 Kotlin 的 DatePickerDialog 中制作日期分隔符?
- nuxt.js - 使用 v-html 实现 Nuxtjs SEO
- javascript - Chartjs 雷达 pointLabels 颜色 Scriptable 出错
- asp.net - 收到 JWT 令牌并登录后如何使用?
- python - 将字典的值转换为键值对
- python - 数据框的分类数据之间的相关性和概率 - 指标(建议)
- subdomain - 如何设置和维护我的多租户 Web 应用程序的 2 个版本?
- bash - 对于目录中的每个子文件夹,删除具有特定扩展名的文件,只有存在具有相同文件名+不同指定扩展名的文件