时间戳

首页 > 解决方案 > 从我正在使用图像管道的scrapy spider下载/抓取图像时，我遇到了一个值错误

python - 从我正在使用图像管道的scrapy spider下载/抓取图像时，我遇到了一个值错误

问题描述

我得到一个值错误：

raise ValueError('Missing scheme in request url: %s' % self._url)

ValueError: Missing scheme in request url: h

我的 items.py 代码是：

class Brand(scrapy.Item):
    name = scrapy.Field()
    url = scrapy.Field()
    brand_image = scrapy.Field()
    image_urls = scrapy.Field()
    images = scrapy.Field()

我的setting.py是：

BOT_NAME = 'scraper'    
SPIDER_MODULES = ['scraper.spiders']
NEWSPIDER_MODULE = 'scraper.spiders'
ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1}
IMAGES_STORE = 'images'

我的蜘蛛代码：

import scrapy
import json
from scraper.items import Brand


class QuotesSpider(scrapy.Spider):
    name = "brandDetails"
    allowed_domains = ["ozhat-turkiye.com"]
    with open('brands.json') as data_file:
        data_item = json.load(data_file)
        start_urls = list()
    for item in data_item:
        start_urls.append(item["url"])

    def parse(self, response):
        item = Brand()
        name = response.css("div.th::text").extract_first()
        name = name.replace('Products of ', '')
        item['name'] = name
        item['url'] = response.url
        urls = response.css("div.productimage img::attr(src)").extract_first()
        urls = response.urljoin(urls)
        item['image_urls'] = urls
        yield item

标签： pythonpython-3.xweb-scrapingscrapypipeline

解决方案

Missing scheme in request url

总是意味着您的 URL 无效，它的丢失http://和https://

因此，在您拥有的图片网址之前https://或http://之前

`https://` + response.css("div.productimage img::attr(src)").extract_first()

推荐阅读