时间戳

首页 > 解决方案 > 如何在 Scrapy 中无错误地下载图像？

python - 如何在 Scrapy 中无错误地下载图像？

问题描述

我是scrapy的新手。我正在尝试下载图像。

import scrapy
from scrapy.http import Request


class PlayerSpider(scrapy.Spider):
name = 'player'
#allowed_domains = ['nba.com/players']
start_urls = ['http://www.nba.com/players/']

def parse(self, response):
    Player_Name = response.css('div#content.nba-player-index a ::attr(title)').extract()
    Player_link = response.css('.nba-player-index__trending-item a::attr(href)').extract()
    links  = [url for url in Player_link if url.startswith("/players")]
    for link in links:

        absolute_url = response.urljoin(link)
        yield Request(absolute_url, callback=self.parse_players)

def parse_players(self, response):
    Player_Name = response.css('section.nba-player-header__details-bottom ::text').extract()
    items=[]
    for images in Player_Name:
        item = PlayerSpider()
        images_link = response.css('section.nba-detail-header-wrapper .nba-player-header__headshot img::attr(src)').extract_first()
        image_urls = 'http:{}'.format(images_link)
        item[image_urls]
        return item

    Player_Height = response.css('section.nba-player-vitals__top-left.small-6 p.nba-player-vitals__top-info-imperial ::text').extract()
    Player_Weight = response.css('section.nba-player-vitals__top-right.small-6 p.nba-player-vitals__top-info-imperial ::text').extract()
    yield {
                'Player_name' : Player_Name,
                'Player_Height' : Player_Height,
                'Player Weight' : Player_Weight
        }

我认为文件很好。但我无法编写正确的蜘蛛来获取图像。我能够获取图像 URL，但不知道如何使用 imagePipeline 存储图像。

items.py

import scrapy
from scrapy.item import Item

class PlayerSpider(scrapy.Item):
   image_url = scrapy.Field()
   images = scrapy.Field()
   pass

标签： pythonweb-scrapingscrapy

解决方案

要启用您的图像管道，您必须首先将其添加到您的项目 ITEM_PIPELINES 设置中。

设置.py：

ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = 'images'

关联

推荐阅读