时间戳

首页 > 解决方案 > Scrapy Crawler 不跟随链接

python - Scrapy Crawler 不跟随链接

问题描述

我在试图弄清楚为什么我的辅助功能无法遵循新链接然后输出数据时遇到了一些麻烦。该parse功能工作得很好。当它回调时parse_puppy，什么都没有发生。当我检查 json 输出时，我发现 from 的所有内容puppy都已成功抓取，但parse_puppy.

在第 28 行，如果我将方法更改为followI get results，但大约十几次都是相同的结果。

代码：

import scrapy
from scrapy.cmdline import execute

class Spider(scrapy.Spider):
    name = "puppyDetails"

    def start_requests(self):
        urls = ['https://ws.petango.com/webservices/adoptablesearch/wsAdoptableAnimals.aspx?species=Dog&gender=A&agegroup=UnderYear&location=&site=&onhold=A&orderby=name&colnum=3&css=http://ws.petango.com/WebServices/adoptablesearch/css/styles.css&authkey=io53xfw8b0k2ocet3yb83666507n2168taf513lkxrqe681kf8&recAmount=&detailsInPopup=No&featuredPet=Include&stageID=&wmode=opaque']
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        # GRAB ALL TOPICAL PUPPY DETAILS
        for animal in response.css("div.list-animal-info-block"):
            yield {
                'puppy_name': animal.css('div.list-animal-name a::text').get(),
                'puppy_id': animal.css('div.list-animal-id::text').get(),
                'puppy_sex': animal.css('div.list-animal-sexSN::text').get(),
                'puppy_breed': animal.css('div.list-animal-breed::text').get(),
                'puppy_age': animal.css('div.list-animal-age::text').get(),
                'puppy_link': animal.css('div.list-animal-name a::attr(href)').get()
            }

            # DIVE INTO DETAILS PAGE
            detail_page = response.css('div.list-animal-name a::attr(href)').get()
            self.logger.info('get puppy details')
            # GO TO THE PUPPY DETAILS
            yield response.follow_all(detail_page, callback=self.parse_puppy)

    def parse_puppy(self, response):
        # GRAB PUPPY DETAILS
        for puppyDetails in response.xpath('//*[@class="detail-table"]//tr'):
            yield {
                'puppy_id': puppyDetails.xpath('//*[@id="lblID"]/text()').extract(),
                'puppy_status': puppyDetails.xpath('//*[@id="lblStage"]/text()').extract(),
                'puppy_intake_date': puppyDetails.xpath('//*[@id="lblIntakeDate"]/text()').extract()
            }

execute(['scrapy','crawl','puppyDetails'])

错误：

ERROR: Spider must return Request, BaseItem, dict or None, got 'generator' in <GET https://ws.petango.com/webservices/adoptablesearch/wsAdoptableAnimals.aspx?species=Dog&gender=A&agegroup=UnderYear&location=&site=&onhold=A&orderby=name&colnum=3&css=http://ws.petango.com/WebServices/adoptablesearch/css/styles.css&authkey=io53xfw8b0k2ocet3yb83666507n2168taf513lkxrqe681kf8&recAmount=&detailsInPopup=No&featuredPet=Include&stageID=&wmode=opaque>

标签： pythonhtmlscrapy

解决方案

该行应该是

yield from response.follow_all(detail_page, callback=self.parse_puppy)

推荐阅读