python - scrapy异常CloseSpider关闭爬虫功能

Question

抛出“CloseSpider”异常时会发生什么以及我该如何处理。

from scrapy.exceptions import CloseSpider
    #code
    def parse(self,response):
        #code
        if condition:
            raise CloseSpider(reason="bandwidth_exceed")


    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super(FlickrSearchSpider, cls).from_crawler(crawler, *args, **kwargs)
        crawler.signals.connect(spider.spider_closed, signal=scrapy.signals.spider_closed)
        return spider
    def spider_closed(self):
        print("completed")

如何区分 spider_close 与正常运行或满足条件（reason="bandwidth_exceed"）。如果可能的话，我希望能够以不同的方式处理每一个

标签： pythonscrapy

Answer 1

很简单，只要定义closed：

def closed(self, reason):
    if reason == "bandwidth_exceed":
        process_bandwidth_exceed()
    else:
        something_else()
    pass