scrapy - Scrapy - 空闲信号蜘蛛遇到错误
问题描述
我正在尝试创建一个蜘蛛,它一直在运行,一旦它进入空闲状态,它应该从数据库中获取下一个 url 来解析。不幸的是,我已经在一开始就得到了堆栈:
# -*- coding: utf-8 -*-
import scrapy
from scrapy import signals
from scrapy import Spider
import logging
class SignalspiderSpider(Spider):
name = 'signalspider'
allowed_domains = ['domain.de']
yet = False
def start_requests(self):
logging.log(logging.INFO, "______ Loading requests")
yield scrapy.Request('https://www.domain.de/product1.html')
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
logging.log(logging.INFO, "______ From Crawler")
spider = spider = super(SignalspiderSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.idle, signal=scrapy.signals.spider_idle)
return spider
def parse(self, response):
self.logger.info("______ Finished extracting structured data from HTML")
pass
def idle(self):
logging.log(logging.INFO, "_______ Idle state")
if not self.yet:
self.crawler.engine.crawl(self.create_request(), self)
self.yet = True
def create_request(self):
logging.log(logging.INFO, "_____________ Create requests")
yield scrapy.Request('https://www.domain.de/product2.html?dvar_82_color=blau&cgid=')
以及我得到的错误:
2019-03-27 21:41:38 [root] INFO: _______ Idle state
2019-03-27 21:41:38 [root] INFO: _____________ Create requests
2019-03-27 21:41:38 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method RefererMiddleware.request_scheduled of <scrapy.spidermiddlewares.referer.RefererMiddleware object at 0x7f93bcc13978>>
Traceback (most recent call last):
File "/home/spidy/Documents/spo/lib/python3.5/site-packages/scrapy/utils/signal.py", line 30, in send_catch_log
*arguments, **named)
File "/home/spidy/Documents/spo/lib/python3.5/site-packages/pydispatch/robustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "/home/spidy/Documents/spo/lib/python3.5/site-packages/scrapy/spidermiddlewares/referer.py", line 343, in request_scheduled
redirected_urls = request.meta.get('redirect_urls', [])
AttributeError: 'NoneType' object has no attribute 'meta'
我究竟做错了什么?
解决方案
尝试:
def idle(self, spider):
logging.log(logging.INFO, "_______ Idle state")
if not self.yet:
self.yet = True
self.crawler.engine.crawl(Request(url='https://www.domain.de/product2.html?dvar_82_color=blau&cgid=', callback=spider.parse), spider)
我不确定在方法 spider_idle 中创建请求是否正确,传递另一个方法来发出请求,就像你做的那样。
在 Scrapy spider_idle 信号中查看更多信息 - 需要添加带有解析项回调的请求
推荐阅读
- python - 将数据写入xls并在python中下载
- reactjs - 如何在下一个 js 中从节点模块外部的文件夹中导入模块
- c# - C# 依赖注入:检索一个单例注入到其他单例构造函数
- javascript - Is there any way to determine if a time input is partial filled or empty?
- c# - 当前上下文中不存在名称“xxx”(您是否缺少对程序集的引用)
- php - 将用户登录限制在特定时间段
- node.js - 无法在 selenium webdriver node.js 中找到元素错误
- php - 在php中显示来自realpath的图像
- java - 为什么我会在java中打印“NaN”?
- php - 使用 php 和命令行/命令提示符/powershell 打开一个 excel