python - Scrapy,Twisted 严重错误:ImportError: DLL load failed
问题描述
我一直在运行我的scrapy代码,直到我在调用“scrapy crawl file”时收到关于找不到我的模块的错误。我不记得改变任何重要的东西,这个错误突然出现。
我重新安装了scrapy,现在出现了一个新错误:
2019-05-27 17:39:19 [scrapy.core.engine] INFO: Spider opened
Unhandled error in Deferred:
2019-05-27 17:39:19 [twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "c:\users\Me\virtual_workspace\lib\site-packages\scrapy\crawler.py", line 172, in crawl
return self._crawl(crawler, *args, **kwargs)
File "c:\users\Me\virtual_workspace\lib\site-packages\scrapy\crawler.py", line 176, in _crawl
d = crawler.crawl(*args, **kwargs)
File "c:\users\Me\virtual_workspace\lib\site-packages\twisted\internet\defer.py", line 1613, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "c:\users\Me\virtual_workspace\lib\site-packages\twisted\internet\defer.py", line 1529, in _cancellableInlineCallbacks
_inlineCallbacks(None, g, status)
--- <exception caught here> ---
File "c:\users\Me\virtual_workspace\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "c:\users\Me\virtual_workspace\lib\site-packages\scrapy\crawler.py", line 82, in crawl
yield self.engine.open_spider(self.spider, start_requests)
builtins.ImportError: DLL load failed: The specified module could not be found.
2019-05-27 17:39:19 [twisted] CRITICAL:
Traceback (most recent call last):
File "c:\users\Me\virtual_workspace\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "c:\users\Me\virtual_workspace\lib\site-packages\scrapy\crawler.py", line 82, in crawl
yield self.engine.open_spider(self.spider, start_requests)
ImportError: DLL load failed: The specified module could not be found.
我试图查看文件目录并且 crawler.py 仍然存在。其他一些帖子告诉我安装 pywin32 但我已经安装了,所以我重新安装无济于事。我什至将基本构造函数复制到我的,但它仍然不起作用。任何帮助表示赞赏。
我的简化代码:
import scrapy
from scrapy_splash import SplashRequest
import requests
import xml.etree.ElementTree as ET
import math
from datetime import date
from collections import deque
class mySpider(scrapy.Spider):
name = 'myScraper'
#requests
def start_requests(self):
urls = [
'https://www.msn.com/en-ca/'
]
self.link_queue = deque()
self.headers = {'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36 OPR/60.0.3255.109'}
for url in urls:
yield SplashRequest(url=url, callback=self.parse, endpoint = 'render.html', args = {'wait': 7}, headers = self.headers)
#response
def parse(self, response):
pass
解决方案
推荐阅读
- node.js - Socket.io 与生产中的 PM2 集群 - websocket 与粘性会话
- c# - 如何用另一个列表的对象切换一个列表的对象
- python - os.path.sep.join 在两个“args”上不起作用,为什么?
- python-3.x - 如何在小数点前后打印固定位数
- c++ - 如何使用 EC2 实例作为 API 服务器?当前使用 C++ lambda
- python - 使用请求库登录抓取程序时获取状态码 500。我不知道为什么会发生这种情况
- groovy - 将多个键值对添加到 groovy 映射
- powershell - 如何根据文件名中的搜索字符串创建文件夹并将文件移动到其中?
- c# - 使用Union时如何防止产生外层Select语句?
- r - 如何将时间延迟变量拟合到 R 中的数据?