python - 如何动态生成字段
问题描述
查阅scrapy官方文档,发现可以动态添加字段,但是不知道怎么做
我尝试了 ItemLoader 的演示,它像测试代码一样成功:
我为 ItemLoader 传递了一个 field_name_list,就像一个代码:
当我从 ItemClass 观看时,就像两个代码:
当我运行代码时,它告诉我错误,但我打印 args,它成功通过,就像三个代码:
我忘记了通过运行代码,比如四个代码:
我的self.field,像这样:
"field": {
"content": [
{
"expression": [
"//td[@id='article_content']//text()"
],
"method": "xpath"
}
],
"datetime": [
{
"expression": [
"//p[@class='xg1']/text()"
],
"method": "xpath",
"re" : "\\d{2,4}年\\d{1,2}月\\d{1,2}日|\\d{1,2}月\\d{1,2}日|\\d{2,4}[-|/|.]\\d{1,2}[-|/|.]\\d{1,2}"
}
],
# Test Code
class Test(Item):
field_list = ["title", "urls", "image", "content","name", "source","pubdate"]
fields = {field_name: Field() for field_name in field_list}
# one code
field_list = []
for key, value in field.items():
field_list.append(key)
loader = ItemLoader(item=Demo(field_list), response=response)
# two code
class Demo(Item):
def __init__(self, *args, **kwargs):
print(args,1111111111111111111111111111111111111111111111111111)
self._values = {}
if args or kwargs: # avoid creating dict for most common case
for k, v in six.iteritems(dict(*args, **kwargs)):
self[k] = v
# super(Demo, self).__init__()
# fields = {field_name: Field() for field_name in field_list}
# three code
2019-04-02 17:57:13 [scrapy.core.scraper] ERROR: Spider error processing <GET http://news.wmxa.cn/beilin/201904/615036.html via http://192.168.99.100:8050/render.html> (referer: None)
Traceback (most recent call last):
File "D:\python\Scripts\test\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
File "D:\python\Scripts\test\lib\site-packages\scrapy_splash\middleware.py", line 156, in process_spider_output
for el in result:
File "D:\python\Scripts\test\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
File "D:\python\Scripts\test\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "D:\python\Scripts\test\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "D:\python\Scripts\test\lib\site-packages\scrapy\spiders\crawl.py", line 78, in _parse_response
for requests_or_item in iterate_spider_output(cb_res):
File "F:\Newspider\news_project\news\news\spiders\newspider.py", line 141, in parse_item
loader = ItemLoader(item=Demo(field_list), response=response)
File "F:\Newspider\news_project\news\news\items.py", line 70, in __init__
for k, v in six.iteritems(dict(*args, **kwargs)):
TypeError: dict expected at most 1 arguments, got 8
2019-04-02 17:57:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://news.wmxa.cn/beilin/201904/615065.html via http://192.168.99.100:8050/render.html> (referer: None)
(['title', 'content', 'blei', 'image', 'pay', 'pubdate', 'source', 'url'],) 1111111111111111111111111111111111111111111111111111
# four code
field = self.field
loader = ItemLoader(item=Demo(field_list), response=response)
for key, value in field.items():
for extractor in value:
try:
if extractor.get("method") == "xpath":
loader.add_xpath(key, *extractor.get("expression"), **{"re": extractor.get("re")})
if extractor.get("method") == "css":
loader.add_css(key, *extractor.get("expression"), **{"re": extractor.get("re")})
if extractor.get('method') == 'attr':
loader.add_value(key, getattr(response, *extractor.get('expression')))
我希望它是动态生成的,我该怎么办?
解决方案
有一种非常简单的使用方式ItemLoader
,使用这种方式,您可以ItemLoader
在解析响应时从响应中添加任何您想要的数据。
from scrapy import Item
from scrapy.loader import ItemLoader
def parse(self, response):
l = ItemLoader(item=Item(), response=response) #first arg can be scrapy.Item object
for item in response:
l.add_value(item, response[item]) # you can also use literal values
return l.load_item()
推荐阅读
- django - dajngo 泛型类中的 select_related 属性是什么,有人可以解释这意味着什么吗?
- swift - ObservedObject 更改时的通知
- java - 无法在图库中显示图片(Android 10+)
- wso2 - 为什么 WSO2 API Manager 获取令牌时主机名验证失败?
- node.js - 如何使用 mongoDB 过滤数组对象中的数据?
- node.js - 全栈:url 的 Http 失败响应:0 未知错误
- android - MIUI 状态栏主题的问题
- javascript - 如何在打字稿中使用文字字符串类型键
- reactjs - 使用 withIronSession() 访问 Next.js 页面中的会话数据
- state - 是否可以根据状态隐藏/不可见的笔记本页面?(奥多 13)