python - Scrapy CSV incorrectly formatted
问题描述
I'm new to scrapy package, and here's my problem:
import scrapy
class simpleSpider(scrapy.Spider):
name = "simple_spider"
start_urls = ['http://quotes.toscrape.com/login']
def parse(self, response):
token = response.css("input[name=csrf_token] ::attr(value)").extract_first()
formdata = {
'csrf_token' : token,
'username' : 'rseiji',
'password' : 'seiji1234'
}
yield scrapy.FormRequest(response.url, formdata=formdata, callback=self.parse_logged)
def parse_logged(self, response):
yield {
'text' : response.css('span.text::Text').extract(),
'author' : response.css('small.author::Text').extract(),
'tags' : response.css('div.tags a.tag::Text').extract()
}
This is my spider. And it does work. But when I try to:
scrapy crawl simple_spider -o mySpider.csv
the .csv file doesn't seen to be correctly formated. It extracts only the "text" column.
What's wrong?
Thank you!
Editted: This is my .csv file:
text,author,tags
"“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”,“It is our choices, Harry, that show what we truly are, far more than our abilities.”,“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”,“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”,“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”,“Try not to become a man of success. Rather become a man of value.”,“It is better to be hated for what you are than to be loved for what you are not.”,“I have not failed. I've just found 10,000 ways that won't work.”,“A woman is like a tea bag; you never know how strong it is until it's in hot water.”,“A day without sunshine is like, you know, night.”","Albert Einstein,J.K. Rowling,Albert Einstein,Jane Austen,Marilyn Monroe,Albert Einstein,André Gide,Thomas A. Edison,Eleanor Roosevelt,Steve Martin","change,deep-thoughts,thinking,world,abilities,choices,inspirational,life,live,miracle,miracles,aliteracy,books,classic,humor,be-yourself,inspirational,adulthood,success,value,life,love,edison,failure,inspirational,paraphrased,misattributed-eleanor-roosevelt,humor,obvious,simile"
...
Figure out now that it is not that there are empty columns. The .csv file format is not well defined. Everything came up in just one row!
解决方案
解决了!
import scrapy
class simpleSpider(scrapy.Spider):
name = "simple_spider"
start_urls = ['http://quotes.toscrape.com/login']
def parse(self, response):
formdata = {
'username' : 'rseiji',
'password' : 'seiji1234'
}
yield scrapy.FormRequest.from_response(response, formdata=formdata, callback=self.parse_logged,)
def parse_logged(self, response):
# Get list of Selector objects and loop through them
for quote in response.css('div.quote'):
# yield each item individually
yield {
'text' : quote.css('span.text::Text').extract_first(),
'author' : quote.css('small.author::Text').extract_first(),
'author_goodreads_url' : quote.css('span a[href*="goodreads.com"]::attr(href)').extract_first(),
'tags' : quote.css('div.tags a.tag::Text').extract()
}
问题是我使用的是 extract()。我想做的是得到一个Selector对象的列表。
使用 extract() 将始终产生一个列表输出。当您使用 extract 时,您将获得您使用选择器请求的 html 字符串列表,或者在使用 extract_first() 时获得单个字符串。通过不使用 extract() 或 extract_first(),您可以创建一个选择器列表,然后您可以对其进行迭代并在其上链接一个新选择器,从而允许您挑选每个单独的项目。
推荐阅读
- c# - 使用带有 EF Core 的投影跳过和接收子实体时出错
- active-directory - 撤销对通过 MS Active Directory 身份验证接收的 access_token 的访问权限
- python - DJANGO 错误。未找到带有参数“(”,)”的“收据”的反向。尝试了 1 种模式:['receipt/(?P
[^/]+)/$'] 尝试使用 ID:PK - java - 将一个数组转换为另一个数组
- google-apps-script - 应用程序脚本——打印指定范围 X 时间
- r - 使用令牌通过 API 连接到 Google Ads
- sql - SQL/Snowflake - 添加未来日期行并使用另一个表中的最后一个值填充
- javascript - Javascript函数返回未定义的输出
- reactjs - 在useEffect中使用redux时如何修复内存泄漏?
- oracle - Oracle PL/SQL 问题:PLS-00306:调用中的参数数量或类型错误