时间戳

首页 > 解决方案 > 保存 Scrapy 'start_urls' 并正确存储在数据框中

python - 保存 Scrapy 'start_urls' 并正确存储在数据框中

问题描述

我正在使用 Scrapy 来抓取一些网站数据。但我无法正确获取我的数据。

这是我的代码的输出（见下面的代码）：

在命令行中：

scrapy crawl myspider -o items.csv

输出：

asin_product                                    product_name
ProductA,,,ProductB,,,ProductC,,,            BrandA,,,BrandB,,,BrandC,,,    

ProductA,,,ProductD,,,ProductE,,,            BrandA,,,BrandB,,,BrandA,,,    

#Note that the rows are representing the start_urls and that the ',,,' 
#three commas are separating the data.

期望的输出：

scrapy crawl myspider -o items.csv

Start_URL     asin_product      product_name 
URL1           ProductA           BrandA
URL1           ProductB           BrandB
URL1           ProductC           BrandC
URL2           ProductA           BrandA
URL2           ProductD           BrandB
URL2           ProductE           BrandA

我在 Scrapy 中使用的代码：

import scrapy
from amazon.items import AmazonItem

class AmazonProductSpider(scrapy.Spider):
  name = "AmazonDeals"
  allowed_domains = ["amazon.com"]

#Use working product URL below
   start_urls = [
      "https://www.amazon.com/s?k=shoes&ref=nb_sb_noss_2",   # This should 
       be #URL 1       
      "https://www.amazon.com/s?k=computer&ref=nb_sb_noss_2" # This should 
       be #URL 2 
 ]

def parse(self, response):

  items = AmazonItem()

  title = response.xpath('//*[@class="a-size-base-plus a-color-base a- 
  text-normal"]/text()').extract()
  asin =  response.xpath('//*[@class ="a-link-normal"]/@href').extract()  

  # Note that I devided the products with ',,,' to make it easy to separate 
  # them. I am aware that this is not the best approach. 
  items['product_name'] = ',,,'.join(title).strip()
  items['asin_product'] = ',,,'.join(asin).strip()

  yield items

标签： pythonpandasdataframescrapy

解决方案

首先，按类查询时推荐使用css。

现在到您的代码：

产品名称在 a 标签（产品 url）内。因此，您可以遍历链接并存储 URL 和标题。

<a class="a-link-normal a-text-normal" href="/adidas-Mens-Lite-Racer-Running/dp/B071P19D3X/ref=sr_1_3?keywords=shoes&amp;qid=1554132536&amp;s=gateway&amp;sr=8-3">
    <span class="a-size-base-plus a-color-base a-text-normal">Adidas masculina Lite Racer byd tênis de corrida</span>
</a>

您需要AmazonItem在 csv 文件的每一行创建一个对象。

def parse(self, response):

    # You need to improve this css selector because there are links which
    # are not a product, this is why I am checking if title is None and continuing.
    for product in response.css('a.a-link-normal.a-text-normal'):
        # product is a selector
        title = product.css('span.a-size-base-plus.a-color-base.a-text-normal::text').get()
        if not title:
            continue
        # The selector is already the a tag, so we only need to extract it's href attribute value.
        asin =  product.xpath('./@href').get()

        item = AmazonItem()
        item['product_name'] = title.strip()
        item['asin_product'] = asin.strip()

        yield item

推荐阅读