时间戳

首页 > 解决方案 > How to scrape the items loaded via a "view more" button using Scrapy

python - How to scrape the items loaded via a "view more" button using Scrapy

问题描述

Here is the inspection of View more button in a website. I can crawl through data that are shown in the website but I want it somehow that it can crawl through items that are hidden behind the view more button. How do I do that? 

 <div id="view-more" class="p20px pt10px">
                        <div id="view-more-loader" class="tac"></div>

                        <a href="javascript:void(0);" onclick="add_more_product_classified();$('#load_more_a_id').hide();" class="xxxxlarge ffrc lightbginfo gbiwb bdr darkbdrinfo p10px20px db w180px m0a tac" id="load_more_a_id" style="display: block;"><b class="icon-refresh xsmall mr5px"></b>View More Products..</a>
                        </div>

My scrapy code:

import scrapy




class DummymartSpider(scrapy.Spider):
    name = 'dummymart'
    allowed_domains = ['dummymart.net']
    start_urls =['https://www.dummymart.com/catalog/car-dvd-player_cid100001018.html']



    def parse(self, response):
            Product = response.xpath('//div[@class="attr"]/h2/a/@title').extract()
            Company =  response.xpath('//div[@class="supplier"]/p/a/@title').extract()
            Country =  response.xpath('//*[@class="location a-color-secondary"]/span/text()').extract()
            Category = response.xpath('//*[@class="attr category hide--mobile"]/span/a/text()').extract()

            for item in zip(Product,Company,Country,Category):
                scraped_info = {
                    'Product':item[0],
                    'Company': item[1],
                    'Country':item[2],
                    'Category':item[3]

                }
                yield scraped_info

标签： pythonxpathweb-scrapingscrapy

解决方案

此类问题的通常解决方案是：

  1. 在浏览器中启动开发者工具；
  2. 转到网络面板，以便您可以查看浏览器发出的请求；
  3. 单击页面中的“查看更多”按钮，检查您的浏览器执行了哪个请求来获取数据；
  4. 对你的蜘蛛发出同样的请求。

这篇博文可能会对您有所帮助。

推荐阅读