时间戳

首页 > 解决方案 > Scrapy：检查页面是否包含 HTML 表单元素

python - Scrapy：检查页面是否包含 HTML 表单元素

问题描述

我需要一个scrapy 脚本来探索整个网站，并且只保存其中包含formHTML 标记的页面。

这是我目前无法正常工作的方法

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

    class MySpider(CrawlSpider):
        name = 'mps'
        allowed_domains = ['some.url.com']
        start_urls = ['https://some.url.com/']

        rules = (
            Rule(LinkExtractor(), callback='parse_item', follow=True),
        )

        def parse_item(self, response):
            hasForm = response.xpath("//form[@id = 'aspnetForm']/form").extract_first(default='not-found')
            if hasForm == 'not-found':
                pass
            else:
                filename = response.url.split("/")[-2] + '.html'
                with open(filename, 'wb') as f:
                    f.write(response.body)
                pass

更新：

我还需要用form特定的 id排除

标签： pythonxpathscrapy

解决方案

例子

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):
    name = 'mps'
    allowed_domains = ['some.url.com']
    start_urls = ['https://some.url.com/']

    rules = (
        Rule(LinkExtractor(), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        hasForm = response.xpath("//form").extract_first(default='not-found')            
        if hasForm != 'not-found':
            page = response.url.split("/")[-2]
            filename = 'test-%s.html' % page
            with open(filename, 'wb') as f:
                f.write(response.body)

推荐阅读