python - Scrapy不发送帖子请求

Question

我试图向 URL 发出 post 请求，但 scrapy 没有发送 post 请求。我没有得到正确的回应。

下面是我的代码。

import scrapy

class MainSpider(scrapy.Spider):
    name = 'main'
    directory = 'https://www.constructionenquirer.com/directory'
    start_urls = [directory]

    def parse(self, response):
        sectors = response.xpath('(//select[@name="sector"]/option)[position()>1]')
        for sector in sectors:
            vals = sector.xpath('.//@value').get()

            data = {
                'ce-directory-action': 'ce-directory-action',
                'sector': vals,
                'action': 'find-firms-by-sector'
            }

            yield scrapy.FormRequest(url=self.directory, formdata=data, callback=self.parse_sectors)
    
    def parse_sectors(self, response):
        yield {
            "Name": response.xpath('//h3/a/text()').get()
        }

标签： pythonscrapy

Answer 1

您的代码中有错字：

'ce-directory-action': 'sector-search',