时间戳

首页 > 解决方案 > Scrapy 每页只捕获一个问题，但有 10 个

python - Scrapy 每页只捕获一个问题，但有 10 个

问题描述

我已经尝试解决这个问题 5 天了。如果有人可以帮助我，谢谢：

Scrapy 每页只捕获一个问题。每页有 10 个问题。

我已经用过 CSS、xpath + regex、相对地址、绝对地址、LinkExtractor。

我已经禁用了服从 robots.txt，我已经使用了代理。

在scrapy shell 中，get() 也只捕获一个问题，而get_all() 捕获all in one。

我的scrapy.py：

import scrapy
from items import GabariteItem


class GbSpider(scrapy.Spider):

    name = "gb"
    start_urls = ['https://www.gabarite.com.br/questoes-de-concursos/assunto/agentes-publicos-e-lei-8112-de-1990']

    def parse(self, response):
        items = response.xpath("//body/div[3]")
        gb = GabariteItem()
        gb['url'] = response.url
        gb['area'] = items.xpath(".//h3/a[2]/text()").extract_first()
        gb['cargo'] = items.xpath(".//h3/a[3]/text()").extract_first()
        gb['curso'] = items.xpath(".//h3/a/text()").extract_first()
        gb['pergunta'] = items.xpath(".//li[@class='pergunta']").extract_first()
        gb['alternativaA'] = items.xpath(".//li[@class='respostas']//label[1]/text()").extract_first()
        gb['alternativaB'] = items.xpath(".//li[@class='respostas']//label[2]/text()").extract_first()
        gb['alternativaC'] = items.xpath(".//li[@class='respostas']//label[3]/text()").extract_first()
        gb['alternativaD'] = items.xpath(".//li[@class='respostas']//label[4]/text()").extract_first()
        gb['alternativaE'] = items.xpath(".//li[@class='respostas']//label[5]/text()").extract_first()
        yield gb

        #Próxima Página
        next_page = response.xpath("//a[@title='Próxima página']/@href").extract_first()
        if next_page:
            # self.log ('Próxima Página: https://www.gabarite.com.br/' + next_page)
            next_page_url = response.urljoin(next_page)
            yield scrapy.Request(url=next_page_url, callback=self.parse)

我的物品.py

import scrapy
class GabariteItem(scrapy.Item):
    url = scrapy.Field()
    area = scrapy.Field()
    cargo = scrapy.Field()
    curso = scrapy.Field()
    ano = scrapy.Field()
    nivel = scrapy.Field()
    pergunta = scrapy.Field()
    alternativaA = scrapy.Field()
    alternativaB = scrapy.Field()
    alternativaC = scrapy.Field()
    alternativaD = scrapy.Field()
    alternativaE = scrapy.Field()
    alternativaCorreta = scrapy.Field()

标签： pythonscrapy

解决方案

您需要遍历每个问题容器，然后选择与之相关的字段。这是scrapy shell中的一个例子：

>>> for question in response.css('article.lista-questoes'):
...     print(question.css('.numero h3 ::text').get())
...     # Create item here
... 
Questão 30979.   
Questão 25714.   
Questão 35985.   
Questão 35986.   
Questão 26362.   
Questão 28203.   
Questão 34446.   
Questão 35978.   
Questão 35981.   
Questão 30981.  

推荐阅读