pandas - Scrapy - 仅将 xpath 复制到 .csv 文件中

Question

我有许多其他具有类似基本代码的脚本，但是当我在 cmd 中运行这个蜘蛛时，我打开 .csv 文件查看保存的“标题”，我将 xpath 复制到了 excel 中。知道为什么吗？

import scrapy


class MovieSpider(scrapy.Spider):
    name = 'movie'
    allowed_domains = ['https://www.imdb.com/search/title?start=1']
    start_urls = ['https://www.imdb.com/search/title?start=1/']

    def parse(self, response):
        titles = response.xpath('//*[@id="main"]/div/div/div[3]/div[1]/div[3]/h3/a')
        pass
        print(titles)


        for title in titles:
            yield {'Title': title}

---尝试以下两个：-----

for subject in titles:
            yield {
                'Title': subject.xpath('.//h3[@class="lister-item-header"]/a/text()').extract_first(),
                'Runtime': subject.xpath('.//p[@class="text-muted"]/span/text()').extract_first(),
                'Description': subject.xpath('.//p[@class="text-muted"]/p/text()').extract_first(),
                'Director': subject.xpath('.//*[@id="main"]/a/text()').extract_first(),
                'Rating': subject.xpath('.//div[@class="inline-block ratings-imdb-rating"]/strong/text()').extract_first()
            }

标签： pandasscrapy

Answer 1

使用extract()or extract_first()，也为 xpath 使用更短、更宽泛的符号：

import scrapy


class MovieSpider(scrapy.Spider):
    name = 'movie'
    allowed_domains = ['https://www.imdb.com/search/title?start=1']
    start_urls = ['https://www.imdb.com/search/title?start=1/']

    def parse(self, response):
        subjects = response.xpath('//div[@class="lister-item mode-advanced"]')

        for subject in subjects:
            yield {
                'Title': subject.xpath('.//h3[@class="lister-item-header"]/a/text()').extract_first(),
                'Rating': subject.xpath('.//div[@class="inline-block ratings-imdb-rating"]/strong/text()').extract_first(),
                'Runtime': subject.xpath('.//span[@class="runtime"]/text()').extract_first(),
                'Description': subject.xpath('.//p[@class="text-muted"]/text()').extract_first(),
                'Directior': subject.xpath('.//p[contains(text(), "Director")]/a[1]/text()').extract_first(),
            }

输出：