python - 我的第一个 Scrapy Spider 不适用于 MySQL 数据库

问题描述

我是网络抓取的新手，我的抓取代码不起作用，我不知道！我想抓取这个网站（http://quotes.toscrape.com），然后将数据保存到 MySQL 数据库中。所以，我设计了一个基本的 Spider ：

import scrapy
from ..items import QuotetutorialItem

class QuoteSpider(scrapy.Spider) :
    name = 'quotes'
    start_urls = [
        'http://quotes.toscrape.com/'
    ]

    def parse(self, response) :

        items = QuotetutorialItem()

        all_div_quotes = response.css('div.quote')

        for quotes in all_div_quotes:

            title = quotes.css('span.text::text').extract()
            author = quotes.css('.author::text').extract()
            tag = quotes.css('.tag::text').extract()

            items['title'] = title
            items['author'] = author
            items['tag'] = tag

            yield items

这是我的“pipelines.py”代码：

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html

# Scraping data - > Item Containers - > Json/csv files
# Scraping data - > Item Containers - > Pipeline - > SQL/Mongo database

import mysql.connector

class QuotetutorialPipeline(object):

    def __int__(self):
        self.create_connection()
        self.create_table()

    def create_connection(self):
        self.conn = mysql.connector.connect(
                host = 'localhost',
                user = 'root',
                passwd = 'jozefleonel',
                database = 'myquotes'
            )
        self.curr = self.conn.cursor()

    def create_table(self):
        self.curr.execute("""DROP TABLE IF EXISTS quotes_tb""")
        self.curr.execute("""create table quotes_tb(
                        title text,
                        author text,
                        tag text
                        )""")

    def process_item(self, item, spider):
        self.store_db(item)
        return item

    def store_db(self,item):
        self.curr.execute("""insert into quotes_tb values (%s,%s,%s)""", (
            item['title'][0],
            item['author'][0],
            item['tag'][0]
            ))

        self.conn.commit()

您在回复中找到错误消息，谢谢^^

标签： pythonscrapy

python - 我的第一个 Scrapy Spider 不适用于 MySQL 数据库

问题描述

解决方案

推荐阅读