python - Scrapy - 达到最大重定向时生成 URL [301]
问题描述
代码
# -*- coding: utf-8 -*-
import scrapy
import pandas as pd
from ..items import Homedepotv2Item
from scrapy.http import HtmlResponse
class HomedepotspiderSpider(scrapy.Spider):
name = 'homeDepotSpider'
#allowed_domains = ['homedepot.com']
start_urls = ['https://www.homedepot.com/p/305031636', 'https://www.homedepot.com/p/311456581']
handle_httpstatus_list = [301]
def parseHomeDepot(self, response):
#get top level item
items = response.css('.pip-container-fluid')
for product in items:
item = Homedepotv2Item()
productSKU = product.css('.modelNo::text').getall() #getSKU
productURL = response.request.url #Get URL
item['productSKU'] = productSKU
item['productURL'] = productURL
yield item
终端消息
没有handle_httpstatus_list = [301]
2020-03-12 12:24:58 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.homedepot.com/p/ZLINE-Kitchen-and-Bath-ZLINE-30-in-Wall-Mount-Range-Hood-in-DuraSnow-%C3%91-Stainless-Steel-8687S-30-8687S-30/305031636> from <GET https://www.homedepot.com/p/ZLINE-Kitchen-and-Bath-ZLINE-30-in-Wall-Mount-Range-Hood-in-DuraSnow-%C3%91-Stainless-Steel-8687S-30-8687S-30/305031636>
2020-03-12 12:24:58 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.homedepot.com/p/ZLINE-Kitchen-and-Bath-ZLINE-30-in-Wooden-Wall-Mount-Range-Hood-in-Walnut-Includes-Remote-Motor-KBRR-RS-30/311456581>
{'productName': ['ZLINE 30 in. Wooden Wall Mount Range Hood in Walnut - '
'Includes Remote Motor'],
'productOMS': '311456581',
'productPrice': [' 979.95'],
'productSKU': ['KBRR-RS-30'],
'productURL': 'https://www.homedepot.com/p/ZLINE-Kitchen-and-Bath-ZLINE-30-in-Wooden-Wall-Mount-Range-Hood-in-Walnut-Includes-Remote-Motor-KBRR-RS-30/311456581'}
2020-03-12 12:25:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.homedepot.com/p/ZLINE-Kitchen-and-Bath-ZLINE-30-in-Wall-Mount-Range-Hood-in-DuraSnow-%C3%91-Stainless-Steel-8687S-30-8687S-30/305031636> from <GET https://www.homedepot.com/p/ZLINE-Kitchen-and-Bath-ZLINE-30-in-Wall-Mount-Range-Hood-in-DuraSnow-%C3%91-Stainless-Steel-8687S-30-8687S-30/305031636>
2020-03-12 12:25:01 [scrapy.downloadermiddlewares.redirect] DEBUG: Discarding <GET https://www.homedepot.com/p/ZLINE-Kitchen-and-Bath-ZLINE-30-in-Wall-Mount-Range-Hood-in-DuraSnow-%C3%91-Stainless-Steel-8687S-30-8687S-30/305031636>: max redirections reached
2020-03-12 12:25:01 [scrapy.core.engine] INFO: Closing spider (finished)
2020-03-12 12:25:01 [scrapy.extensions.feedexport] INFO: Stored csv feed (1 items) in: stdout:
和handle_httpstatus_list = [301]
2020-03-12 12:27:30 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-03-12 12:27:31 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://www.homedepot.com/p/305031636> (referer: None)
2020-03-12 12:27:31 [scrapy.core.engine] DEBUG: Crawled (301) <GET https://www.homedepot.com/p/311456581> (referer: None)
2020-03-12 12:27:31 [scrapy.core.engine] INFO: Closing spider (finished)
这是我用来导出到excel的scrapy crawl homeDepotSpider -t csv -o - > "pathname"
问题
所以我最初遇到的问题是“ https://www.homedepot.com/p/305031636 ”被我的蜘蛛忽略了,因为链接会丢弃错误代码301(重定向太多)。在研究了这个问题后,我发现handle_httpstatus_list = [301]
应该已经解决了这个问题。但是,当我使用这段代码时,由于工作链接(' https://www.homedepot.com/p/311456581 ')重定向到不同的页面,它也会被忽略。
基本上我想做的是从所有没有
ERR_TOO_MANY_REDIRECTS
但从具有该错误代码的链接中获取 URL,然后将该数据导出到 Excel。
编辑:问我问题的更好方法:由于我正在使用的所有 URL 都经过重定向,我如何处理无法重定向和抓取这些 URL 的页面?
这也不是我的整个程序,我只包含了我认为对程序来说必要的部分。
解决方案
您可以通过以下方式手动处理 301 错误代码:
class HomedepotspiderSpider(scrapy.Spider):
name = 'my_spider'
retry_max_count = 3
start_urls = ['https://www.homedepot.com/p/305031636', 'https://www.homedepot.com/p/311456581']
handle_httpstatus_list = [301]
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.retries = {} # here are stored URL retries counts
def parse(self, response):
if response.status == 301:
retries = self.retries.setdefault(response.url, 0)
if retries < self.retry_max_count:
self.retries[response.url] += 1
yield response.request.replace(dont_filter=True)
else:
# ...
# DO SOMETHING TO TRACK ERR_TOO_MANY_REDIRECTS AND SAVE response.url
# ...
return
请注意,302 代码也可用于重定向。
推荐阅读
- typescript - Typescript 类方法,它采用调用者的实例类型
- haskell - 如何使`co-log`的`withLog`与`Scotty`一起工作?
- sql - 获取重叠时间段的计数
- security -
AppScanConfig.xml 中的标记在 App Scan 期间不排除目录 - javascript - 在 {array[]} 中访问并添加对象{},它是 Reactjs 中另一个数组 [] 的属性?
- asp.net-core - 从 JWT 令牌获取我的所有服务中的用户信息的模式
- java - 这是得墨忒耳法则的一个很好的例子吗?
- python - 根据数据框中选定/过滤的行绘制折线图
- concatenation - qlik sense - 连接同一列几次
- c# - 如何在 CLASS 而不是 Code-Behind 中编写此 LIST?