python - 使用 Scrapy 难以从网页中抓取所需数据
问题描述
我正在抓取以下网页http://www.starcitygames.com/catalog/category/Duel%20Decks%20Venser%20vs%20Koth,我需要获取卡片名称、价格、库存和状况。好吧,我得到了四个工作中的三个,但我的病情有问题。无论我尝试什么,它都只会给我 NULL 或其他不正确的东西。
部分 HTML 代码
<td class="deckdbbody search_results_7">
<a href="http://www.starcitygames.com/content/cardconditions">NM/M</a>
</td>
SplashSpider.py
import csv
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import GameItem
# process the csv file so the url + ip address + useragent pairs are the same as defined in the file # returns a list of dictionaries, example:
# [ {'url': 'http://www.starcitygames.com/catalog/category/Rivals%20of%20Ixalan',
# 'ip': 'http://204.152.114.244:8050',
# 'ua': "Mozilla/5.0 (BlackBerry; U; BlackBerry 9320; en-GB) AppleWebKit/534.11"},
# ...
# ]
def process_csv(csv_file):
data = []
reader = csv.reader(csv_file)
next(reader)
for fields in reader:
if fields[0] != "":
url = fields[0]
else:
continue # skip the whole row if the url column is empty
if fields[1] != "":
ip = "http://" + fields[1] + ":8050" # adding http and port because this is the needed scheme
if fields[2] != "":
useragent = fields[2]
data.append({"url": url, "ip": ip, "ua": useragent})
return data
class MySpider(Spider):
name = 'splash_spider' # Name of Spider
# notice that we don't need to define start_urls
# just make sure to get all the urls you want to scrape inside start_requests function
# getting all the url + ip address + useragent pairs then request them
def start_requests(self):
# get the file path of the csv file that contains the pairs from the settings.py
with open(self.settings["PROXY_CSV_FILE"], mode="r") as csv_file:
# requests is a list of dictionaries like this -> {url: str, ua: str, ip: str}
requests = process_csv(csv_file)
for req in requests:
# no need to create custom middlewares
# just pass useragent using the headers param, and pass proxy using the meta param
yield SplashRequest(url=req["url"], callback=self.parse, args={"wait": 3},
headers={"User-Agent": req["ua"]},
splash_url = req["ip"],
)
# Scraping
def parse(self, response):
item = GameItem()
for game in response.css("tr[class^=deckdbbody]"):
# Card Name
item["card_name"] = game.css("a.card_popup::text").extract_first()
item["condition"] = game.css("a::text").extract_first() #Problem is here
item["stock"] = game.css("td[class^=deckdbbody].search_results_8::text").extract_first()
item["price"] = game.css("td[class^=deckdbbody].search_results_9::text").extract_first()
yield item
解决方案
您需要在 CSS 表达式中指定目标单元格:
item["condition"] = game.css("td[class^=deckdbbody].search_results_7 a::text").get()
推荐阅读
- python - 颜色填充两条曲线之间的区域 - Python
- c - 蓝牙C编译
- qthread - 从另一个类向主类发出信号
- html - 当三种替代颜色重复到列表中时,如何在列表中添加:nth-child()?
- reactjs - 为在 Enzyme 中模拟 onClick 事件编写测试用例?
- java - 检查多个变量是否为空的更短方法
- javascript - 为多个页面调用多个获取调用的最佳方法?(也等待/然后)
- python - 如何找到未能映射的行列?
- ssl - SSL 自签名证书单向或双向TLS
- jupyter-notebook - 是否可以更改 PyCharm 的 Jupyter Notebook 的预览窗格的背景颜色?