python - 抓取网站时获取 Null 输出
问题描述
我正在为这个网站编写一个网页抓取程序http://www.starcitygames.com/catalog/category/Duel%20Decks%20Venser%20vs%20Koth,并且在网站上的数据表中,有一张卡片名称的实例有两种不同的条件/价格/库存...我需要这两行信息,但正如您在附图中注意到的那样,卡名仅位于顶行而不是底行。因此,例如使用 Aether Membrane,它将输出卡名加上所有其他信息,但在下一行我得到所有其他信息,但对于卡名,我得到 Null。有没有办法像设置一个 if 语句或其他东西来查看卡名是否为空,如果是,则设置卡名等于先前的条目。我需要一些东西来用正确的卡片名称替换 Null 字符。
SplashSpider.py
import csv
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import GameItem
# process the csv file so the url + ip address + useragent pairs are the same as defined in the file # returns a list of dictionaries, example:
# [ {'url': 'http://www.starcitygames.com/catalog/category/Rivals%20of%20Ixalan',
# 'ip': 'http://204.152.114.244:8050',
# 'ua': "Mozilla/5.0 (BlackBerry; U; BlackBerry 9320; en-GB) AppleWebKit/534.11"},
# ...
# ]
def process_csv(csv_file):
data = []
reader = csv.reader(csv_file)
next(reader)
for fields in reader:
if fields[0] != "":
url = fields[0]
else:
continue # skip the whole row if the url column is empty
if fields[1] != "":
ip = "http://" + fields[1] + ":8050" # adding http and port because this is the needed scheme
if fields[2] != "":
useragent = fields[2]
data.append({"url": url, "ip": ip, "ua": useragent})
return data
class MySpider(Spider):
name = 'splash_spider' # Name of Spider
# notice that we don't need to define start_urls
# just make sure to get all the urls you want to scrape inside start_requests function
# getting all the url + ip address + useragent pairs then request them
def start_requests(self):
# get the file path of the csv file that contains the pairs from the settings.py
with open(self.settings["PROXY_CSV_FILE"], mode="r") as csv_file:
# requests is a list of dictionaries like this -> {url: str, ua: str, ip: str}
requests = process_csv(csv_file)
for req in requests:
# no need to create custom middlewares
# just pass useragent using the headers param, and pass proxy using the meta param
yield SplashRequest(url=req["url"], callback=self.parse, args={"wait": 3},
headers={"User-Agent": req["ua"]},
splash_url = req["ip"],
)
# Scraping
def parse(self, response):
item = GameItem()
for game in response.css("tr[class^=deckdbbody]"):
# Card Name
item["card_name"] = game.css("a.card_popup::text").extract_first()
item["condition"] = game.css("td[class^=deckdbbody].search_results_7 a::text").get()
item["stock"] = game.css("td[class^=deckdbbody].search_results_8::text").extract_first()
item["price"] = game.css("td[class^=deckdbbody].search_results_9::text").extract_first()
yield item
样本输出
{"card_name": "\nAether Membrane", "Condition": "NM/M", "stock": "93", "Price": "$0.59"},
{"card_name": null, "Condition": "PL", "stock": "59", "Price": "$0.49"},
{"card_name": "\nAngelic Shield", "Condition": "NM/M", "stock": "35", "Price": "$0.25"},
{"card_name": "\nAnger", "Condition": "NM/M", "stock": "9", "Price": "$1.49"},
{"card_name": null, "Condition": "PL", "stock": "49", "Price": "$1.19"},
{"card_name": "\nArmillary Sphere", "Condition": "NM/M", "stock": "87", "Price": "$0.25"},
{"card_name": "\nAugury Owl", "Condition": "NM/M", "stock": "Out of Stock", "Price": "$0.29"},
解决方案
就像评论说这解决了问题:
for game in response.css("tr[class^=deckdbbody]"):
saved_name = game.css("a.card_popup::text").extract_first() or saved_name
item["card_name"] = saved_name.strip()
if item["card_name"] != None:
saved_name = item["card_name"].strip()
else:
item["card_name"] = saved_name
item["Condition"] = game.css("td[class^=deckdbbody].search_results_7 a::text").get()
item["stock"] = game.css("td[class^=deckdbbody].search_results_8::text").extract_first()
item["Price"] = game.css("td[class^=deckdbbody].search_results_9::text").extract_first()
yield item
推荐阅读
- c# - 通过位移组合两个可变长度的字节数组
- php - 如何将数据从控制器传递到codeigniter3中的自定义钩子
- javascript - 如何在 vis.js 时间线中获取开始和结束时间?
- mysql - MYSQL - 如何使用 group by 获取每个类型字段中的最新记录
- vba - 循环隐藏工作表
- java - 仅当子字符串是整个单词时才替换字符串中的所有子字符串
- r - 如何将 PAD 函数(来自 PADR() 包)用于多个数据帧?
- java - API 测试。如何管理有效负载
- sql-server - 在 SQL Server 中根据时间可用性过滤表记录
- c# - BindingSource.Path 和 BindingSource.Query 之间的区别