时间戳

首页 > 解决方案 > 我想用 bs4 从亚马逊获取产品的标题

python - 我想用 bs4 从亚马逊获取产品的标题

问题描述

我想通过 BeuatifulSoup 和请求获得这个亚马逊产品的标题。当我运行时，这是说：

Traceback (most recent call last):
  File "scraper.py", line 15, in <module>
    title = soup.find('span', id='productTitle').get_text()
AttributeError: 'NoneType' object has no attribute 'get_text'

请帮助我

import bs4
import requests
from bs4 import BeautifulSoup
from urllib.request import urlopen

url = 'https://www.amazon.de/OnePlus-Smartphone-Almond-Display-Speicher/dp/B07RWL3K1Y/ref=sr_1_2? __mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=oneplus+7+pro&qid=1598088298&sr=8-2'

headers = {
    "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36'
}
page = requests.get(url, headers = headers)

soup = BeautifulSoup(page.content, 'html.parser')
title = soup.find('span', id='productTitle').get_text()
print(title)

标签： pythonweb-scrapingbeautifulsouppython-requests

解决方案

问题是使用“html.parser”作为您的 bs4 解析器。改用 lxml （它将更优雅地处理损坏的 html）。该错误试图告诉您它从未找到<span id='productTitle'>- 我们可以看到它在那里，因此它可能是与非标准 HTML 相关的解析失败。

import requests
from bs4 import BeautifulSoup

url = 'https://www.amazon.de/OnePlus-Smartphone-Almond-Display-Speicher/dp/B07RWL3K1Y/ref=sr_1_2? __mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&dchild=1&keywords=oneplus+7+pro&qid=1598088298&sr=8-2'

headers = {
"User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36'
}
page = requests.get(url, headers = headers)

soup = BeautifulSoup(page.content, 'lxml')
title = soup.find('span', id='productTitle').get_text().strip()
print(title)

输出：

OnePlus 7 Pro Smartphone Almond (16,9 cm) AMOLED Display 8 GB RAM + 256 GB Speicher, Triple Kamera (48 MP) Pop-up Kamera (16 MP) – Dual SIM Handy

推荐阅读