python - BeautifulSoup 抓取.text 属性问题

Question

我有以下代码来抓取页面，https://www.hotukdeals.com

from bs4 import BeautifulSoup
import requests

url="https://www.hotukdeals.com/hot"
r = requests.get(url)
soup = BeautifulSoup(r.text,"html.parser")
deals = soup.find_all("article")
for deal in deals:
    priceElement = deal.find("span",{"class":"thread-price"})
    try:
        print(priceElement,priceElement.text)
    except AttributeError:
        pass

出于某种原因，这有效，在循环中抓取交易价格一定次数，然后停止工作。

程序输出：

<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£9.09</span> £9.09
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£39.95</span> £39.95
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£424.98</span> £424.98
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£8.10</span> £8.10
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£14.59</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£2.50</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl text--color-greyShade">£20</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£19</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£29</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl text--color-greyShade">£49.97</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">FREE</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£2.49</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£1.99</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£54.99</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£12.85</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£1.99</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£21.03</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£5.29</span>

从输出可以看出，在前四行之后，.text属性为空，但元素中有文本。

有人知道这件事吗？有什么想法或解决方案吗？

标签： pythonhtmlweb-scrapingbeautifulsoup

Answer 1

我不确定到底是什么导致了这个问题，但我找到了解决方法，只需找到文本字段的开头和结尾，然后使用 string.find() 来获取它的索引。这是一个实现：

from bs4 import BeautifulSoup
import requests

url = "https://www.hotukdeals.com/hot"
r = requests.get(url)
soup = BeautifulSoup(r.text, "html.parser")
deals = soup.find_all("article")
for deal in deals:
    priceElement = deal.find("span", {"class": "thread-price"})

    if priceElement is not None:
        price = str(priceElement)
        start_price = price.find('">') + len('">')  # finds the start of the price
        end_price = price.find("</span")  # finds the end of the price area
        price = price[start_price:end_price] 
    else:
        price = None
    try:
        print(priceElement, price)
    except AttributeError:
        pass