python - 如何在循环中从 BeautifulSoup 中的 span 标签获取文本

Question

我正在尝试从一个网站上抓取一些信息，该网站重复了以下 html 多次：

<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup</h2>
   <span class="reference-number">REF NO. A1400.5</span>

我正在尝试抓取产品名称和文本“REF NO. A1400.5”。我需要在同一页面上抓取多个产品名称和参考编号并将它们存储在一个列表中。我试过了：

product_new = []
product_ref = []

for caption in soup.find_all(class_='product-details'):
    product_name_new = caption.find(class_='product-name').text
    product_new.append(product_name_new)
    product_name_ref = (soup.select_one("span[class*=reference]").text)
    product_ref.append(product_name_ref)    
product_size_new = len(product_new)
print("Setup Complete", product_size_new)
print(*product_new,sep='\n')
print(*product_ref,sep='\n')

product_new完美运行并返回所有产品名称的列表，但product_ref仅包含REF NO. A1400.5重复该类reference出现在网页上的次数。如何更改它以正确存储reference每次出现在页面上的信息？

谢谢！

标签： pythonweb-scrapingbeautifulsoup

Answer 1

在您的代码中，product_name_ref始终是相同的值，因为您选择的是 from soup，而不是 from caption。

要获得所需的信息，您可以使用以下示例：

from bs4 import BeautifulSoup


txt = '''
<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup</h2>
   <span class="reference-number">REF NO. A1400.5</span>
</div>

<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup 2</h2>
   <span class="reference-number">REF NO. A1400.5 2</span>
</div>
'''

soup = BeautifulSoup(txt, 'html.parser')

product_new = []
product_ref = []

for product in soup.select('div.product-details'):
    product_new.append(product.h2.get_text(strip=True))
    product_ref.append(product.select_one('span.reference-number').get_text(strip=True))

print(product_new)
print(product_ref)

印刷：

['Weekly Roundup', 'Weekly Roundup 2']
['REF NO. A1400.5', 'REF NO. A1400.5 2']

---

编辑：

product_new = []
product_ref = []

for product in soup.select('div.product-details'):
    n = product.h2
    r = product.select_one('span.reference-number')

    if n and r:
        product_new.append(n.get_text(strip=True))
        product_ref.append(r.get_text(strip=True))

print(product_new)
print(product_ref)

---

编辑2：

from bs4 import BeautifulSoup


txt = '''
<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup</h2>
   <span class="reference-number">REF NO. A1400.5</span>
</div>

<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup 2</h2>
   <span class="reference-number">REF NO. A1400.6</span>
</div>
'''

soup = BeautifulSoup(txt, 'html.parser')

product_new = []
product_ref = []

for product in soup.select('div.product-details'):
    n = product.h2
    r = product.select_one('span.reference-number')

    if n and r:
        product_new.append(n.get_text(strip=True))
        product_ref.append(r.get_text(strip=True).rsplit(maxsplit=1)[-1])

print(product_new)
print(product_ref)

印刷：

['Weekly Roundup', 'Weekly Roundup 2']
['A1400.5', 'A1400.6']

---

编辑 3：

for a, b in zip(product_new, product_ref):
    print('{:<30} {}'.format(a, b))

印刷：

Weekly Roundup                 A1400.5
Weekly Roundup 2               A1400.6