时间戳

首页 > 解决方案 > Python 抓取问题

python - Python 抓取问题

问题描述

我正在尝试从 NFL Teams Site 中抓取一些数据，并且在 python 输出中不断得到一个空字符串 []。

我正在尝试获取团队名称以及与之相关的链接。

这是我正在尝试的 HTML findAll()：

<td customsortid="fullName" id="standingsTile" data="[object Object]" class="rmq-30d300f6" data-radium="true" style="border-top: 1px solid rgb(238, 238, 238); box-sizing: border-box; height: auto; line-height: 40px; padding-left: 16px; padding-right: 16px; text-align: right; white-space: nowrap; vertical-align: top; box-shadow: rgb(238, 238, 238) 4px 0px 1px; left: 0px; position: absolute; width: 160px;">
    <a href="http://www.nfl.com/teams/profile?team=NE" tabindex="0" data-metrics-link-module="divisionStandings0" data-radium="true" style="cursor: pointer; text-decoration: none;">
    <div class="rsv-ae9db127" data-radium="true">
        <div class="rsv-ae9db127" data-radium="true" style="align-items: center; padding: 0px; background-color: transparent; height: 40px; -webkit-box-align: center;">
            <div data-radium="true">
                <img alt="" data-test-id="facemask-image-container" sizes="100vw" src="https://static.nfl.com/static/content/public/static/wildcat/assets/img/logos/teams/NE.svg" width="48" data-radium="true" style="border: 0px; display: block; max-width: 100%; margin-right: 8px; width: 24px;">
            </div>
            <div class="rsv-ae9db127" data-radium="true">
                <div class="rmq-9da922a7" data-radium="true" style="line-height: 1; font-size: 12px; margin-bottom: 0px; color: rgb(51, 51, 51); text-transform: none; font-family: &quot;Endzone Sans Medium&quot;, sans-serif; display: none; margin-right: 4px; text-decoration: none;">
                    New England 
                </div>
                <div data-radium="true" style="color: rgb(51, 51, 51); font-family: &quot;Endzone Sans Medium&quot;, sans-serif; font-size: 12px; line-height: 1; margin-top: 0px; text-decoration: none; text-transform: none;">
                    Patriots
                </div>
            </div>
        </div>
        <div class="rsv-ae9db127" data-radium="true" style="font-size: 10px; line-height: 1; padding-left: 4px; padding-top: 8px;">
            z
        </div>
    </div>
    </a>
</td>

这是我的代码，它一直给我一个空白列表[]

from bs4 import BeautifulSoup as bsoup
from urllib.request import urlopen as uReq

nfl_url = ("https://www.nfl.com/standings")
webpage = uReq(nfl_url)
page_html = webpage.read()
page_parser = bsoup(page_html , "html.parser")
odd = page_parser.findAll("td", {"class": "rmq-30d300f6"})
print(odd)

我试图从 https://www.nfl.com/standings获取团队名称和链接

关于我做错了什么的任何想法？

标签： pythonbeautifulsoup

解决方案

您所追求的 HTML 内容是由 Javascript 生成的，因此它不在您从网站上抓取的原始 HTML 中。

你应该考虑使用 Selenium 来加载你的页面，并等到你需要的元素被加载。像这样的东西：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

from bs4 import BeautifulSoup as bsoup


nfl_url = "https://www.nfl.com/standings"
browser = webdriver.Chrome('/home/your_username/Downloads/chromedriver')
browser.get(nfl_url)
delay = 10
try:
    # Using 'fullName' ID as a check that the relevant portion of the page is loaded.
    # There may be other elements that are more appropriate - up to you to figure out.
    myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'fullName')))
except TimeoutException:
    print( "Loading took too much time!")

page_html = browser.page_source
browser.close()

page_parser = bsoup(page_html , "html.parser")
odd = page_parser.findAll("td", {"class": "rmq-30d300f6"})
print(odd)

请注意，您需要指定 的路径chromedriver，如果您的系统上没有它，请下载它。

推荐阅读