时间戳

首页 > 解决方案 > 如何使用 Selenium 和 Python 在 https://tengrnews.kz 中提取 7 个主要新闻的时间和标题

python - 如何使用 Selenium 和 Python 在 https://tengrnews.kz 中提取 7 个主要新闻的时间和标题

问题描述

我需要从这个网站上抓取 7 条主要新闻——tengrnews.kz ，每条新闻的日期、时间和标题。我使用 selenium 并安装了 Firefox 开发者版。

我检查了网站，7条新闻位于这个结构中：

<body>
   <header> ... some stuff </header>
   <main>
      <div class="tn-main-news-grid">

         <div class="tn-main-news-item firs-column tn-three-column tn-background-cover">  
            <span class="tn-main-news-title" style="z-index: 1;">BIG MAJOR NEWS TEXT</span>
            <a href="/kazakhstan_news/major-news/" class="tn-link"><span class="tn-hidden">BIG MAJOR NEWS TEXT</span></a>
         </div>



         <div class="tn-main-news-item"> 
            <span class="tn-main-news-title">news1 TEXT</span>
            <a href="/kazakhstan_news/news1/" class="tn-link">
            <span class="tn-hidden">news1 TEXT</span></a>
         </div>



         <div class="tn-main-news-item"> 
            <span class="tn-main-news-title">news2 TEXT</span>
            <a href="/kazakhstan_news/news2/" class="tn-link">
            <span class="tn-hidden">news2 TEXT</span></a>
         </div>



         <div class="tn-main-news-item"> 
            <span class="tn-main-news-title">news3 TEXT</span>
            <a href="/kazakhstan_news/news3/" class="tn-link">
            <span class="tn-hidden">news3 TEXT</span></a>
         </div>

      </div>
   </main>
</body>

我通过 xpath 或 css_selector 找到了包含所有 7 个新闻的 div 框架。我确实得到了 Firefox 网页元素，但它是一个列表而且它是空的！

如果我尝试定位单个 href 或 div，它会返回一些“列表”类型的 web 元素，并且这个 href 必须具有文本属性（根据 selenium 文档） - 但它给我错误“无属性文本”

from selenium import webdriver
driver = webdriver.Firefox()

driver.get("https://tengrinews.kz")

css_to_big_news = 'html body div.my-app main section.tn-main-section.tn-container div.tn-main-news-container.tn-sub-container div.tn-main-news-grid div.tn-main-news-item.firs-column.tn-three-column.tn-background-cover a.tn-link'


href_big = driver.find_elements_by_css_selector(css_to_big_news)
print('type of href_big is %s and length is %d' %(type(href_big), len(href_big)))

print(href_big[0].text) #this is wrong
print(href_big.text()) # this is wrong with parenthesis

怎么了？

标签： pythonseleniumxpathcss-selectorswebdriverwait

解决方案

要提取文本，例如TEXT，从每个<span>使用Selenium和你必须诱导WebDriverWait forvisibility_of_all_elements_located()并且你可以使用以下任一Locator Strategies：

  • 使用CSS_SELECTOR：

    driver.get("https://tengrinews.kz/")
print("Date and Time:")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.tn-main-news-grid div.tn-main-news-item ul.tn-data-list>li>span time")))])
print("Title:")
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.tn-main-news-grid div.tn-main-news-item span.tn-main-news-title")))])

  • 使用XPATH：

    driver.get("https://tengrinews.kz/")
print("Date and Time:")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='tn-main-news-grid ']//div[contains(@class, 'tn-main-news-item')]//ul[@class='tn-data-list']/li/span//time")))])
print("Title:")
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='tn-main-news-grid ']//div[contains(@class, 'tn-main-news-item')]//span[@class='tn-main-news-title']")))])

  • 控制台输出：

    Date and Time:
['вчера, 18:27', 'вчера, 21:45', 'вчера, 20:52', 'вчера, 19:48', 'вчера, 17:34', 'вчера, 14:50', 'вчера, 14:32']
Title:
['Жара до 42 градусов ожидается в регионах Казахстана', 'Строгий карантин вводят в Мангистауской области', 'Нехватку вакцин и новую "суровую" волну COVID-19 предрекли в мире', 'Столицу Казахстана "оживили"', 'Жители Актау собрались на площади из-за отсутствия лекарств в аптеках', 'Строгий карантин в Нур-Султане продлили до 2 августа', '"Едят антибиотики". Врач из Павлодара объяснил рост числа тяжелых больных']

  • 注意：您必须添加以下导入：

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

奥特罗

链接到有用的文档：

推荐阅读