时间戳

首页 > 解决方案 > Beautiful Soup 没有通过 ID 找到元素

python-3.x - Beautiful Soup 没有通过 ID 找到元素

问题描述

我正在尝试从 forexfactory.com 抓取传入的头条新闻。

from bs4 import *
import requests

source = requests.get("https://www.forexfactory.com/news").text

soup = BeautifulSoup(source, 'lxml')

list = soup.find(id="ui-outer")

print(list)

当它应该返回包含所有标题的 Div 时，这将返回 None。知道会有什么问题吗？我尝试过按 div、Ul id、li id 和其他一些方式进行搜索。它总是返回无。谢谢你。

标签： python-3.xmacosweb-scrapingbeautifulsoup

解决方案

对于动态创建的元素，试试Selenium

from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'C:\Program Files\ChromeDriver\chromedriver.exe')
URL = 'https://www.forexfactory.com/news'

driver.get(URL)

等待几秒钟，直到动态内容也被加载

driver.implicitly_wait(5) # wait for seconds

获取您的元素

uiOuter = driver.find_element_by_id('ui-outer')

所有链接的示例（故事图块）

aHref = driver.find_elements_by_css_selector('div.flexposts__story-title a')
[x.text for x in aHref]

输出

    ['EU\'s Barnier says "fundamental divergences" persist in UK trade talks',
 'With end of crisis programs, Fed faces tricky post-pandemic transition',
 'Markets Look Past Near-Term Challenges',
 'Time is short, Divergences remain, but we continue to work hard for a deal',
 'EURUSD consolidates around 1.1900; indicators are flat',
 'Rush for emerging market company bonds as investors look beyond COVID-19',
 'Europe’s Virus Lockdowns Push Economy Into Another Contraction',
 'Interactive Brokers enhances Client Portal',
 'BoE’s Haldane: Risk That Anxiety Leads To Gloom Loop',
 'Sharpest fall in UK private sector output since May. Manufacturing growth offset by renewed...',
 'Remote Working Shift Offers Silver Lining for Finance Gender Gap',
 'EU Flash PMI signals steep downturn in November amid COVID-19 lockdowns',
 'German PMI drops to five-month low in November due to tightening of COVID-19 restrictions, but...',
 'Sharp decline in French business activity amid fresh COVID-19 lockdown',
 'Rishi Sunak says Spending Review will not spell austerity',
 'Remote Working Shift Offers Silver Lining for Finance Gender Gap',
 'Japan’s Labor Thanksgiving Day goes viral',
 'Ranking Asset Classes by Historical Returns (1985-2020)',
 'Time is short, Divergences remain, but we continue to work hard for a deal',
 'EURUSD consolidates around 1.1900; indicators are flat',
 'US Dollar stuck near support, NZ$ strikes two-year high',
 'US Dollar stuck near support, NZ$ strikes two-year high',
 'Georgia confirms results in latest setback for Trump bid to overturn Biden win',
 'Canada to roll over terms of EU trade deal with UK',
 'Time is short, Divergences remain, but we continue to work hard for a deal',
 'German PMI drops to five-month low in November due to tightening of COVID-19 restrictions, but...',
 "COVID-19: 'No return to austerity', says chancellor as he hints at public sector pay freeze",
 'EURUSD consolidates around 1.1900; indicators are flat',
 'New Zealand Dollar May Rise as RBNZ Holds Fire on Negative Rates',
 'Interactive Brokers enhances Client Portal']

推荐阅读