python - 我没有使用 selenium 获取网站的所有元素
问题描述
这是代码:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver.support import expected_conditions as EC
import time
import sys
login_url = 'https://www.researchgate.net/login'
base_url = "https://www.researchgate.net/institution/Islamia_College_Peshawar/department/Department_of_Computer_Science/members"
chrome_driver_path = 'home/danish-khan/scrapers/researchgate/chromedriver'
chrome_options = Options()
#chrome_options.add_argument('--headless')
webdriver = webdriver.Chrome(
executable_path=chrome_driver_path, options=chrome_options
)
# default login credential and search query
username = 'your username'
password = 'your password'
with webdriver as driver:
# Set timeout time
wait = WebDriverWait(driver, 2)
# retrive url in headless browser
driver.get(login_url)
driver.find_element_by_id("input-login").send_keys(username)
driver.find_element_by_id("input-password").send_keys(password)
driver.find_element_by_class_name("nova-c-button__label").find_element(By.XPATH, "./..").click()
time.sleep(2)
driver.get(base_url)
time.sleep(10)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(20)
names = driver.find_elements_by_css_selector('.display-name')
print('total names:',len(names))
time.sleep(10)
driver.close()
这是输出
总名称:20 回溯(最近一次通话最后一次):文件“/home/danish-khan/scrapers/scrpers/lib/python3.8/site-packages/urllib3/connection.py”,第 159 行,在 _new_conn conn = connection .create_connection(文件“/home/danish-khan/scrapers/scrpers/lib/python3.8/site-packages/urllib3/util/connection.py”,第84行,在create_connection引发错误文件“/home/danish-khan /scrapers/scrpers/lib/python3.8/site-packages/urllib3/util/connection.py”,第 74 行,在 create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection denied
我试图找出为什么它不返回所有元素,因为在网站上有超过 30 个名称/配置文件,但它只显示 20 个名称。我应用等待直到找到元素的策略但无法工作
有什么解决办法吗?
解决方案
推荐阅读
- docker - Bazel Monorepo - 如何仅重建和发布已更改的 Docker 映像?
- r - Negative lookahead in strsplit puzzling behaviour
- javascript - 什么触发了 onsubmit 事件?如何配置?
- puppeteer - 赛普拉斯可以向全局对象公开函数吗?
- python - 使用 python 在二维列表中列出日期和小时
- python-3.x - 为什么将一个列表的一部分分配给另一个列表而不改变原始列表?
- python - 表格行和列的 Lambda 表达式
- ruby-on-rails - Ruby on Rails:错误“require”:无法加载此类文件--bundler/setup (LoadError)
- sql-server - 当过程包含与不存在的序列的连接时,SQL Server“CREATE PROCEDURE”不执行?
- c# - Visual Studio 2017 无法与 SQL Server 2008 通信:客户端和服务器无法通信,因为它们不具备通用算法