javascript - 使用显示更多按钮抓取网页

Question

我想用“显示更多”按钮抓取谷歌学者页面。使用该平台的帮助解决我之前提出的问题，我编写了以下代码，以便单击“显示更多”按钮。但是，我仍然遇到问题。对于带有多个“显示更多”按钮的配置文件，只有第一个被点击。我不明白为什么会这样。我将不胜感激任何帮助。

from selenium import webdriver
import time
from bs4 import BeautifulSoup
import pandas as pd

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
chrome_path = r"C:\Users\ish05\Desktop\python\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)

driver.get("https://scholar.google.com/citations?user=cp-8uaAAAAAJ&hl=en")
time.sleep(3)
show_more = driver.find_elements_by_tag_name('button')
for x in range(len(show_more)):
    if show_more[x].is_displayed():
      driver.execute_script("arguments[0].click();", show_more[x])
      time.sleep(3)

标签： javascriptpythonseleniumweb-scrapingxpath

Answer 1

它运行一个的原因是因为它在每一页上出现一个。

您需要使用无限循环，然后在页面上搜索，如果有，然后单击 else no more button break from the loop。

from selenium import webdriver
import time
chrome_path = r"C:\Users\ish05\Desktop\python\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)

driver.get("https://scholar.google.com/citations?user=cp-8uaAAAAAJ&hl=en")
time.sleep(3)
while True:
    try:       
        show_more = driver.find_element_by_xpath("//button[.//span[text()='Show more'] and not(@disabled)]")
        driver.execute_script("arguments[0].click();", show_more)
        print("Show more button clicked")
        time.sleep(2)
    except:
        print("No more Show more button")
        break

您将在控制台上看到以下输出

Show more button clicked
Show more button clicked
Show more button clicked
Show more button clicked
Show more button clicked
No more Show more button