selenium-webdriver - Webdriver Actionchains - 单击下一页不起作用

Question

我是新来的，我试图编写一个代码，该代码应该遍历所有 3090 页并提取表中给出的信息（框-b）。

但它不起作用。它似乎停留在第一页，并一遍又一遍地提取这些信息。

如果你们中的任何人知道我该如何解决它，请帮助我:)

劳拉

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.common.keys import Keys
import time

url = "https://www.dieversicherer.de/versicherer/auto---reise/typklassenabfrage#orderBy=kh&orderDirection=ASC"

driver = webdriver.Chrome('/Users/laraschneider/Downloads/chromedriver')
driver.get(url)

time.sleep(5)

html = driver.page_source
soup = BeautifulSoup(html, "html.parser")
number = 0
while number < 3090:
    results_html = soup.find_all('div', "ttw-entry__box-b")

    results = list()

    for result in results_html:
        results.append(result.text)

    for t in results:
        print(t)

    time.sleep(5)

    element = driver.find_element_by_xpath("/html/body/main/div/div[5]/div/div/div/div[2]/div[2]/div[2]/div[2]/button[2]/div")
    action = ActionChains(driver)
    action.move_to_element(element)
    action.click(element).perform()


    number = number+1

标签： selenium-webdriverbeautifulsoupselenium-chromedriverweb-crawler

Answer 1

使用driver.execute_script("arguments[0].click();", element)而不是ActionChains. 这是完整的代码：

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

url = "https://www.dieversicherer.de/versicherer/auto---reise/typklassenabfrage#orderBy=kh&orderDirection=ASC"

driver = webdriver.Chrome('/Users/laraschneider/Downloads/chromedriver')
driver.get(url)

time.sleep(5)

html = driver.page_source
soup = BeautifulSoup(html, "html.parser")
number = 0
while number < 3090:
    results_html = soup.find_all('div', "ttw-entry__box-b")

    results = list()

    for result in results_html:
        results.append(result.text)

    for t in results:
        print(t)

    time.sleep(5)

    element = driver.find_element_by_xpath('/html/body/main/div/div[5]/div/div/div/div[2]/div[2]/div[2]/div[2]/button[2]/div')
    driver.execute_script("arguments[0].click();", element)
    number = number+1