python - 无法通过 Selenium Python 在 booking.com 中找到以查看文本

Question

我正在练习从 booking.com 获取评论，所有评论都隐藏在侧栏中。我只选择了一家酒店，看到有 93 条评论，所以我需要在该侧栏上循环查看多个页面。即使我在 Chrome 开发者控制台中检查标签是否匹配，我在 switch_to_window 上工作的代码也无法找到这些评论。我可以有你的建议我应该怎么做？

import time
from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome("./driver/chromedriver.exe")

driver.get("https://booking.com")

time.sleep(2)
driver.find_element_by_css_selector("input[class*='sb-searchbox__input']").send_keys("Eden The Residence at The Sea")
driver.find_element_by_css_selector("button[type='submit']").click()
driver.find_element_by_xpath("//span[contains(text(),'Eden The Residence at The Sea')]").click()
time.sleep(3)
driver.switch_to.window(driver.window_handles[1])
driver.find_element_by_css_selector("div[class='room_info_heading'")

标签： pythonseleniumweb-scraping

Answer 1

您可以使用Explicit Waits而不是 time.sleep() 来等待元素被加载。

import time
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

...

driver.switch_to.window(driver.window_handles[1])
time.sleep(3)
// Add this line to show reviews
driver.find_element_by_css_selector('#show_reviews_tab').click() 
// Use WebDriverWait to wait element be visibility
room_info = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[class='room_info_heading']"))) 
print(room_info.text)

输出：

Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub

您可以使用如下循环：

room_info_list = []
while True:
    // visibility_of_all_element's'_located
    room_info_eles = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div[class='room_info_heading']")))
    room_info = [room_info_eles[i].text for i in range(0, len(room_info_eles))]
    print(room_info)
    room_info_list.extend(room_info)
    try:
        WebDriverWait(driver, 3).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[class='pagenext']"))).click()
    except Exception:  // If next button cannot be click, break loop
        break

print(room_info_list)

输出：

['Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Three-Bedroom Villa with Private Pool and Rooftop Lounge with Tub', 'Stayed in: Beachside Five-Bedroom Villa with Private Pool and Rooftop with Tub']