python - 使用 selenium/python 点击​​“下一步”按钮后，Url 不会改变

Question

我从这个 url 抓取数据，然后单击Next按钮并等待 10 秒，然后使用 requests 和 bs4 抓取下一页，但 url 没有改变，所以我最终只抓取了两次原始页面数据。我已经尝试WebDriverWait直到第一页上的元素变得陈旧以及尝试使用请求直接获取 xhr log api 调用（但是我不精通 ajax）并且找不到解决方案。这是代码：

loop = True
        while loop:
            try:
                current_url = driver.current_url
                next_btn = WebDriverWait(driver,5).until(EC.element_to_be_clickable((By.XPATH, '//button[text()="Next"]')))
                actions = ActionChains(driver)
                actions.move_to_element(next_btn).perform()
                if next_btn:
                    next_btn.click()
            except Exception as e:
                current_url = driver.current_url
                loop = False
                print(e,f"somewhere in {current_url} while loop")
            else:
                time.sleep(10)
                next_page = driver.current_url
                get_page_content(next_page)
                break

这是第一页的 URL：https ://www.hunterdouglas.com/locator/results?address=San%20Ramon&country=US&source=

任何方向将不胜感激！谢谢！

标签： pythonselenium

Answer 1

对于任何有兴趣的人，我只使用硒就可以做到这一点。这是代码，参数data只是我要提交的城市名称master_function(data)

def get_links(page):
    for p in page:
        for l in p.find_elements_by_tag_name("a"):
            link = l.get_attribute('href')
            if link != None:
                link_split = link.split('\n')
                for l in link_split:
                    if "http" in link:
                        test_list.append(link)
                        
def master_function(data):
    for d in data:
        base_url = "https://www.hunterdouglas.com/locator"
        driver.get(base_url)
        url = pop_up_one(driver)
        submit(url,driver,d)
        loop = True
        while loop:
            try:
                current_url = driver.current_url
                next_btn = WebDriverWait(driver,5).until(EC.element_to_be_clickable((By.XPATH, '//button[text()="Next"]')))
                actions = ActionChains(driver)
                actions.move_to_element(next_btn).perform()
                if next_btn:
                    next_btn.click()
            except Exception as e:
                current_url = driver.current_url
                loop = False
                print(e,f"somewhere in {current_url} while loop")
            else:
                time.sleep(1)
                page = WebDriverWait(driver,5).until(EC.presence_of_all_elements_located((By.XPATH, '//div[@id="loc-results"]')))
                get_links(page)