时间戳

首页 > 解决方案 > 单击下一页按钮时无法从网站上抓取标题

python - 单击下一页按钮时无法从网站上抓取标题

问题描述

我已经编写了一个脚本来python结合selenium从不同页面抓取不同帖子的链接，同时单击下一页按钮并从其内页获取每个帖子的标题。尽管我在这里尝试处理的内容是静态的，但我使用 selenium 来查看它在单击下一页时如何解析项目。I'm only after any soultion related to selenium.

网站地址

如果我定义一个空白列表并将所有链接扩展到它，那么最终我可以在单击下一页按钮完成时解析所有标题，重用它们内页中的这些链接，但这不是我想要的。

但是，我打算做的是收集每个页面的所有链接，并在单击下一页按钮时从其内页解析每个帖子的标题。简而言之，我希望同时做这两件事。

我试过：

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://stackoverflow.com/questions/tagged/web-scraping"

def get_links(url):
    driver.get(url)
    while True:
        items = [item.get_attribute("href") for item in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".summary .question-hyperlink")))]
        yield from get_info(items)

        try:
            elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,".pager > a[rel='next']")))
            driver.execute_script("arguments[0].scrollIntoView();",elem)
            elem.click()
            time.sleep(2)
        except Exception:
            break

def get_info(links):
    for link in links:
        driver.get(link)
        name = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.question-hyperlink"))).text
        yield name

if __name__ == '__main__':
    driver = webdriver.Chrome()
    wait = WebDriverWait(driver,10)
    for item in get_links(link):
        print(item)

当我运行上面的脚本时，它通过重用第一页的链接来解析不同帖子的标题，但是raise TimeoutException(message, screen, stacktrace) 当它到达这一elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,".pager > a[rel='next']")))行时会抛出这个错误。

如何从他们的内页中抓取每个帖子的标题，从第一页收集链接，然后单击下一页按钮以重复该过程直到完成？

标签： pythonpython-3.xseleniumselenium-webdriverweb-scraping

解决方案

您没有获得下一个按钮的原因是在遍历该循环末尾的每个内部链接时找不到下一个按钮。

您需要像下面那样获取每个 nexturl 并执行。

urlnext = ' https://stackoverflow.com/questions/tagged/web-scraping?tab=newest&page= {}&pagesize=30'.format(pageno) #页面将从2开始

试试下面的代码。

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = "https://stackoverflow.com/questions/tagged/web-scraping"

def get_links(url):
    urlnext = 'https://stackoverflow.com/questions/tagged/web-scraping?tab=newest&page={}&pagesize=30'
    npage = 2
    driver.get(url)
    while True:
        items = [item.get_attribute("href") for item in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".summary .question-hyperlink")))]
        yield from get_info(items)
        driver.get(urlnext.format(npage))
        try:
            elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,".pager > a[rel='next']")))
            npage=npage+1
            time.sleep(2)
        except Exception:

            break

def get_info(links):
    for link in links:
        driver.get(link)
        name = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.question-hyperlink"))).text
        yield name

if __name__ == '__main__':
    driver = webdriver.Chrome()
    wait = WebDriverWait(driver,10)

    for item in get_links(link):
        print(item)

推荐阅读