python - Selenium 代码无法抓取堆栈溢出
问题描述
我正在用 python 构建一个网络爬虫selenium
。该脚本抓取了 amazon 和 Flipcart 等网站,但无法抓取堆栈溢出和 offashion。它总是给我一个空白的 .csv 文件。
这是我的堆栈溢出代码:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import pandas as pd
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) \ Chrome/80.0.3987.132 Safari/537.36'
driver_exe = 'chromedriver'
options = Options()
options.add_argument("--headless")
options.add_argument(f'user-agent={user_agent}')
options.add_argument("--disable-web-security")
options.add_argument("--allow-running-insecure-content")
options.add_argument("--allow-cross-origin-auth-prompt")
options.add_argument("--disable-cookie-encryption")
driver = webdriver.Chrome(executable_path=r"C:\Users\intel\Downloads\Setups\chromedriver.exe", options=options)
driver.get("https://stackoverflow.com/")
class_Name = "." + "question-hyperlink"
x = driver.find_elements_by_css_selector(class_Name.replace(' ','.'))
web_content_list = []
for i in x:
web_content_dict = {}
web_content_dict["Title"] = i.text
web_content_list.append(web_content_dict)
df = pd.DataFrame(web_content_list)
df.to_csv(r'C:\Users\intel\Desktop\data_file.csv', index=False, mode='a', encoding='utf-8')
上面的 stackovrflow 代码是通过在链接中放入 /questions 来解决的。
OFashion 网站的代码
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import pandas as pd
import time
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) ' \
'Chrome/80.0.3987.132 Safari/537.36'
driver_exe = 'chromedriver'
options = Options()
#options.add_argument("--headless")
options.add_argument(f'user-agent={user_agent}')
options.add_argument("--disable-web-security")
options.add_argument("--allow-running-insecure-content")
options.add_argument("--allow-cross-origin-auth-prompt")
driver = webdriver.Chrome(executable_path=r"C:\Users\intel\Downloads\Setups\chromedriver.exe", options=options)
driver.get("https://www.ofashion.com.cn/goods/10001?t=15777838840003")
class_Name = "." + "content-par"
x = driver.find_elements_by_css_selector(class_Name.replace(' ','.'))
web_content_list = []
for i in x:
web_content_dict = {}
web_content_dict["Title"] = i.text
web_content_list.append(web_content_dict)
df = pd.DataFrame(web_content_list)
df.to_csv(r'C:\Users\intel\Desktop\data_file.csv',
index=False, mode='a', encoding='utf-8')
解决方案
这是您的问题-您尚未登录。 StackOverflow 仅在您登录时向您显示问题-因此,如果您使程序单击“登录”按钮,输入您的用户名和密码并提交,则代码将从那里开始工作。(我通过取出无头属性发现了这一点)