selenium - 无法使用 Beautiful Soup 和 Selenium 访问整个页面
问题描述
我一直在尝试使用漂亮的汤和硒来收集数据，但无法获得超过表上的前 20 个结果（总共超过 2000 个）。我看到了一些相关的问题和答案，建议尝试不同的解析器，所以我尝试了 lxml、html.parser 和 html5lib，但都没有奏效。我还看到一些建议使用 selenium 和 webdriver 的答案，但无法使用任何一个来获取整个页面。这是我现在的代码
import requests, time
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import urllib.request
#import pandas as pd
from bs4 import BeautifulSoup
import requests, time
import lxml
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('https://coinmarketcap.com/exchanges/uniswap-v2/')
#supposed to scroll to the bottom of the page
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
page = driver.page_source
soup = BeautifulSoup(''.join(page), 'lxml')
pairs = soup.find_all('div', attrs = {"class" : "hmd6df-0 kCRNNr"})
data = [i.find_all('a')[0] for i in pairs]
任何帮助都是极好的。谢谢。
解决方案
我想出了一个纯粹的硒解决方案——下面的 python 脚本成功地从表中抓取了所有数据。我只是以这样的打印为例，您将想弄清楚如何组织您的数据。
更新：为了展开和抓取所有行，我滚动到列表底部，然后单击“展开更多”按钮，它展开了 100 多行。我迭代了足够多的时间来遍历 1850 行数据。
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import time
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('https://coinmarketcap.com/exchanges/uniswap-v2/')
total_height = int(driver.execute_script("return document.body.scrollHeight"))
for x in range(20):
for i in range(1, total_height, 130):
driver.execute_script("window.scrollTo(0, {});".format(i))
if x == 0:
driver.find_element_by_css_selector('div.cmc-cookie-policy-banner__close').click()
driver.find_element_by_xpath('//button[text() = "Load More"]').click()
time.sleep(2)
first_column = driver.find_elements_by_css_selector('td.cmc-table__cell.cmc-table__cell--sticky.cmc-table__cell--sortable.cmc-table__cell--left.cmc-table__cell--sort-by__rank > div')
second_column = driver.find_elements_by_css_selector('div.cwwgik-0.bCvAgC')
third_column = driver.find_elements_by_css_selector('div.hmd6df-0.kCRNNr')
fourth_column = driver.find_elements_by_css_selector('div.cmc-table__column-market-pair-volume-24h')
fifth_column = driver.find_elements_by_css_selector('div.cmc-table__column-market-pair-volume-percent')
sixth_column = driver.find_elements_by_css_selector('td.cmc-table__cell.cmc-table__cell--sortable.cmc-table__cell--right.cmc-table__cell--sort-by__quote-usd-effective-liquidity > div')
seventh_column = driver.find_elements_by_css_selector('td.cmc-table__cell.cmc-table__cell--sortable.cmc-table__cell--right.cmc-table__cell--sort-by__fee-type > div')
eighth_column = driver.find_elements_by_css_selector('div.ghkc60-0.fLaXDt')
for i in range(len(second_column)):
print(str(first_column[i].get_attribute("innerText")) + ' ' + str(second_column[i].get_attribute("innerText")) + ' ' + str(third_column[i].get_attribute("innerText")) + ' ' + str(fourth_column[i].get_attribute("innerText")) + ' ' + str(fifth_column[i].get_attribute("innerText")) + ' ' + str(sixth_column[i].get_attribute("innerText")) + ' ' + str(seventh_column[i].get_attribute("innerText")) + ' ' + str(eighth_column[i].get_attribute("innerText")))
输出：
1 USD Coin USDC/WETH $250,651,977 20.81% 935 Percentage Recently
2 Fei Protocol FEI/WETH $157,435,972 13.07% - Percentage Recently
3 WETH WETH/USDT $153,706,066 12.76% 915 Percentage Recently
4 Dai DAI/WETH $58,995,907 4.90% 850 Percentage Recently
5 Tendies TEND/WETH $41,429,102 3.44% - Percentage 778 hours ago
6 Wrapped Bitcoin WBTC/WETH $37,838,161 3.14% 901 Percentage Recently
7 SHIBA INU SHIB/WETH $30,416,126 2.53% 612 Percentage Recently
...
1770 yplutus yPLT/WETH $? 0.00% - Percentage 671 hours ago
1771 Deflect DEFLCT/RFI $? 0.00% - Percentage 96 days ago
1772 Blaze DeFi BNFI/WETH $? 0.00% - Percentage 781 hours ago
1773 Buy-Sell BSE/WETH $? 0.00% - Percentage Recently
1774 PIRANHAS $PIR/WETH $? 0.00% - Percentage Recently
1775 HLand Token HLAND/USDT $? 0.00% - Percentage Recently
1776 Hub - Human Trust Protocol HUB/WETH $? 0.00% - Percentage 126 days ago
1777 WETH WETH/YVS $? 0.00% - Percentage Recently
1778 Reflector.Finance RFCTR/WETH $? 0.00% - Percentage Recently
1779 WETH WETH/R34P $? 0.00% - Percentage Recently
1780 WETH WETH/RFR $? 0.00% 265 Percentage Recently
1781 Golden Ratio Per Liquidity GRPL/WETH $? 0.00% - Percentage Recently
1782 xETH-G xETH-G/WETH $? 0.00% - Percentage 75 days ago
1783 3XT TOKEN 3XT/WETH $? 0.00% - Percentage 62 days ago
1784 Diffract Finance DFR/WETH $? 0.00% - Percentage Recently
1785 Bitpower BPP/WETH $? 0.00% - Percentage Recently
1786 IDL Token IDL/ELYX $? 0.00% - Percentage 77 days ago
1787 IDL Token IDL/WETH $? 0.00% - Percentage 81 days ago
1788 Dai DAI/BCC $? 0.00% - Percentage 42 days ago
1789 Stand Share SAS/USDT $? 0.00% - Percentage 123 days ago
1790 Definex Dswap/USDT $? 0.00% - Percentage 314 hours ago
1791 Vaultz VAULTZ/WETH $? 0.00% - Percentage 781 hours ago
1792 Fission Cash FCX/WETH $? 0.00% - Percentage Recently
1793 DeltaHub Community DHC/USDT $? 0.00% - Percentage 821 hours ago
1794 Dai DAI/DST $? 0.00% - Percentage 111 days ago
1795 AGAr AGAr/WETH $? 0.00% - Percentage 311 hours ago
1796 Basis Cash BAC/Mars $? 0.00% - Percentage 809 hours ago
1797 Tether USDT/NOW $? 0.00% - Percentage 781 hours ago
1798 USD Coin USDC/wCUSD $? 0.00% - Percentage Recently
1799 zzz.finance v2 ZZZV2/WETH $? 0.00% - Percentage 781 hours ago
1800 Rigel Finance RIGEL/WETH $? 0.00% - Percentage Recently
1801 Bitbot Protocol BBP/WETH $? 0.00% - Percentage Recently
1802 HeroSwap HERO/WETH $? 0.00% - Percentage 108 days ago
1803 XUSD Stable XUSD/LINK $? 0.00% - Percentage 69 days ago
1804 XUSD Stable XUSD/DAI $? 0.00% - Percentage 96 days ago
1805 CURE Farm CURE/WETH $? 0.00% - Percentage Recently
1806 WETH WETH/stETH $? 0.00% 518 Percentage Recently
1807 Xstable.Protocol XST/WETH $? 0.00% - Percentage Recently
1808 ZCore WZCR/WETH $? 0.00% - Percentage 94 days ago
1811 Wrapped BIND wBIND/WETH $? 0.00% - Percentage Recently
1812 USDFreeLiquidity USDFL/USDT $? 0.00% - Percentage Recently
1813 USDFreeLiquidity USDFL/DAI $? 0.00% - Percentage 113 hours ago
1814 USDFreeLiquidity USDFL/USDN $? 0.00% - Percentage Recently
1815 WETH WETH/MCX $? 0.00% - Percentage Recently
1816 Mythic Finance MYTHIC/WETH $? 0.00% - Percentage 781 hours ago
1817 Polkabase PBASE/WETH $? 0.00% - Percentage Recently
1818 WETH WETH/RAC $? 0.00% - Percentage Recently
1819 MiraQle MQL/WETH $? 0.00% - Percentage 68 days ago
1820 MiraQle MQL/USDT $? 0.00% - Percentage 68 days ago
1821 Parsiq Boost PRQBOOST/WETH $? 0.00% - Percentage Recently
1822 WETH WETH/PUX $? 0.00% - Percentage Recently
1823 Previse PRVS/WETH $? 0.00% - Percentage Recently
1824 SIMBA Storage SIMBA/USDT $? 0.00% - Percentage 781 hours ago
1825 CryptoPing PING/USDT $? 0.00% - Percentage Recently
1826 Wrapped Bitcoin WBTC/INSTAR $? 0.00% - Percentage Recently
1827 Vow VOW/WETH $? 0.00% - Percentage Recently
1828 Value Set Dollar VSD/DAI $? 0.00% - Percentage 471 hours ago
1829 Value Set Dollar VSD/USDT $? 0.00% - Percentage 110 hours ago
1830 Value Set Dollar VSD/WETH $? 0.00% - Percentage 44 days ago
1831 WETH WETH/DEGENS $? 0.00% - Percentage Recently
1832 Xriba XRA/WETH $? 0.00% - Percentage 630 hours ago
1833 Tower token TOWER/LYM $? 0.00% - Percentage 469 hours ago
1834 Shadetech SHD/WETH $? 0.00% - Percentage Recently
1835 Wrapped Bitcoin WBTC/DGCL $? 0.00% - Percentage 333 hours ago
1836 Rare Pepe rPepe/WETH $? 0.00% - Percentage Recently
1837 xSigma SIG/USDT $? 0.00% - Percentage 52 days ago
1838 Dollar Protocol USDf/USDC $? 0.00% - Percentage 156 hours ago
1839 MYFinance MYFI/WETH $? 0.00% - Percentage Recently
1840 Delta DELTA/WETH $? 0.00% - Percentage Recently
1841 Kambria Yield Tuning Engine KYTE/WETH $? 0.00% - Percentage Recently
1842 ClinTex CTi CTI/WETH $? 0.00% - Percentage Recently
1843 BasenjiDAO BSJ/WETH $? 0.00% - Percentage Recently
1844 EURxb EURxb/USDT $? 0.00% - Percentage Recently
1845 Landbox LAND/WETH $? 0.00% - Percentage 442 hours ago
1846 Folder Protocol FOL/WETH $? 0.00% - Percentage Recently
1847 Databroker DTX/USDT $? 0.00% - Percentage Recently
1848 STATERA STA/WSTA $? 0.00% - Percentage Recently
1849 Delta Exchange Token DETO/WETH $? 0.00% - Percentage Recently
1850 Elongate Deluxe ELongD/WETH $? 0.00% - Percentage Recently
推荐阅读
- sql - 嵌套查询评估
- c# - 将 VBA 宏添加到 powerpoint
- c# - 编写自定义身份验证 + 我如何检查到哪个端点请求
- python - 将简单的 IDL 参数代码转换为 python 代码
- mysql - PhpMyAdmin 突然显示错误
- angular - 如何在源数组更改后从特定索引以角度启动引导轮播
- node.js - Webpack + Electron：如果应用程序没有它，为什么我需要“电子渲染器”脚本？
- ios - 我可以在没有 https 的情况下将我的应用程序发布到 App Store 吗？
- php - 我运行报告时需要保留三位小数
- php - 使用带有特定 postID 的 get_field 返回空