时间戳

首页 > 解决方案 > 如何使用 BeautifulSoup 抓取隐藏的数据元素

python - 如何使用 BeautifulSoup 抓取隐藏的数据元素

问题描述

Level2StockQuotes.com 提供免费的实时图书报价，我想使用 BeautifulSoup 在 python 中捕获这些报价。问题是即使我可以在浏览器检查器中看到实际数据值，我也无法将这些值抓取到 python 中。

BeautifulSoup 返回所有数据行，每个数据元素为空白。Pandas 为每个数据元素返回一个带有 NaN 的数据框。

import bs4 as bs
import urllib.request
import pandas as pd

symbol = 'AAPL'
url = 'https://markets.cboe.com/us/equities/market_statistics/book/'+ symbol + '/'
page = urllib.request.urlopen(url).read()
soup = bs.BeautifulSoup(page,'lxml')

rows = soup.find_all('tr')
print(rows)

for tr in rows:
    td = tr.find_all('td')
    row =(i.text for i in td)
    print(row)

#using pandas to get dataframe
dfs = pd.read_html(url)
for df in dfs:
    print(df)

比我更有经验的人可以告诉我如何提取这些数据吗？谢谢！

标签： pythonbeautifulsoup

解决方案

页面是动态的。您需要使用 Selenium 来模拟浏览器并在抓取 html 之前让页面呈现，或者您可以直接从 json XHR 获取数据。

import requests
import pandas as pd
from pandas.io.json import json_normalize



url = 'https://markets.cboe.com/json/bzx/book/AAPL' 

headers = {
'Referer': 'https://markets.cboe.com/us/equities/market_statistics/book/AAPL/',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest'}

jsonData = requests.get(url, headers=headers).json()

df_asks = pd.DataFrame(jsonData['data']['asks'], columns=['Shares','Price'] )
df_bids = pd.DataFrame(jsonData['data']['bids'], columns=['Shares','Price'] )
df_trades = pd.DataFrame(jsonData['data']['trades'], columns=['Time','Price','Shares','Time_ms'])

输出：

df_list = [df_asks, df_bids, df_trades]
for df in df_list:
    print (df)

   Shares   Price
0      40  209.12
1     100  209.13
2     200  209.14
3     100  209.15
4      24  209.16
   Shares   Price
0     200  209.05
1     200  209.02
2     100  209.01
3     200  209.00
4     100  208.99
       Time Price    Shares         Time_ms
0  10:45:57   300  209.0700  10:45:57.936000
1  10:45:57   300  209.0700  10:45:57.936000
2  10:45:55    29  209.1100  10:45:55.558000
3  10:45:52    45  209.0900  10:45:52.265000
4  10:45:52    50  209.0900  10:45:52.265000
5  10:45:52     5  209.0900  10:45:52.265000
6  10:45:51   100  209.1100  10:45:51.902000
7  10:45:48   100  209.1400  10:45:48.528000
8  10:45:48   100  209.1300  10:45:48.528000
9  10:45:48   200  209.1300  10:45:48.528000

推荐阅读