时间戳

首页 > 解决方案 > 如何从滚动网络表格中抓取特定的单词？

python - 如何从滚动网络表格中抓取特定的单词？

问题描述

我想从表格中抓取股票代码并检查“最终结算价格”行中是否存在某些关键字{“GD”，“每日价格”，“TSI”}。在表中，有多个动态列。下面是一个例子：我想抓取“Ambuja Cements Futures”的股票代码，即“ZACE”并将其存储在列表中，并检查“最终结算价格”行中是否存在某些关键字。 您可以在图像中看到 ZACE

向下滚动到表格后，您可以看到最终结算价格行

以下是股票代码行的 HTML 代码：

<tr>
   <th>Ticker Symbol</th>
   <td>ZADS</td>
   <td>ZACE</td>
   <td>ZAPN</td>
   <td>ZARB</td>
   <td>ZAXS</td>
   <td>ZBJA</td>
   <td>ZBAF</td>
   <td>ZBPC</td>
   <td>ZBHA</td>
   <td>ZBHI</td>
   <td>ZBOS</td>
   <td>ZCIP</td>
   <td>ZCOA</td>
   <td>ZDRR</td>
   <td>ZEIM</td>
   <td>ZGAI</td>
   <td>ZHCL</td>
   <td>ZHDB</td>
   <td>ZHMC</td>
   <td>ZHND</td>
   <td>ZHPC</td>
   <td>ZHUV</td>
   <td>ZHDF</td>
   <td>ZICI</td>
   <td>ZIHF</td>
   <td>ZIOC</td>
   <td>ZIIB</td>
   <td>ZINF</td>
   <td>ZITC</td>
   <td>ZKMB</td>
   <td>ZLT</td>
   <td>ZLPC</td>
   <td>ZMM</td>
   <td>ZMSI</td>
   <td>ZNTP</td>
   <td>ZONG</td>
   <td>ZPWG</td>
   <td>ZRIL</td>
   <td>ZSBI</td>
   <td>ZSUN</td>
   <td>ZTCS</td>
   <td>ZTTM</td>
   <td>ZTAT</td>
   <td>ZTEC</td>
   <td>ZUTC</td>
   <td>ZUPL</td>
   <td>ZVED</td>
   <td>ZWPR</td>
   <td>ZYES</td>
   <td>ZZEE</td>
</tr>

以下是最终结算价格行的 HTML 代码：

<tr>
   <th>Final Settlement Price</th>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td></td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
</tr>

您可以访问相同的网站：https ://www.sgx.com/derivatives/products/indiassf?cc=ZADS

标签： pythonseleniumweb-scrapingbeautifulsouphtml-table

解决方案

表中的数据是动态加载的，所以 BeautifulSoup 看不到它。但是你可以使用re//模块来获取数据json：requests

import json
import re
import requests


url = 'https://www.sgx.com/derivatives/products/indiassf?cc=ZADS'

app_config_url = 'https://www.sgx.com/config/appconfig.js'
n = re.search(r"apiVersion: '(.*?)'", requests.get(app_config_url).text).group(1)
variables = '''{{"path":"/derivatives/products/{}","lang":"EN"}}'''.format( url.split('?')[0].split('/')[-1] )
url2 = 'https://api2.sgx.com/content-api?queryId={}:page&variables={}'.format(n, variables)

data = requests.get(url2).json()

# uncomment this to prin all data:
#print(json.dumps(data, indent=4))

for c in data['data']['route']['data']['data']['contracts']:
    if c['data']['title'] == 'Ambuja Cements Futures':
        print(c['data']['tickerSymbol'])
        print(c['data']['finalSettlementPrice'])

印刷：

ZACE
The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.

编辑：站点更新后的新版本：

import json
import requests


url = 'https://www.sgx.com/derivatives/products/indiassf?cc=ZADS'

config = requests.get('https://www.sgx.com/config/appconfig.json').json()

variables = '''{{"path":"/derivatives/products/{}","lang":"EN"}}'''.format( url.split('?')[0].split('/')[-1] )
url2 = 'https://api2.sgx.com/content-api?queryId={}:page&variables={}'.format(config['cms']['apiVersion'], variables)

data = requests.get(url2).json()

# # uncomment this to prin all data:
# print(json.dumps(data, indent=4))

for c in data['data']['route']['data']['data']['contracts']:
    if c['data']['title'] == 'Ambuja Cements Futures':
        print(c['data']['tickerSymbol'])
        print(c['data']['finalSettlementPrice'])

推荐阅读