时间戳

首页 > 解决方案 > 抓取支持 JS 的站点以实现自动化

python - 抓取支持 JS 的站点以实现自动化

问题描述

我正在尝试抓取一个特定站点 ( https://chartink.com/screener/test-121377 ) 我正在尝试访问 URL 中提供的表格。该表可以是动态形式的。该站点启用了 JS，因此尝试 HTML 抓取是多余的。接下来，我尝试使用 cURL 访问 XHR 元素，它确实有效。我使用了以下代码：

import requests
headers = {
'authority': 'chartink.com',
'pragma': 'no-cache',
'cache-control': 'no-cache',
'accept': 'application/json, text/javascript, */*; q=0.01',
'x-csrf-token': 'HYMZ6XnOySvn1w4fHEN0Hu1ublftvidgdBD5qywR',
'x-requested-with': 'XMLHttpRequest',
'user-agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36',
'sentry-trace': 'd9b8f7ffcae445f7a74913d5ac143918-9fa7556ebb76186c-0',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'origin': 'https://chartink.com',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
'referer': 'https://chartink.com/screener/test-121377',
'accept-language': 'en-IN,en;q=0.9,hi-IN;q=0.8,hi;q=0.7,en-GB;q=0.6,en-US;q=0.5',
'cookie': '_ga=GA1.2.468738165.1600680012; remember_web_59ba36addc2b2f9401580f014c7f58ea4e30989d=eyJpdiI6Inp6YWdERUJHSGxoWmhRaE00MGhONHc9PSIsInZhbHVlIjoiSXN1d09PXC9USmpGb2lFR21naFd1Q3RlcCtKSDdUXC8zdGRnMkRoQmNcL2loXC9tT0tMXC9sc0JtallJVCtNczVhVjhIdGUyb0xqMTIydlc0RWtQYVZcLzF1aFd3aXRPM0M3MDVObUtNK1FtdmJtZjY1YUxNbTlyTHhNWFhjU3VZNXpUTWY0SjZ4M1o1VEcydk1DWDY5YWVSdWRjbVwvQmkyYmlDT2VMWFVCM043Y2JIY21UTjNsdnJLaE1qSnNybmM2WVhISyIsIm1hYyI6IjhlNjBiZWUxMWU3NTBkOTRjM2NlYTZiNGI3YzJhYjg4M2E0OWI2MDU3ZDZkYWUzNzE1YjYxOTRhMTllMjY2ZWQifQ%3D%3D; __utmc=102564947; PHPSESSID=e3797d9e58f9f56b0307e7840f6839f7; bfp_sn_rf_8b2087b102c9e3e5ffed1c1478ed8b78=Direct/External; bfp_sn_rt_8b2087b102c9e3e5ffed1c1478ed8b78=1601299430046; bafp=d8afd2b0-018d-11eb-b966-a1985446bf25; hbcm_sd=1%7C1602315380999; _gid=GA1.2.729526333.1602491131; __utma=102564947.468738165.1600680012.1601134509.1602499714.2; __utmz=102564947.1602499714.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); XSRF-TOKEN=eyJpdiI6InVXSEFpN2xiYWZqWDkzNEdWZVBTSkE9PSIsInZhbHVlIjoiQlM2eVBYMjdYR3N4QTh6YmtFWmNoTUtOVHBIOFVoUG04WTdcL05iUm4yZExQR3ZucHlPVHhucVwvbjlHTW9VcmZCIiwibWFjIjoiMWEyMzViZDQ5YWI1YzIzY2UzYTdkNzRlN2UyZmYzNmY1ZWNkYTBmZmJmOWI2YjZmOGRmN2U2Y2IzOGE5ZmQwNiJ9; ci_session=eyJpdiI6ImtcLzNBOGxOb1l2bFlyOEFnQ2dxVHNBPT0iLCJ2YWx1ZSI6IjRpNVhXOWtGZlV0TWdwZlhIVGRmWFgrcGFyblwveGlkdVU5aHJraVdvbFROOURUa2Zhc2pcLzJNOE9qa0FwcjlwbSIsIm1hYyI6IjVlYTUwNDgxN2U0NTU4OWE0NTgxOTMwNjNjMzdhM2Q5MjRiMjJjMTBkYzI0ZTdkOTg1YTg5YmIxNTM3NzFlMDQifQ%3D%3D',}
data = {'scan_clause': '( {33489} ( latest close > 10 ) ) '}
response = requests.post('https://chartink.com/screener/process', headers=headers, data=data)
data = json.loads(response.text)
df = pd.json_normalize(data['data'])

有效。但是，它的某些元素将在一定时间后过期。CSRF 令牌、哨兵跟踪和 cookie。

一段时间后，由于令牌不匹配，代码在运行时将开始出现错误。

然后我使用了下面给出的方法（在另一个线程之后），但它也不起作用。

import requests_html
session = requests_html.HTMLSession()
r = session.get('https://chartink.com/screener/test-121377')
r.html.render(sleep=5)
items = r.html.find("table#DataTables_Table_0",first=True)
for item in items.find("tr"):
data = [td.text for td in item.find("th,td")]
print(data)

但我再次收到错误：

RuntimeError: Cannot use HTMLSession within an existing event loop. Use AsyncHTMLSession instead.

当我使用 AsyncHTMLSession 时，还有其他一些错误。总而言之，这对我来说是一个死胡同。

可以在任何一种方法中添加解释。

标签： pythonajaxweb-scrapingbeautifulsoup

解决方案

您可以一次获得所有 137 个结果。尝试使用基本 url 来收集 csrf 令牌，然后发送 http post 请求以及适当的参数以获取 json 响应，以便从中获取所需的内容。您可以这样做：

import requests
from bs4 import BeautifulSoup

link = "https://chartink.com/screener/test-121377"
url = 'https://chartink.com/screener/process'

payload = {
    'scan_clause': '( {33489} ( latest close > 10 ) )'
}

with requests.Session() as s:
    r = s.get(link)
    soup = BeautifulSoup(r.text,"html.parser")
    csrf = soup.select_one("[name='csrf-token']")['content']
    s.headers['x-csrf-token'] = csrf
    r = s.post(url,data=payload)
    for item in r.json()['data']:
        print(item['name'],item['nsecode'],item['per_chg'],item['close'],item['volume'])

结果如下：

Mrf Limited MRF -0.34 58587.1 9057
BANKNIFTY BANKNIFTY -0.56 23712.8 0
Shree Cements Limited SHREECEM -1.14 20956.1 56254
Page Industries Limited PAGEIND -0.29 20898.75 26311
Nestle India Limited NESTLEIND -0.82 15902.2 58171
Bosch Limited BOSCHLTD -1.64 12529.1 37951

推荐阅读