时间戳

首页 > 解决方案 > Beautifulsoup 没有返回网页上的所有文本

python - Beautifulsoup 没有返回网页上的所有文本

问题描述

尝试对网站进行网页抓取，但 Beautifulsoup 仅在查看网页时不会返回所有可见的文本。请看下面的代码：

import requests
from bs4 import BeautifulSoup

f = open("data.txt", "w")
url = "https://www.hiltongrandvacations.com/en/resorts-and-destinations"

response = requests.get(url)
soup = BeautifulSoup(response.text, 'html5lib')
f.write(str(soup))
f.close()

例如，以下文本在网页上可见，但 Beautifulsoup 未返回（写入文本文件）： Grand Pacific Palisades Resort

我尝试了不同的解析器（html、lxml），但仍然没有得到它。另外，似乎文本不是由 Javascript 生成的，我可能错了。

标签： pythonweb-scrapingbeautifulsoup

解决方案

您看到的数据是通过 JavaScript 动态加载的。您可以使用此示例加载数据：

import json
import requests


payload = {"locations":[],"amenities":[],"vacationTypes":[],"page":1,"pageSize":9}
api_url = 'https://www.hiltongrandvacations.com/sitecore/api/ssc/apps/PropertySearch'

data = requests.put(api_url, json=payload).json()

# uncomment this to prin all data:
# print(json.dumps(data, indent=4))

# print some info on screen:
for card in data['Cards']:
    print(card['Title'])
    print(card['Description'])
    print('-' * 80)

印刷：

Sunrise Lodge, a Hilton Grand Vacations Club
Revel in the peak of adventure
--------------------------------------------------------------------------------
The District by Hilton Club
A capital experience in the capital city
--------------------------------------------------------------------------------
The Central at 5th by Hilton Club
At the heart of city life
--------------------------------------------------------------------------------
The Hilton Club – New York
Make a break for the Big Apple.
--------------------------------------------------------------------------------
The Residences by Hilton Club
Wake up in the city that never sleeps.
--------------------------------------------------------------------------------
Grand Pacific Palisades Vacation Resort
A window to the Pacific Ocean. 
--------------------------------------------------------------------------------
Carlsbad Seapointe Resort
A quintessentially Californian vacation
--------------------------------------------------------------------------------
Hilton Grand Vacations Chicago Downtown/Magnificent Mile
A sky-high sanctuary amidst the big-city bustle
--------------------------------------------------------------------------------
Hilton Grand Vacations Club at Trump International Hotel Las Vegas

--------------------------------------------------------------------------------

推荐阅读