python - 从具有许多页面的网站中抓取网页(Python)
问题描述
使用 BeautifulSoap 库,我想从例如 booking.com 中抓取信息。这是我的代码:
from bs4 import BeautifulSoup
from selenium import webdriver
import pandas as pd
url= 'https://www.booking.com/searchresults.en-gb.html?label=gen173nr-1FCAEoggJCAlhYSDNYBGgOiAEBmAEuwgEKd2luZG93cyAxMMgBDNgBAegBAfgBC5ICAXmoAgM&lang=en-gb&sid=1dcb4bc0b28bc6f79fb85ac861a6e6ce&sb=1&src=searchresults&src_elem=sb&error_url=https%3A%2F%2Fwww.booking.com%2Fsearchresults.en-gb.html%3Flabel%3Dgen173nr-1FCAEoggJCAlhYSDNYBGgOiAEBmAEuwgEKd2luZG93cyAxMMgBDNgBAegBAfgBC5ICAXmoAgM%3Bsid%3D1dcb4bc0b28bc6f79fb85ac861a6e6ce%3Bclass_interval%3D1%3Bdest_id%3D-390625%3Bdest_type%3Dcity%3Bfrom_sf%3D1%3Bgroup_adults%3D2%3Bgroup_children%3D0%3Blabel_click%3Dundef%3Bno_rooms%3D1%3Boffset%3D0%3Braw_dest_type%3Dcity%3Broom1%3DA%252CA%3Bsb_price_type%3Dtotal%3Bslp_r_match%3D0%3Bsrc%3Dindex%3Bsrc_elem%3Dsb%3Bsrpvid%3D9fe26fd0b0e90202%3Bss%3DMadrid%3Bssb%3Dempty%3Bssne%3DMadrid%3Bssne_untouched%3DMadrid%26%3B&ss=Madrid&ssne=Madrid&ssne_untouched=Madrid&city=-390625&checkin_monthday=15&checkin_month=10&checkin_year=2018&checkout_monthday=19&checkout_month=10&checkout_year=2018&group_adults=2&group_children=0&no_rooms=1&from_sf=1'
driver = webdriver.Chrome(r"C:\Users\yefida\Desktop\Study_folder\Online_Courses\python mega course\Project 6 - Web Scraping\chromedriver.exe")
driver.get(url)
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
list_data = []
data = soup.find_all('div',{'class':'sr_item_content sr_item_content_slider_wrapper '})
for item in data:
temp = {}
temp['Title'] = item.find('span',{'class':'sr-hotel__name'}).text.replace('\n','')
temp['Address'] = item.find('div',{'class':'address'}).text.replace('\n','').lstrip(' ').partition(',')[0]
try:
temp['Demand'] = item.find('span',{'class':'sr_property_in_demand'}).text.replace('!','')
except:
temp['Demand'] = None
temp['Raiting Users Words'] = item.find('span',{'class':'review-score-widget__text'}).text.replace('\n','')
try:
temp['Number Reviews'] = item.find('span',{'class':'review-score-widget__subtext'}).text.replace('\n','').replace(' reviews','')
except:
temp['Number Reviews'] = None
temp['Raiting Users Numbers'] = item.find('span',{'class':'review-score-badge'}).text.replace('\n','')
try:
temp['Location'] = item.find('div',{'class':'search-secondary-review-score'}).text.replace('\n','').replace('Location','')
except:
temp['Location'] = None
try:
temp['Price'] = item.find('strong',{'class':'price'}).text.replace('\n','').replace('€\xa0','')
except:
temp['Price'] = None
list_data.append(temp)
基本上,我从booking com的第一页得到了信息。但这里是第二页的链接:
"https://www.booking.com/searchresults.en-gb.html?aid=304142&label=gen173nr-1FCAEoggJCAlhYSDNYBGgOiAEBmAEuwgEKd2luZG93cyAxMMgBDNgBAegBAfgBC5ICAXmoAgM&sid=b7189b7c140bf45ceae5bfd81d08947f&checkin_month=10&checkin_monthday=15&checkin_year=2018&checkout_month=10&checkout_monthday=19&checkout_year=2018&city=-390625&class_interval=1&dest_id=-390625&dest_type=city&from_sf=1&group_adults=2&group_children=0&label_click=undef&no_rooms=1&raw_dest_type=city&room1=A%2CA&sb_price_type=total&slp_r_match=0&src=searchresults&src_elem=sb&srpvid=49147618d7ba03be&ss=Madrid&ssb=empty&ssne=Madrid&ssne_untouched=Madrid&rows=50&offset=50"
如您所见,第一页的链接和第二页的链接之间的差异很大(url 字符串的差异)。那么我怎样才能对这个问题的数据进行网络抓取呢?有什么技巧吗?
解决方案
您所要做的就是&offset=100
在 url 查询中添加示例..
或者,您可以使用 bs4 获取下一页链接。像这样的东西:
next_page = soup.find('a',{'class':'sr-pager__link js-pagination-next-link'}).get('href')
推荐阅读
- reactjs - 当状态是数据库表时,如何从子级更改父级的状态?
- python - Django 用户表单未显示在管理页面中
- javascript - React Datepicker 包括没有周末的未来几天
- r - 如何编写R函数
- sitefinity - sitefinity ActionResult 缺少查询字符串
- xml - XSLT:如何调用以字符串形式给出的用户定义函数
- python - 如何在 Jupyter 笔记本中使用 Pandas 永久更改数据框样式?
- python - 具有多个键的 Python 字典列表
- sql - 在 SQL Server 中选择以显示表 OnHand、Sale 和 Purchase 中的所有不同行,其中一个/或 Qty 字段不为空
- c++ - ffmpeg/Libavcodec 找不到编解码器