python - 使用 beautifulsoup 抓取房地产经纪人数据

Question

我试图通过使用 beautifulsoup 从 realtor.com 上抓取一些数据来帮助一些房地产经纪人朋友。

我正在尝试获取房地产经纪人的姓名和电话号码列表，但我将每个项目作为单独的项目获取，并且页面上的每个房地产经纪人都有重复项。

这是我目前拥有的：

from bs4 import BeautifulSoup
import requests
import numpy as np
import pandas as pd

allRealtors = []
pages = np.arange(1, 2, 1)
for page in pages:
    page = requests.get("https://www.realtor.com/realestateagents/New-Orleans_LA/pg-" + str(page))
    soup = BeautifulSoup(page.text, 'html.parser')
    realtors = soup.find_all('div', {"class", ['jsx-1448471805 agent-name text-bold', 'jsx-1448471805 agent-phone hidden-xs hidden-xxs']})
    for item in realtors:
        allRealtors += item
print(allRealtors)

以下是我对 allRealtors 列表的当前结果：

['Lisa Shedlock', '[phone]', 'Lisa Shedlock', '[phone]', 'Heather Laughlin', '[phone]', 'Heather Laughlin', '[phone]', 'LIZ ASHE', '[phone]', 'LIZ ASHE', '[phone]', 'Richard Haffner', '[phone]', 'Richard Haffner', '[phone]', 'Shelly Vallee', '[phone]', 'Shelly Vallee', '[phone]', 'Britt Galloway, Agent', '[phone]', 'Britt Galloway, Agent', '[phone]', 'Catherine Goens Gerrets, Agent', '[phone]', 'Catherine Goens Gerrets, Agent', '[phone]', 'Suzy Lamore', '[phone]', 'Suzy Lamore', '[phone]', 'Patti Faulder', '[phone]', 'Patti Faulder', '[phone]']

它正在为每个房地产经纪人的姓名和电话号码创建副本。理想情况下，我会将 2 个值作为字典输入，如下所示：

{name:'Lisa Shedlock', number:'[phone]'; name:'Heather Laughlin', number:'[phone]'}

然后我会将该字典转换为带有列名称和电话号码的 pandas 数据框。

然而，这是我第一次使用beautifulsoup，不知道如何做到这一点。有什么建议么？

有没有更简单的方法来实现这一点？

谢谢！

标签： pythonbeautifulsoup

Answer 1

那么你可以以这种方式使用选择器

from bs4 import BeautifulSoup
import requests
import numpy as np
import pandas as pd

realtors_data = {}
pages = np.arange(1, 2, 1)
print("PAGES: ", pages)
names_selector = "ul > div > div > div > div > div > a > div"
phone_selectors = "ul > div > div > div > div > div > div.jsx-1448471805.agent-phone.hidden-xs.hidden-xxs"
for page in pages:
    page = requests.get("https://www.realtor.com/realestateagents/New-Orleans_LA/pg-" + str(page))
    soup = BeautifulSoup(page.text, 'html.parser')
    names = soup.select(names_selector)
    phones = soup.select(phone_selectors)

    realtors = zip(names, phones)
    for name, phone in realtors:
        realtors_data[name.get_text()] = phone.get_text()


# Printing data
print(realtors_data)

输出：

{'Lisa Shedlock': '[phone]', 'Heather Laughlin': '[phone]', 'LIZ ASHE': '[phone]', 'Richard Haffner': '[phone]', 'Shelly Vallee': '[phone]', 'Britt Galloway, Agent': '[phone]', 'Catherine Goens Gerrets, Agent': '[phone]', 'Suzy Lamore': '[phone]', 'Patti Faulder': '[phone]', 'Susan Ann Bourgeois': '[phone]', 'Lane Washburn': '[phone]', 'Brandy Dufrene': '[phone]', 'Claire E Hohensee': '[phone]', 'Aaron DareTeam': '[phone]', 'Kara Breithaupt': '[phone]', 'Joli Tolbert-Burrell': '[phone]', 'AMANDA MILLER': '[phone]', 'Carla Lawson': '[phone]', 'Michael D. Lester': '[phone]', 'Michael A. Newcomer': '[phone]'}