python-3.x - 将来自网络抓取的列表输出列表转换为数据框

Question

我创建了以下代码来从网站上抓取地址，这似乎工作正常。然而，输出是一个列表列表，我无法将其转换为数据框。

我尝试使用 pd.DataFrame(addresses) 但这不会产生预期的输出。我也试过 pd.DataFrame(list(zip(addresses))) 但也没有得到预期的输出。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException 
from bs4 import BeautifulSoup
import time
import pandas as pd
import re
base_url = 'https://www.thechristhospital.com/locations-search-results?Type=AdvancedSearch'
browser = webdriver.Chrome()
browser.get(base_url)
soup = BeautifulSoup(browser.page_source,'html.parser')

addresses = []
time.sleep(5)
button = browser.find_element_by_css_selector('#ctl00_ctl35_g_5f6e70e2_119c_48b6_a627_dbce7ca77728_cntrlPaging_btnPageFwd')
time.sleep(2)
count = 0
while True:
    try:

        WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ctl00_ctl35_g_5f6e70e2_119c_48b6_a627_dbce7ca77728_cntrlPaging_btnPageFwd"))).click()
        count += 1
        time.sleep(2)
        soup = BeautifulSoup( browser.page_source,'html.parser')
        add= [add.text.strip() for add in soup.find_all('div',{'class':'address'})]
        addresses.append(add)
        time.sleep(2)
    except TimeoutException:
            break
for add in add:

browser.quit()

我期望的输出是一个列出每个位置地址的数据框。在不同的字段中按名称/地址拆分是理想的，但如果它在一个字段中也可以。

任何帮助表示赞赏。

标签： python-3.xweb-scraping

Answer 1

下面是一种有点不同的方法——有点笨拙而且可能很脆弱，但它可以完成工作，你应该可以轻松地对其进行修改，让它做你想做的事情。

我只在第一页上进行了尝试，因此您也必须对其进行修改才能捕获其他页面。

data = pd.read_html(base_url)
info = data[0].iloc[:,0] #this is where the relevant info is located

#remove irrelevant parts and split into lists
places = []
for place in info:
    place_list = place.replace('Get Directions ','').replace('Hours','').replace('Providers  ','').replace('Services','NA').split('  ')[:-1]
    if len(place_list)== 6: #some entries don't have a second address line, some do
        place_list.insert(3,'NA')
    places.append(place_list)


#create the dataframe
columns = ['Hospital','Division','Street Address','Address 2','Address 3','Phone','Providers']

new_df = pd.DataFrame(places, columns=columns) 
new_df.head(3)

输出：

                   Hospital                      Division              Street Address   Address 2   Address 3             Phone     Providers
0   The Christ Hospital Interventional Radiology    The Christ Hospital     2139 Auburn Ave.    Level C - Interventional Radiology  Cincinnati, OH 45219    [phone]  Charity N. DeArmond, CNPVickie M. Dietrich, CNP
1   The Christ Hospital Inpatient Orthopedics   The Christ Hospital     2139 Auburn Ave.    NA  Cincinnati, OH 45219    [phone]  Stephanie L. Ellis, CNP
2   The Christ Hospital Inpatient Transplant    The Christ Hospital     2139 Auburn Ave.    NA  Cincinnati, OH 45219    [phone]  Rebecca K. Parks, CNP