时间戳

首页 > 解决方案 > 如何在“for”循环中垂直添加多个列表？

python - 如何在“for”循环中垂直添加多个列表？

问题描述

我的“for”循环正在从多个页面中抓取（在这种情况下，我将三个页面放入列表中）。但是打印输出/csv 输出并没有通过循环获取先前的迭代（它只给我最后的第三页的结果）。我认为我在这里寻找的术语是“数组”，因为我希望每个页面的结果都垂直附加到彼此。我似乎误解了这个功能是如何工作的：

results.append(details)

这一切都归功于QHarr在这里找到的出色答案：如何将抓取的数据水平导出到Excel？

这是我正在使用的完整的工作代码：

import requests, re
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time

examplelist = [['1'], ['2'], ['3']]

pages = [i for I in examplelist for i in I]

for key in pages:
    driver = webdriver.Chrome(executable_path=r"C:\Users\User\Downloads\chromedriver_win32\chromedriver.exe")
    driver.get('https://www.restaurant.com/listing?&&st=KS&p=KS&p=PA&page=' + str(key) + '&&searchradius=50&loc=10021')
    time.sleep(10)
    WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".restaurants")))
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    restaurants = soup.select('.restaurants')
    results = []
    for restaurant in restaurants:
        details = [re.sub(r'\s{2,}|[,]', '', i) for i in restaurant.select_one('h3 + p').text.strip().split('\n') if i != '']
        details.insert(0, restaurant.select_one('h3 a').text)
        results.append(details)
#print(results)

df = pd.DataFrame(results, columns= ['Name', 'Address', 'City', 'State', 'Zip', 'Phone', 'AdditionalInfo'])
df.to_csv(r'C:\Users\User\Documents\Restaurants.csv', sep=',', encoding='utf-8-sig', index = False)

driver.close()

谢谢

标签： pythonweb-scrapingbeautifulsoup

解决方案

我认为你一直在循环内部清空results，results = []所以你失去了你已经放在那里的东西。像这样在循环外初始化

results=[]
for key in pages:
    driver = webdriver.Chrome(executable_path=r"C:\Users\User\Downloads\chromedriver_win32\chromedriver.exe")
    driver.get('https://www.restaurant.com/listing?&&st=KS&p=KS&p=PA&page=' + str(key) + '&&searchradius=50&loc=10021')
    time.sleep(10)
    WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".restaurants")))
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    restaurants = soup.select('.restaurants')
    for restaurant in restaurants:
        details = [re.sub(r'\s{2,}|[,]', '', i) for i in restaurant.select_one('h3 + p').text.strip().split('\n') if i != '']
        details.insert(0, restaurant.select_one('h3 a').text)
        results.append(details)
#print(results)

并从循环内部删除该初始化。

推荐阅读