python - 使用 python 抓取网页期间 HTML 类的问题

Question

我是 python 新手，我正在尝试从购物网站上抓取有关计算机的信息。尽管代码看起来很好（没有错误），但在过程的最后它不会给我 csv 文件。我已经改变了课程，但没有任何效果。我在另一个网站上使用了这段代码，它运行良好。另一个小问题：如何抓取多页结果。提前致谢！

from bs4 import BeautifulSoup
import pandas as pd
from selenium import webdriver
import pandas as pd

products=[] #List to store name of the product
prices=[] #List to store price of the product
fretes=[] #List to store rating of the product
driver = webdriver.Chrome(executable_path = r'C:\Users\gomes\Desktop\chromedriver.exe')
driver.get("https://lista.mercadolivre.com.br/macbook")
content = driver.page_source
soup = BeautifulSoup(content, 'html.parser')
for a in soup.findAll('li',href=True, attrs={'class':'results-item'}):
    name=a.find('span', attrs={'class':'main-title'})
    price=a.find('span', attrs={'class':'price__fraction'})
    frete=a.find('span', attrs={'class':'text-shipping'})
    products.append(name.text)
    prices.append(price.text)
    fretes.append(frete.text)
    data = dict({'Product Name': products,
                 'Price': prices,
                 'Frete': fretes
                 })
    # create dataframe
    products_df = pd.DataFrame(
        dict([(k, pd.Series(v)) for k, v in data.items()])
        )
    products_df.to_csv("C:\\Users\\gomes\\Desktop\\mac.csv")

标签： pythonhtmlpandasseleniumweb-scraping

Answer 1

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = 'https://lista.mercadolivre.com.br/macbook'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

data = []
for title, price in zip(soup.select('.item__title'), soup.select('.price__fraction')):
    frete = title.find_previous_sibling(attrs={'class':'item__shipping'})
    data.append({'Product Name': title.get_text(strip=True),
                 'Price': price.get_text(strip=True),
                 'Frete': frete.get_text(strip=True) if frete else '-'})

df = pd.DataFrame(data)
print(df)

df.to_csv('data.csv')

印刷：

                                         Product Name   Price         Frete
0   Macbook Pro 15 Core I7 8gb De Ram 2011 - Leia ...   2.699             -
1   Macbook Pro I5 2.7ghz/8gb 1867mhz/ssd 256gb Mo...   5.100  Frete grátis
2   Apple Macbook Pro Retina 13,3 I5 2.3/8gb/256gb...   9.830  Frete grátis
3   Macbook Pro 2015 A1502 Retina I5 8gb 256ssd + ...   4.799             -
4   Macbook A1502 Pro 2015 Retina I5 8gb 256ssd + ...   4.800             -
5   Apple Macbook A1534 2017 Mnyk2bza Core M3 8gb ...   8.499  Frete grátis
6   Macbook Pro (ver Detalhes Abaixo No Descritivo...   4.460             -
7   Macbook Pro 2015 I5 8gb 2,7ghz 256ssd Impecáve...   5.993  Frete grátis
8   Macbook Pro I5 2.7ghz/8gb 1867mhz/ssd 256gb Mo...   5.100  Frete grátis
9   Macbook Pro 15 Mid 2009 2,66 Ghz Intel Core Du...   2.749  Frete grátis
10  Macbook Pro Touch Bar 2016 I5 8gb 256ssd - Sem...   8.999  Frete grátis
11  Macbook Pro 2017 - Retina - 2,3 Ghz - I5 - 8gb...   9.999  Frete grátis
12  Macbook Pro 2019 16pol 2.4 I9 32gb 1tb 5500m E...  30.699  Frete grátis
13  Apple Macbook Pro A1278 Core I5 8gb Hd Ssd 240...   4.480  Frete grátis
14  Macbook Pro Md318ll/a Intel Core I7 2.2ghz 8gb...   4.490             -
15             Macbook Air 13 128gb (versao Anterior)   7.499  Frete grátis
16  Macbook Pro 2019 16pol 2.4 I9 64gb 1tb 5500m E...  33.299  Frete grátis
17  Macbook Pro 2015 Retina I5 8gb 256ssd + Carreg...   5.494  Frete grátis
18  Macbook Pro I5 2,3 Ghz 8gb Ram Tela 13,3 500gb...   3.790  Frete grátis
19  Apple Macbook Pro (modelo Completo Na Descriçã...   4.490  Frete grátis
20  Promoção: Macbook Pro 2015 Retina I5 8gb 256ss...   5.493  Frete grátis
21  Macbook Pro 2015 I5 8gb 2,7ghz 256ssd Apple Se...   6.173  Frete grátis
22  Macbook Pro 2015 Retina I5 8gb 256ssd C/ Carre...   5.493  Frete grátis
23       Macbook Pro 2019 16pol 2.4 I9 32gb 1tb 5550m  30.399  Frete grátis
24  Macbook Pro 2019 16 Pol 2.3 I9 16gb 1tb 5500m ...  25.999  Frete grátis
25  Promoção: Macbook A1534 2015 8gb Ssd 256gb + C...   6.999  Frete grátis
26  Macbook Pro Apple 2015 Retina I5 8gb 256ssd - ...   6.174  Frete grátis
27  Macbook Pro A1398 I7 2014 15 Pol. 16gb 256ssd ...   7.680  Frete grátis
28  Macbook Pro 2016 A1706 Touch Bar I5 8gb 256ssd...   8.759  Frete grátis
29  Macbook Pro 2019 16pol 2.4 I9 64gb 1tb 5550x E...  29.499  Frete grátis
30  Macbook Pro 13 - 2.3ghz 8gb 256ssd Cinza Espacial   9.500  Frete grátis
31  Macbook Pro 2019 16pol 2.3 I9 16gb 1tb 5550x E...  22.999  Frete grátis
32  Macbook A1466/2012 13.3 Polegadas 1.8ghz 8gb 1...   4.000             -
33  Oferta! Macbook Pro 2015 I5 8gb 2,7ghz 256 C/ ...   6.173  Frete grátis
34  Apple Macbook Pro 2015 I5 8gb 2,7ghz 256ssd - ...   6.173  Frete grátis
35  Apple Macbook Pro 2015 I5 8gb 2,7ghz 256ssd - ...   6.305  Frete grátis
36  Macbook Pro Touchbar 13 2019 2.4 8gb 512gb 119...  15.999  Frete grátis
37     Macbook Pro 2019 13 I7 2.8 16gb 512gb Envio Ja  19.899  Frete grátis
38  Macbook Pro A1278 Tela 13 Core I5 8gb Ssd 240 ...   4.280             -
39             Macbook Air 13 256gb (versao Anterior)  10.211  Frete grátis
40  Macbook Pro A1398 2014 I7 2,5ghz 16gb 256gb - ...   7.680  Frete grátis
41  Macbook Pro 15 - Intel Core I7 - Ssd 500gb - A...   8.900  Frete grátis
42   Macbook Pro Touchbar 13 2019 2.4 8gb 512gb 11999  17.849  Frete grátis
43  Macbook Pro 2019 16 Pol I7 6 Core 2.6 16gb 512...  20.990  Frete grátis
44  Macbook Pro 15, 16gb, I7, 13 Ciclos, Muito Nov...   7.500  Frete grátis
45     Promoção Macbook Pro 2015 I5 8gb 2,7ghz 256ssd   6.173  Frete grátis

并保存data.csv（来自 LibreOffice 的屏幕截图）：