时间戳

首页 > 解决方案 > 将数据迭代到 Pandas 数据框（python）

python - 将数据迭代到 Pandas 数据框（python）

问题描述

我是 python 新手，我正在尝试从网站收集数据。我现在的问题是我卡在最后一个流程中，我想要获取数据并将其迭代到 Pandas 数据框，然后将其保存到数据库/csv 文件中。

我尝试使用循环附加数据，但似乎我的循环不起作用。如果您可以查看我是否查看“cols”，我设法清理了数据，但它没有进入表格。

import requests, pandas, numpy, matplotlib.pyplot
from bs4 import BeautifulSoup
#### page info ###
page = requests.get("https://postcode.my/search/?keyword=&state=Kedah")
#### check page status (will come 200 if the page is ok) 
page.status_code
### call Library
soup = BeautifulSoup(page.content, 'html.parser')
### Find rows 
rows = soup.find_all(class_="col-lg-12 col-md-12 col-sm-12 col-xs-12")
## define column
LOCATION = []
AREA = []
STATE = []
POSTCODE = []
TABLE = []
counter= 0 
for row in rows:
    cols = row.find_all("td")
    cols = [x.text.strip() for x in cols]
if cols!='':
    TABLE.append(cols)
    counter=counter+1
if counter == 4:
    LOCATION.append(TABLES[0])
    AREA.append(TABLE[1])
    STATE.append(TABLE[2])
    POSTCODE.append(TABLE[3])
    counter = (0)
    TABLE = []
PDTABLE = pandas.DataFrame({
    "LOCATION" : LOCATION,
    "AREA" : AREA,
    "STATE" : STATE,
    "POSTCODE" : POSTCODE
    })

PDTABLE

谢谢你最好的问候Railey Shahril

标签： pythonpandasweb-scraping

解决方案

利用：

import requests, pandas, numpy, matplotlib.pyplot, numpy
from bs4 import BeautifulSoup
#### page info ###
page = requests.get("https://postcode.my/search/?keyword=&state=Kedah")
#### check page status (will come 200 if the page is ok) 
page.status_code
### call Library
soup = BeautifulSoup(page.content, 'html.parser')
### Find rows 
rows = soup.find_all(class_="col-lg-12 col-md-12 col-sm-12 col-xs-12")

通过追加创建列表：

L = []
for row in rows:
    cols = row.find_all("td")
    cols = [x.text.strip() for x in cols]
    L.append(cols)

转换为 numpy 数组并重塑为 4 列：

cols = ['LOCATION','AREA','STATE','POSTCODE']
PDTABLE = pandas.DataFrame(numpy.array(L).reshape(-1, 4), columns=cols)
print (PDTABLE)
                                 LOCATION             AREA  STATE POSTCODE
0                         Akauntan Negeri       Alor Setar  Kedah    05594
1                            Alor Gelegah       Alor Setar  Kedah    05400
2                     Alor Ibus Tepi Laut      Kuala Kedah  Kedah    06600
3                            Alor Janggus       Alor Setar  Kedah    06250
4                              Alor Malai       Alor Setar  Kedah    05460
5                     Alor Melintang Anak       Alor Setar  Kedah    05150
6                   Alor Melintang Gunung       Alor Setar  Kedah    05150
7                              Alor Merah       Alor Setar  Kedah    05250
8                             Alor Nibong  Kota Kuala Muda  Kedah    08500
9                              Alor Selut       Alor Setar  Kedah    05400
10              Alor Setar - Beg berkunci       Alor Setar  Kedah    05990
11         Alor Setar - Peti surat 1 - 80       Alor Setar  Kedah    05700
12  Alor Setar - Peti surat 161 & ke atas       Alor Setar  Kedah    05720
13       Alor Setar - Peti surat 81 - 160       Alor Setar  Kedah    05710
14                     Amanah Raya Berhad       Alor Setar  Kedah    05508
15                        Ambangan Height    Sungai Petani  Kedah    08000
16                          Ampangan Pedu     Kuala Nerang  Kedah    06300
17                             Anak Bukit       Alor Setar  Kedah    06550
18                       Anjung Pedu Lake     Kuala Nerang  Kedah    06300
19                                   Ason            Jitra  Kedah    06000

推荐阅读