时间戳

首页 > 解决方案 > 如何从不同的维基百科页面抓取数据？

python - 如何从不同的维基百科页面抓取数据？

问题描述

我已经使用 Python Beautifulsoup ( https://en.wikipedia.org/wiki/Districts_of_Hong_Kong )抓取了维基百科表。但除了提供的数据（即人口、面积、密度和地区），我想获得每个地区的位置坐标。数据应该来自每个区的另一个页面（表上有超链接）。

以第一区“中西区”为例，页面上可以找到DMS坐标（22°17′12″N 114°09′18″E）。通过进一步单击链接，我可以获得十进制坐标（22.28666、114.15497）。

那么，是否可以为每个地区创建一个经纬度表？

编程世界的新手，如果问题很愚蠢，对不起......

参考：

DMS 坐标：https ://en.wikipedia.org/wiki/Central_and_Western_District

十进制坐标：https ://tools.wmflabs.org/geohack/geohack.php?pagename=Central_and_Western_District¶ms=22.28666_N_114.15497_E_type:adm2nd_region:HK

标签： pythonweb-scrapingwikipedia

解决方案

import requests
from bs4 import BeautifulSoup

res = requests.get('https://en.wikipedia.org/wiki/Districts_of_Hong_Kong')
result = {}
soup = BeautifulSoup(res.content,'lxml')
tables = soup.find_all('table',{'class':'wikitable'})
table = tables[0].find('tbody')
districtLinks = table.find_all('a',href=True)

for link in districtLinks:
    if link.getText() in link.attrs.get('title','') or link.attrs.get('title','') in link.getText():
        district = link.attrs.get('title','')
        if district:
            url = link.attrs.get('href', '')
        else:
            continue
    else:
        continue
    try:
        res = requests.get("https://en.wikipedia.org/{}".format(url))
    except:
        continue
    else:
        soup = BeautifulSoup(res.content, 'lxml')
        try:
            tables = soup.find_all('table',{'class':'infobox geography vcard'})
            table = tables[0].find('tbody')
        except:
            continue
        else:
            for row in table.find_all('tr',{'class':'mergedbottomrow'}):
                geoLink = row.find('span',{'class': 'geo'}) # 'plainlinks nourlexpansion'
                locationSplit = geoLink.getText().split("; ")
                result.update({district : {"Latitude ": locationSplit[0], "Longitude":locationSplit[1]}})

print(result)

结果：

{'Central and Western District': {'Latitude ': '22.28666', 'Longitude': '114.15497'}, 'Eastern District, Hong Kong': {'Latitude ': '22.28411', 'Longitude': '114.22414'}, 'Southern District, Hong Kong': {'Latitude ': '22.24725', 'Longitude': '114.15884'}, 'Wan Chai District': {'Latitude ': '22.27968', 'Longitude': '114.17168'}, 'Sham Shui Po District': {'Latitude ': '22.33074', 'Longitude': '114.16220'}, 'Kowloon City District': {'Latitude ': '22.32820', 'Longitude': '114.19155'}, 'Kwun Tong District': {'Latitude ': '22.31326', 'Longitude': '114.22581'}, 'Wong Tai Sin District': {'Latitude ': '22.33353', 'Longitude': '114.19686'}, 'Yau Tsim Mong District': {'Latitude ': '22.32138', 'Longitude': '114.17260'}, 'Islands District, Hong Kong': {'Latitude ': '22.26114', 'Longitude': '113.94608'}, 'Kwai Tsing District': {'Latitude ': '22.35488', 'Longitude': '114.08401'}, 'North District, Hong Kong': {'Latitude ': '22.49471', 'Longitude': '114.13812'}, 'Sai Kung District': {'Latitude ': '22.38143', 'Longitude': '114.27052'}, 'Sha Tin District': {'Latitude ': '22.38715', 'Longitude': '114.19534'}, 'Tai Po District': {'Latitude ': '22.45085', 'Longitude': '114.16422'}, 'Tsuen Wan District': {'Latitude ': '22.36281', 'Longitude': '114.12907'}, 'Tuen Mun District': {'Latitude ': '22.39163', 'Longitude': '113.9770885'}, 'Yuen Long District': {'Latitude ': '22.44559', 'Longitude': '114.02218'}}

推荐阅读