时间戳

首页 > 解决方案 > 机场数据 - 我如何抓取数据并维护层次结构？

python - 机场数据 - 我如何抓取数据并维护层次结构？

问题描述

我正在尝试从https://skytraxratings.com/airports/hamad-international-airport-rating抓取信息

我想制作一个 csv/excel，其中包含以下列 - 类别， - 子类别 - 评级

category subcategory rating 地面交通 公共交通选择 3.5 地面交通 到公共交通的距离 4 地面交通 机场出租车服务 4.5 地面交通租车 4.5 安检 安检引导标牌 4 安检 等候时间 4 安检 服务效率 4 安检 员工态度和礼貌 4安全检查 员工语言技能 4 安全检查 FastTrack 安全 5

但我将每一列作为一个列表，没有层次结构完整性

category subCategory starRating 地面交通、安检、移民服务、寻路和指示牌、到达、出发、转机、航站楼舒适度、航站楼设施、购物设施、食品和饮料

公共交通选择，到公共交通的距离，机场出租车服务，汽车租赁，筛选指南标牌，等待时间，服务效率，工作人员态度和礼貌，工作人员语言技能，FastTrack安全，

,,

尝试过在scrapy上搜索和阅读文档，但我迷路了

import scrapy


class HiaskytraxratingsSpider(scrapy.Spider):
    name = 'HIASkytraxRatings'
    allowed_domains = ['skytraxratings.com']
    start_urls = ['https://skytraxratings.com/airports/hamad-international-airport-rating']

    def parse(self, response):

        print("processing:"+response.url)
        #Extract using css selectors
        #None

        #Extract using xpath
        category = response.xpath("//div[@class='ratingstable']/h4[@class='ratingstable__title fontsize-1']/text()").extract()
        subCategory = response.xpath("//div[@class='ratingstable']/div[@class='ratingstable__ratings']/div[@class='rating']/span[@class='rating__title']/text()").extract() 
        starRating = response.xpath("//div[@class='ratingstable']/div[@class='ratingstable__ratings']/div[@class='rating']/span[@class='rating__stars']").extract()



        #yield or give the scraped info to scrappy
        yield {'page':response.url,'category': category,'subCategory':subCategory,'starRating':starRating}
        pass

我期望：
类别子类别评分 地面交通 公共交通选择 3.5 地面交通 到公共交通的距离 4 地面交通 机场出租车服务 4.5 地面交通租车 4.5 安检 安检指南标牌 4 安检等候时间 4 安检服务效率 4 安检人员态度4 安全检查 员工语言技能 4 安全检查 FastTrack 安全 5

任何人都可以帮助这个scrapy noob

标签： pythonscrapy

解决方案

您将一次获得所有必填字段。尝试遍历每个部门并解析结果。

这里有一个例子供你理解。如何循环划分并使用相对 xpath。

def parse(self, response):
    print("processing:" + response.url)
    # Extract using css selectors
    # None

    # Extract using xpath
    categoryBox = ratingbox = response.xpath('//*[@class="ratingstable"]')
    for box in categoryBox:
        category = box.xpath('./h4/text()').get(default='')
        ratingBox = box.xpath('.//*[@class="rating"]')
        for rating in ratingBox:
            star_rating = len(rating.xpath('.//*[@class="fa fa-star"]').getall())
            sub_category = rating.xpath('.//*[@class="rating__title"]/text()').get(default='')
            yield {'page': response.url, 'category': category, 'subCategory': sub_category, 'starRating': star_rating}

    # category = response.xpath(
    #     "//div[@class='ratingstable']/h4[@class='ratingstable__title fontsize-1']/text()").extract()
    # subCategory = response.xpath(
    #     "//div[@class='ratingstable']/div[@class='ratingstable__ratings']/div[@class='rating']/span[@class='rating__title']/text()").extract()
    # starRating = response.xpath(
    #     "//div[@class='ratingstable']/div[@class='ratingstable__ratings']/div[@class='rating']/span[@class='rating__stars']").extract()

    # yield or give the scraped info to scrappy

输出

{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Ground Transportation', 'subCategory': 'Public Transport options', 'starRating': 3}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Ground Transportation', 'subCategory': 'Distance to public transport', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Ground Transportation', 'subCategory': 'Airport taxi service', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Ground Transportation', 'subCategory': 'Car Rental', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Security Screening', 'subCategory': 'Screening guide signage', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Security Screening', 'subCategory': 'Waiting times', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Security Screening', 'subCategory': 'Service efficiency', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Security Screening', 'subCategory': 'Staff attitude and courtesy', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Security Screening', 'subCategory': 'Staff language skills', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Security Screening', 'subCategory': 'FastTrack security', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Immigration Service', 'subCategory': 'Waiting times - arrivals', 'starRating': 3}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Immigration Service', 'subCategory': 'Waiting times - departures', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Immigration Service', 'subCategory': 'Staff attitude and courtesy', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Immigration Service', 'subCategory': 'Staff language skills', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Immigration Service', 'subCategory': 'FastTrack immigration', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Ease of wayfinding - arrivals', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Ease of wayfinding - transfer', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Ease of wayfinding - departures', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Clarity of signs & symbols', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Signage language choices', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'FIDS : positioning in terminal', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'FIDS : clarity of display', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Terminal maps', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Airport App navigation', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Airport "help" staff', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Info Kiosks', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Wayfinding & Signage', 'subCategory': 'Info / Help counter service', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Arrival', 'subCategory': 'Walking distances', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Arrival', 'subCategory': 'Availability of toilets', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Arrival', 'subCategory': 'Baggage hall facilities', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Arrival', 'subCategory': 'Baggage delivery times', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Arrival', 'subCategory': 'Baggage carts', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Arrival', 'subCategory': 'Meet and greet facilities', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Departure', 'subCategory': 'Signage to check-in', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Departure', 'subCategory': 'Congestion around check-in', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Departure', 'subCategory': 'Queuing systems', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Departure', 'subCategory': 'Seating near check-in', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Departure', 'subCategory': 'Premium check-in facility', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Departure', 'subCategory': 'Signage to security / departures', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Transfer', 'subCategory': 'Walking distances', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Transfer', 'subCategory': 'Signage to transfer areas', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Transfer', 'subCategory': 'Transfer security screening', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Transfer', 'subCategory': 'Transfer waiting area', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Transfer', 'subCategory': 'Rest area for transfer customers', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Transfer', 'subCategory': 'Ease of transfer process', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Congestion around terminal', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Air temperature', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Décor and condition', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Seat availability', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Seat types and comfort', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Access to power / charging points', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Cleanliness of seats', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Cleanliness of washrooms', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Cleanliness of public areas', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'PRM washrooms', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Comfort', 'subCategory': 'Baby changing rooms', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': "Clarity of airport PA's", 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'PA language translations', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'ATM availability', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'WiFi : ease of log-in', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'WiFi : free time allowance', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'Internet kiosk facilities', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'Childrens Play area', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'Shower facilities', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'Quiet / relaxation areas', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'Dayroom facility', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'Terminal hotel', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'Other leisure facilities', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Terminal Facilities', 'subCategory': 'Vending machines', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Shopping Facilities', 'subCategory': 'Selection of shops : landside', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Shopping Facilities', 'subCategory': 'Selection of shops : airside', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Shopping Facilities', 'subCategory': 'Choice of mid-price shops', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Shopping Facilities', 'subCategory': 'Choice of luxury brands', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Shopping Facilities', 'subCategory': 'Staff service in shops', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'Food and beverage choice : landside', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'Fast food choice : landside', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'International brands : landside', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'Food and beverage choice : airside', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'Fast food options : airside', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'International brands : airside', 'starRating': 5}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'Food and beverage : cleanliness', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'Food and beverage : prices', 'starRating': 4}
{'page': 'https://skytraxratings.com/airports/hamad-international-airport-rating', 'category': 'Food & Beverage', 'subCategory': 'Food and beverage : staff service', 'starRating': 4}

半星我还没数。只需读取半颗星的数量并将计数除以 2 并添加到 star_rating。

推荐阅读