python - 将 json 源中的数据抓取到 csv 中

Question

我正在尝试从网上商店的 python 网站获取一些信息。

我试过这个：

my_url = requests.get(https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL)

data = my_url.json()

name = data['MainContent'][0]['contents'][0]['productList']['products'][0]['productModel']["displayName"]
price = data['MainContent'][0]['contents'][0]['productList']['products'][0]['priceInfo']['priceItemSale']["gross"]
url= data['MainContent'][0]['contents'][0]['productList']['products'][0]['productModel']["url"]

for mc in data['MainContent']:
    for co in mc:
        for prod in co['productList']['products']:
            name = prod['productModel']['displayName']
            price = prod['priceItemSale']['gross']
            url = prod['productModel']['url']

filename = "test.csv"
csv_writer = csv.writer(open(filename, 'w'))
headers = "Name, Price, Link\n"
f.write(headers)
f.close()

在这个网上商店有很多具有这些属性“productModel”的产品，但我怎样才能得到这些并写入 csv？我想在不同的单元格中抓取名称、价格和 URL 链接此页面，但它不起作用。

* 编辑：

def proba():

    my_url = requests.get('https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL')
    data = my_url.json()
    results = []
    products = data['MainContent'][0]['contents'][0]['productList']['products']
    for product in products:
        name = product['productModel']['displayName']
        try:
            priceGross = product['priceInfo']['priceItemSale']['gross']
        except:
            priceGross = product['priceInfo']['priceItemToBase']['gross']
        url = product['productModel']['url']
        results.append([name, priceGross, url])
    df = pd.DataFrame(results, columns = ['Name', 'Price', 'Url'])    
# print(df)  ## print df
    df.to_csv(r'/usr/src/Python-2.7.13/test.csv', sep=',', encoding='utf-8-sig',index = False )

while True:
    mytime=datetime.now().strftime("%H:%M:%S")
    while mytime < "23:59:59":
    print mytime
    proba()
    mytime=datetime.now().strftime("%H:%M:%S")

标签： pythonjsonpython-2.7web-scrapingrequest

Answer 1

并非所有项目都具有相同数量的密钥，因此您无法 prod['priceItemSale']['gross']始终访问。当价值不存在时，您需要决定从哪里获得价值。下面的例子：

import requests
import pandas as pd

my_url = requests.get('https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL')
data = my_url.json()
results = []
products = data['MainContent'][0]['contents'][0]['productList']['products']
for product in products:
    name = product['productModel']['displayName']
    try:
        priceGross = product['priceInfo']['priceItemSale']['gross']
    except:
        priceGross = product['priceInfo']['priceItemToBase']['gross']
    url = product['productModel']['url']
    results.append([name, priceGross, url])
df = pd.DataFrame(results, columns = ['Name', 'Price', 'Url'])    
# print(df)  ## print df
df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8-sig',index = False )

---

输出：