时间戳

首页 > 解决方案 > 根据日期键的值过滤字典

python - 根据日期键的值过滤字典

问题描述

我想从某个日期开始从世界各地的尽可能多的来源导入文章。

import requests
url = ('https://newsapi.org/v2/top-headlines?'
       'country=us&'
       'apiKey=de9e19b7547e44c4983ad761c104278f')
response = requests.get(url)

response_dataframe = pd.DataFrame(response.json())

articles = {article for article in response_dataframe['articles'] if article['publishedAt'] >= '2019-01-04T11:30:00Z'}
print(articles)

但我得到：

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-84-0f21f2f50907> in <module>
      2 response_dataframe['articles'][1]['publishedAt']
      3 
----> 4 articles = {article for article in response_dataframe['articles'] if article['publishedAt'] >= '2018-01-04T11:30:00Z'}
      5 print(articles)

<ipython-input-84-0f21f2f50907> in <setcomp>(.0)
      2 response_dataframe['articles'][1]['publishedAt']
      3 
----> 4 articles = {article for article in response_dataframe['articles'] if article['publishedAt'] >= '2018-01-04T11:30:00Z'}
      5 print(articles)

TypeError: unhashable type: 'dict'

那么如何通过在这些键上进行选择来选择一系列文章呢？预期的输出是按天和报纸对文章进行排序的数据框。

              The New York Times                                The Washington Post                                The Financial Times  
2007-01-01    . What Sticks from '06. Somalia Orders Islamis... New Ebola Vaccine Gives 100 Percent Protecti...
2007-01-02    . Heart Health: Vitamin Does Not Prevent Death... Flurry of Settlements Over Toxic Mortgages M...
2007-01-03    . Google Answer to Filling Jobs Is an Algorith... Jason Miller Backs Out of White House Commun...
2007-01-04    . Helping Make the Shift From Combat to Commer... Wielding Claims of ‘Fake News,’ Conservative...
2007-01-05    . Rise in Ethanol Raises Concerns About Corn a... When One Party Has the Governor’s Mansion an
...

我的 Python 版本是 3.6.6

标签： pythonpython-3.xapidatefilter

解决方案

您正在过滤字典，然后尝试将它们放入一个集合中。您的预期结果不需要您对任何内容进行重复数据删除，因此避免错误的最简单方法是使用列表推导；只需将花括号换成{...}方括号：

articles = [article for article in response_dataframe['articles'] if article['publishedAt'] >= '2019-01-04T11:30:00Z']

但是，如果您要将数据放入数据框中进行处理，那么使用该pandas.io.json.json_normalize()函数会更好；它可以从通常从 JSON 源加载的列表和字典结构中为您生成数据框。

首先将您想要的文章数据加载到数据框中，然后您可以从那里过滤和重新排列；以下代码将所有数据加载到具有新列的单个数据框中，该新date列源自publishAt信息：

import pandas as pd
from pandas.io.json import json_normalize

df = json_normalize(response.json(), 'articles')

# make the datetime column a native type, and add a date-only column
df['publishedAt'] = pd.to_datetime(df['publishedAt'])
df['date'] = df['publishedAt'].dt.date

# move source dictionary into separate columns rather than dictionaries
source_columns = df['source'].apply(pd.Series).add_prefix('source_')
df = pd.concat([df.drop(['source'], axis=1), source_columns], axis=1)

这为您提供了一个包含所有文章信息的数据框，作为具有本机类型的完整数据框，其中包含映射中的列author、content、description、publishedAt、date、title、和列。urlurlToImagesource_idsource_namesource

我注意到API已经允许您按日期过滤，我会依赖它而不是本地过滤，因为您可以通过让 API 为您提供更小的数据集来节省时间和带宽。API 还允许您应用排序，这也是一个好主意。

要按日期和源名称对行进行分组，您必须旋转数据框；日期应该是索引，列是源名称，标题是值：

df.pivot(index='date', columns='source_name', values='title')

然而，这失败了，因为这种格式没有空间为每个来源每天提供一个以上的标题：

ValueError: Index contains duplicate entries, cannot reshape

在提供给我的 JSON 数据中，有多篇 CNN 和 Fox News 文章仅适用于今天。

您可以将多个标题聚合到列表中：

pd.pivot_table(df,
    index='date', columns='source_name', values='title',
    aggfunc=list)

对于“今天”的默认 20 个结果，这给了我：

>>> pd.pivot_table(
...     df, index='date', columns='source_name', values='title',
...     aggfunc=list
... )
source_name                                            Bbc.com                        ...                                                                Youtube.com
date                                                                                  ...
2019-01-05   [Paul Whelan: Russia rules out prisoner swap f...                        ...                          [Bears Buzz: Eagles at Bears - Wildcard Round ...

[1 rows x 18 columns]

就个人而言，我只是将数据框限制为日期、标题和源名称，并带有日期索引：

>>> df[['date', 'source_name', 'title']].set_index('date').sort_values(['date', 'source_name'])
                    source_name                                              title
date
2019-01-05              Bbc.com  Paul Whelan: Russia rules out prisoner swap fo...
2019-01-05            Bloomberg  Russia Says FBI Arrested Russian Citizen on Pa...
2019-01-05                  CNN  Pay raises frozen for Pence, Cabinet members u...
2019-01-05                  CNN  16 big questions on Robert Mueller's Russia in...
2019-01-05            Colts.com  news What They're Saying: Colts/Texans, Wild C...
2019-01-05             Engadget  Pandora iOS update adds offline playback for A...
2019-01-05             Espn.com  Roger Federer wins Hopman Cup with Switzerland...
2019-01-05             Fox News  Japanese 'Tuna King' pays record $3M for prize...
2019-01-05             Fox News  Knicks' Turkish star Enes Kanter to skip Londo...
2019-01-05          Latimes.com  Flu toll mounts in California, with 42 deaths ...
2019-01-05             NBC News  After the fire: Blazes pose hidden threat to t...
2019-01-05           Newser.com  After Backlash, Ellen Not Ditching Support for...
2019-01-05              Npr.org  Three Dead After Fight Escalates Into Shooting...
2019-01-05              Reuters  French 'yellow vests' rail against unrepentant...
2019-01-05             The Hill  Trump: 'I don’t care' that most federal employ...
2019-01-05  The Huffington Post  5 Children Dead After Church Van Crashes On Wa...
2019-01-05            The Verge  Apple seeks to end bent iPad Pro controversy w...
2019-01-05    Thisisinsider.com  Kanye West surprised Kim Kardashian with a $14...
2019-01-05            USA Today  See 'Mean Girls' co-stars Lindsay Lohan and Jo...
2019-01-05          Youtube.com  Bears Buzz: Eagles at Bears - Wildcard Round -...

以上按日期和来源排序，因此来自同一来源的多个标题被分组。

推荐阅读