时间戳

首页 > 解决方案 > 如何用另一个数据框中最接近的日期填充一个数据框中的日期列

python - 如何用另一个数据框中最接近的日期填充一个数据框中的日期列

问题描述

我有一个数据框visit=

visit_occurrence_id  visit_start_date  person_id
    1                2016-06-01        1
    2                2019-05-01        2
    3                2016-01-22        1
    4                2017-02-14        2
    5                2018-05-11        3

和另一个数据框 measurement=

measurement_date    person_id   visit_occurrence_id
2017-09-04          1           Nan
2018-04-24          2           Nan
2018-05-22          2           Nan
2019-02-02          1           Nan
2019-01-28          3           Nan
2019-05-07          1           Nan
2018-12-11          3           Nan
2017-04-28          3           Nan

我想根据 person_id 和可能的最近日期用访问表的 visit_occurrence_id 填充测量表的 visit_occurrence_id。

我已经写了一个代码，但它需要很多时间。

测量有 7*10^5 行。

注意：visit_start_date 和 measure_date 是对象类型

my code - 

import datetime as dt

unique_person_list = measurement['person_id'].unique().tolist()

def nearest_date(row,date_list):
    date_list = [dt.datetime.strptime(date, '%Y-%m-%d').date() for date in date_list]
    row = min(date_list, key=lambda x: abs(x - row))
    return row

modified_measurement = pd.DataFrame(columns = measurement.columns)

for person in unique_person_list:
    near_visit_dates =  visit[visit['person_id']==person]['visit_start_date'].tolist()
    if near_visit_dates:
        near_visit_dates = list(filter(None, near_visit_dates))
        near_visit_dates = [i.strftime('%Y-%m-%d') for i in near_visit_dates]
        store_dates = measurement.loc[measurement['person_id']== person]['measurement_date']
        store_dates= store_dates.apply(nearest_date, args=(near_visit_dates,))
        modified_measurement = modified_measurement.append(store_dates)

我的代码的执行时间相当长。您能帮助我降低时间复杂度还是提供其他解决方案。

编辑 - 添加数据框构造函数。

import numpy as np

measurement = {'measurement_date':['2017-09-04', '2018-04-24', '2018-05-22', '2019-02-02', 
                                   '2019-01-28', '2019-05-07', '2018-12-11','2017-04-28'],
        'person_id':[1, 2, 2, 1, 3, 1, 3, 3],'visit_occurrence_id':[np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan]}

visit = {'visit_occurrence_id':[1, 2, 3, 4, 5], 
         'visit_start_date':['2016-06-01', '2019-05-01', '2016-01-22', '2017-02-14', '2018-05-11'],
         'person_id':[1, 2, 1, 2, 3]}

# Create DataFrame
measurement = pd.DataFrame(measurement)
visit = pd.DataFrame(visit)

标签： pythonpython-3.xpandasmachine-learningpandas-groupby

解决方案

您可以执行以下操作：

df=pd.merge(measurement[["person_id", "measurement_date"]], visit, on="person_id", how="inner")

df["dt_diff"]=df[["visit_start_date", "measurement_date"]].apply(lambda x: abs(datetime.datetime.strptime(x["visit_start_date"], '%Y-%m-%d').date() - datetime.datetime.strptime(x["measurement_date"], '%Y-%m-%d').date()), axis=1)

df=pd.merge(df, df.groupby(["person_id", "measurement_date"])["dt_diff"].min(), on=["person_id", "dt_diff", "measurement_date"], how="inner")

res=pd.merge(measurement, df, on=["measurement_date", "person_id"], suffixes=["", "_2"])[["measurement_date", "person_id", "visit_occurrence_id_2"]]

输出：

  measurement_date  person_id  visit_occurrence_id_2
0       2017-09-04          1                      1
1       2018-04-24          2                      2
2       2018-05-22          2                      2
3       2019-02-02          1                      1
4       2019-01-28          3                      5
5       2019-05-07          1                      1
6       2018-12-11          3                      5
7       2017-04-28          3                      5

推荐阅读