时间戳

首页 > 解决方案 > 在循环变量时获取产生最高/最低皮尔逊相关性的变量

python - 在循环变量时获取产生最高/最低皮尔逊相关性的变量

问题描述

我正在努力实现以下目标：

我有一个数据框，其中包含许多包含指标和一些维度的列，例如country, device, name。这 3 个维度中的每一个都有一些唯一值，我在使用之前使用这些值来过滤数据pd.corr()。

为了演示，我将使用 Titanic 数据集。

import seaborn as sns
df_test = sns.load_dataset('titanic')

for who in df_test['who'].unique():
    for where in df_test['embark_town'].unique():
        print(df_test[(df_test['who']==who)&(df_test['embark_town']==where)].corr())

这会产生不同的df_test['who'].nunique()*df_test['embark_town'].nunique()相关性。9pd.corr()

下面的一个例子：

         survived    pclass       age     sibsp     parch      fare  
survived    1.000000 -0.198092  0.062199 -0.046691 -0.071417  0.108706   
pclass     -0.198092  1.000000 -0.438377  0.008843 -0.015523 -0.485546   
age         0.062199 -0.438377  1.000000 -0.049317  0.077529  0.199062   
sibsp      -0.046691  0.008843 -0.049317  1.000000  0.464033  0.358680   
parch      -0.071417 -0.015523  0.077529  0.464033  1.000000  0.415207   
fare        0.108706 -0.485546  0.199062  0.358680  0.415207  1.000000   
adult_male       NaN       NaN       NaN       NaN       NaN       NaN   
alone       0.030464  0.133638 -0.022396 -0.629845 -0.506964 -0.411392

我正在尝试获取可以回答这个问题的数据：

在什么设置中，每个变量之间的相关性最高/最低，输出可能是 a list, dict，df如下所示：

output = {'highest_corr_survived_p_class':['who = man', 'embark_town = Southampton', 0.65],
         'lowest_corr_survived_p_class':['who = man', 'embark_town = Cherbourg',-0.32],
         'highest_corr_survived_age':['who = female', 'embark_town = Cherbourg',0.75],
         'lowest_corr_survived_age':['who = man', 'embark_town = Cherbourg',-0.3]
         ...
         'lowest_corr_alone_fare':['who = man', 'embark_town = Cherbourg',-0.7]}

我被困的地方是找到一种创建此数据的好方法以及如何将其放入df.

我试过的：

output = {}

for who in df_test['who'].dropna().unique():
    for where in df_test['embark_town'].dropna().unique():
        output[f'{who}_{where}_corr'] =  df_test[(df_test['who']==who)&(df_test['embark_town']==where)].corr().loc['survived','pclass']

产生output：

{'man_Southampton_corr': -0.19809207465001574,
 'man_Cherbourg_corr': -0.2102998217386208,
 'man_Queenstown_corr': 0.06717166132798494,
 'woman_Southampton_corr': -0.5525868192717193,
 'woman_Cherbourg_corr': -0.5549942419871897,
 'woman_Queenstown_corr': -0.16896381511084563,
 'child_Southampton_corr': -0.5086941796202842,
 'child_Cherbourg_corr': -0.2390457218668788,
 'child_Queenstown_corr': nan}

而且这种方法不关心什么是相关性max或min相关性，这是我的目标。

我不确定如何在使用的列之间添加所有可能的变化loc[]，或者是否有更好/更简单的方法将所有内容放入df？

标签： pythonpandascorrelation

解决方案

您可以使用DataFrameGroupBy.corrwith DataFrame.stack、 remove1和-1rows 并获得每组的最大值和最小值 by DataFrameGroupBy.idxmax， DataFrameGroupBy.idxmin对于索引 with Series.locfor select ， join together byconcat和 last 使用字典理解 for final dict：

import seaborn as sns
df_test = sns.load_dataset('titanic')
# print (df_test)

s = df_test.groupby(['who','embark_town']).corr().stack()
s = s[~s.isin([1, -1])]
s = (pd.concat([s.loc[s.groupby(level=[2,3]).idxmax()], 
                s.loc[s.groupby(level=[2,3]).idxmin()]], keys=('highest','lowest'))
       .sort_index(level=[3,4], sort_remaining=False))
print (s)
         who    embark_town                  
highest  child  Queenstown   age       alone     0.877346
lowest   woman  Queenstown   age       alone    -0.767493
highest  woman  Queenstown   age       fare      0.520461
lowest   child  Queenstown   age       fare     -0.877346
highest  woman  Queenstown   age       parch     0.633627
  
lowest   woman  Queenstown   survived  parch    -0.433029
highest  man    Queenstown   survived  pclass    0.067172
lowest   woman  Cherbourg    survived  pclass   -0.554994
highest  man    Queenstown   survived  sibsp     0.232685
lowest   child  Southampton  survived  sibsp    -0.692578
Length: 84, dtype: float64

output = {f'{k[0]}_corr_{k[3]}_{k[4]}':
          [f'who = {k[1]}', f'embark_town = {k[2]}',v] for k, v in s.items()}

print(output)

编辑：对于 TOP3 和 BOTTOM3 是可能的排序和GroupBy.head使用GroupBy.tail：

import seaborn as sns
df_test = sns.load_dataset('titanic')
# print (df_test)

s = df_test.groupby(['who','embark_town']).corr().stack()
s = s[~s.isin([1, -1])].sort_values()

s = (pd.concat([s.groupby(level=[2,3]).head(3), 
                s.groupby(level=[2,3]).tail(3)], keys=('highest','lowest'))
        .sort_index(level=[3,4], sort_remaining=False)
        )
print (s)
         who    embark_town                 
highest  woman  Queenstown   age       alone   -0.767493
                Cherbourg    age       alone   -0.073881
         man    Queenstown   age       alone   -0.069001
lowest   child  Southampton  age       alone    0.169244
                Cherbourg    age       alone    0.361780
  
highest  woman  Southampton  survived  sibsp   -0.252524
         man    Southampton  survived  sibsp   -0.046691
lowest   man    Cherbourg    survived  sibsp    0.125276
         woman  Queenstown   survived  sibsp    0.143025
         man    Queenstown   survived  sibsp    0.232685
Length: 252, dtype: float64

推荐阅读