python - List of NAN values while calculating p value and Z score in Scipy
问题描述
I am calculating Z score and P value for different sub segments within a data frame.
The data frame has two columns, here are the top 5 values in my data frame:
df[["Engagement_score", "Performance"]].head()
Engagement_score Performance
0 6 0.0
1 5 0.0
2 7 66.3
3 3 0.0
4 11 0.0
Here's the distribution of engagement score:
Here's the distribution of performance:
I am grouping my dataframe by engagement score and then I calculate these three statistics for those groups:
1) Average performance score(sub_average) and number of values within that group(sub_bookings)
2) Average performance score for rest of the groups(rest_average) and number of values in rest of the groups(rest_bookings)
Overall performance score and overall bookings are calculated for the overall data frame.
Here's my code to do that.
def stats_comparison(i):
df.groupby(i)['Performance'].agg({
'average': 'mean',
'bookings': 'count'
}).reset_index()
cat = df.groupby(i)['Performance']\
.agg({
'sub_average': 'mean',
'sub_bookings': 'count'
}).reset_index()
cat['overall_average'] = df['Performance'].mean()
cat['overall_bookings'] = df['Performance'].count()
cat['rest_bookings'] = cat['overall_bookings'] - cat['sub_bookings']
cat['rest_average'] = (cat['overall_bookings']*cat['overall_average'] \
- cat['sub_bookings']*cat['sub_average'])/cat['rest_bookings']
cat['z_score'] = (cat['sub_average']-cat['rest_average'])/\
np.sqrt(cat['overall_average']*(1-cat['overall_average'])
*(1/cat['sub_bookings']+1/cat['rest_bookings']))
cat['prob'] = np.around(stats.norm.cdf(cat.z_score), decimals = 10) # this is the p value
cat['significant'] = [(lambda x: 1 if x > 0.9 else -1 if x < 0.1 else 0)(i) for i in cat['prob']]
# if the p value is less than 0.1 then I can confidently say that the 2 samples are different.
print(cat)
stats_comparison('Engagement_score')
I get the following output when I execute my code:
Engagement_score sub_average sub_bookings overall_average \
0 3 57.281118 1234 34.405373
1 4 56.165374 722 34.405373
2 5 52.896404 890 34.405373
3 6 50.275880 966 34.405373
4 7 43.475344 1018 34.405373
5 8 37.693290 1222 34.405373
6 9 30.418053 1695 34.405373
7 10 16.458142 2874 34.405373
8 11 25.604145 1375 34.405373
9 12 10.910013 789 34.405373
overall_bookings rest_bookings rest_average z_score prob significant
0 12785 11551 31.961544 NaN NaN 0
1 12785 12063 33.102984 NaN NaN 0
2 12785 11895 33.021850 NaN NaN 0
3 12785 11819 33.108233 NaN NaN 0
4 12785 11767 33.620702 NaN NaN 0
5 12785 11563 34.057900 NaN NaN 0
6 12785 11090 35.014797 NaN NaN 0
7 12785 9911 39.609727 NaN NaN 0
8 12785 11410 35.465995 NaN NaN 0
9 12785 11996 35.950709 NaN NaN 0
I don't know why I am getting a list of NAN values in ZScore and P value columns. There are no negative values in my data set.
I also get the following warning when I run the code in Jupyter Notebook:
C:\Users\User\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: FutureWarning: using a dict on a Series for aggregation
is deprecated and will be removed in a future version
after removing the cwd from sys.path.
C:\Users\User\Anaconda3\lib\site-packages\ipykernel_launcher.py:8: FutureWarning: using a dict on a Series for aggregation
is deprecated and will be removed in a future version
C:\Users\User\Anaconda3\lib\site-packages\ipykernel_launcher.py:15: RuntimeWarning: invalid value encountered in sqrt
from ipykernel import kernelapp as app
C:\Users\User\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in greater
return (self.a < x) & (x < self.b)
C:\Users\User\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in less
return (self.a < x) & (x < self.b)
C:\Users\User\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:1738: RuntimeWarning: invalid value encountered in greater_equal
cond2 = (x >= self.b) & cond0