> when trying to save data

I know this question was already on the OS, but I still haven't found a solution to my problem. I think the problem is that X has dimensio,python"/>

时间戳

首页 > 解决方案 > 获取 <> when trying to save data

I know this question was already on the OS, but I still haven't found a solution to my problem. I think the problem is that X has dimensio

python - 获取 <> when trying to save data

I know this question was already on the OS, but I still haven't found a solution to my problem. I think the problem is that X has dimensio

问题描述

I know this question was already on the OS, but I still haven't found a solution to my problem. I think the problem is that X has dimension 3. How can I transmit X to data so that the size is taken into account? In my sample, I try to save data using h5py. X has dtype = uint8.

file.create_dataset('X', data=np.array(X))

In this case the following error appear:

TypeError: Object dtype dtype('O') has no native HDF5 equivalent

Multivariate normal calculation not producing standard normal variables

This may be more of a math question than numpy, so I'm not sure, this may need to be posted on Cross Validated instead.

Problem description

I am using numpy's function np.random.multivariate_normal() to produce randomized, correlated data. I want to have 5 correlated variables, each distributed standard normal. I believe my covariance matrix is correctly specified, with diagonals equal to 1.0 and the inner values being my desired correlations. The generated data consistently have the correct mean of 0, but the standard deviation is unpredictable.

Reproducible example

Here is a reproducible code example:

# rho = randomizer.config.customers.correlation
rho = np.array([
       [  1.0 , -0.2 ,  0.7 ,  0.1 ,  0.1 ],
       [ -0.2 ,  1.0 ,  0.7 ,  0.5 ,  0.2 ],
       [  0.7 ,  0.7 ,  1.0 ,  0.6 ,  0.5 ],
       [  0.1 ,  0.5 ,  0.6 ,  1.0 ,  0.3 ],
       [  0.1 ,  0.2 ,  0.5 ,  0.3 ,  1.0 ]
])
# rho = np.identity(5)
mu = np.repeat(0, 5)


standard_deviations = []
for i in range(10000):
    np.random.seed()
    sd = np.std(np.random.multivariate_normal(mu, rho, size=1000)[:, 0])
    standard_deviations.append(sd)
    
plt.show(plt.hist(standard_deviations, edgecolor="black", bins=40))

This produces this plot:

enter image description here

The standard deviation is clearly greater than the 1.0 specified in the covariance/correlation matrix.

Is there an obvious explanation for this that I'm missing? I notice that if I use an identity matrix instead of my correlation matrix, the problem goes away:

rho = np.identity(5)
mu = np.repeat(0, 5)


standard_deviations = []
for i in range(10000):
    np.random.seed()
    sd = np.std(np.random.multivariate_normal(mu, rho, size=1000)[:, 0])
    standard_deviations.append(sd)
    
plt.show(plt.hist(standard_deviations, edgecolor="black", bins=40))

enter image description here

Really appreciate any help!

标签： python

解决方案

当您的 numpy 数组是参差不齐的数组（第二维子数组的大小不同）时，通常会发生这种情况。例如; ragged_array_X = np.array([[3,3],[1,1,1],[2,0,10,90]]). 快速修复将类似于：file.create_dataset('X', data=np.array(ragged_array_X).to_list())

如果不是这种情况，并且您没有参差不齐的数组，那么您可以尝试将 numpy 数组显式转换为 uint8 数据类型。如果此实例化失败，则数组的某些部分是损坏的数据类型（可能是 nan 或其他缺失值），您应该先修复它。

file.create_dataset('X', data=np.array(ragged_array_X, dtype=uint8)

似乎这些建议可能不是您想要的，因为您说“考虑了大小”。看起来您的数组是 3 维的（不确定是否参差不齐）。

假设最坏的情况：你真的想保留一个参差不齐但 3 维数组的形状。HDF5 对此有一个新功能：

来自：（参见：https ://docs.h5py.org/en/stable/special.html ）

dt = h5py.vlen_dtype(np.dtype('int32'))
dset = f.create_dataset('vlen_int', (100,), dtype=dt)
dset[0] = [1,2,3]
dset[1] = [1,2,3,4,5]

当它应用您的示例时，解决方案很简单：

dt = h5py.vlen_dtype(np.dtype('uint8'))
dset = f.create_dataset('X', np.array(X), dtype=dt)

推荐阅读