python - 在 Keras 中使用 fit_generator 的输入/输出错误

Question

我有一个大型训练矩阵，以 h5 格式存储图像和相应的类。我想阅读图像及其标签以在 Keras 中训练模型。由于数据集太大，我创建了自己的数据生成器函数imageLoader()并在model.fit_generator中使用它。imageLoader ()以batch_size大小的批次读取训练数据并将其用于训练。

训练矩阵（h5 文件）包含如下数据：

Train_features：（58160 x 25 x 25 x 50）=> 没有。图像 x 行 x 列 x 通道

train_class: (58160 x 1) => 没有。图像 x 标签

但是，在 Keras 中使用 fit_generator 时仍然出现内存错误。以下是我的代码：

import os
os.environ['PYTHONHASHSEED'] = '0'
import numpy as np
np.random.seed(7)  
from tensorflow import set_random_seed
set_random_seed(2)
import keras
from keras.layers import Input, Dense, Dropout, BatchNormalization
from keras.models import Model, Sequential
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Flatten, Activation
from keras.callbacks import ReduceLROnPlateau
import h5py


h5f = h5py.File('path to h5 file', 'r')

#Data Generator
def imageLoader(h5f, batch_size):
    L= len(h5f['train_class'])
    while True:
        batch_start=0
        batch_end= batch_size
        while batch_start<L:
            limit = min(batch_end, L)
            X= h5f['train_features'][batch_start:limit]
            Y= h5f['train_class'][batch_start:limit]
            print(batch_start, batch_end)

            yield (X,keras.utils.to_categorical(Y,2)) #I have two classes

            batch_start +=batch_size
            batch_end +=batch_size

# Building Model
model= Sequential()
model.add(Conv2D(80,(4,4), activation='relu', input_shape= (25,25,50)))
model.add(MaxPooling2D(pool_size= (2,2)))
model.add(Conv2D(150,(4,4), activation = 'relu'))
model.add(MaxPooling2D(pool_size= (2,2)))
model.add(Conv2D(200,(4,4), activation = 'relu')) 
model.add(Flatten())
model.add(Dense(2, activation = 'softmax'))
model.summary()


model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = 
['accuracy'])

model.fit_generator(imageLoader(h5f,100), 
steps_per_epoch=len(h5f['train_class'])/100, epochs=3)

这是错误日志：

Epoch 1/3
0 100
100 200
200 300
300 400
400 500
500 600
600 700
700 800
800 900
900 1000
1000 1100
1/54 [..............................] - ETA: 22s - loss: 0.9183 - acc: 
0.11001100 1200
1200 1300
1300 1400
1400 1500
5/54 [=>............................] - ETA: 4s - loss: 0.9579 - acc: 0.0380 
1500 1600
7/54 [==>...........................] - ETA: 42s - loss: 0.8970 - acc: 
0.1429
Exception in thread Thread-28:

 Traceback (most recent call last):
 File "/home/saror/.pyenv/versions/anaconda3-4.4.0/envs/py3dl2/lib/python3.6/threading.py", line 916, in _bootstrap_inner
 self.run()
 File "/home/saror/.pyenv/versions/anaconda3-4.4.0/envs/py3dl2/lib/python3.6/threading.py", line 864, in run
 self._target(*self._args, **self._kwargs)

File "/home/saror/.pyenv/versions/anaconda3- 4.4.0/envs/py3dl2/lib/python3.6/site-packages/keras/utils/data_utils.py", 
line 579, in data_generator_task
generator_output = next(self._generator)
  File "<ipython-input-119-7a1d852e803c>", line 10, in imageLoader
    X= h5f['test_features'][batch_start:limit]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/saror/.pyenv/versions/anaconda3-4.4.0/envs/py3dl2/lib/python3.6/site-packages/h5py/_hl/dataset.py", line 496, in __getitem__
self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (file read failed: time = Wed Oct  3 23:43:01 2018
, filename = '/mnt/hdd/shreya/change_detection_files/san_test_data.h5', file 
descriptor = 86, errno = 5, error message = 'Input/output error', buf = 
0x7f53b2990c80, total read size = 6415424, bytes this sub-read = 6415424, 
bytes actually read = 18446744073709551615, offset = 418586624)

Traceback (most recent call last):

  File "<ipython-input-120-e74f6fee0af9>", line 15, in <module>
    model.fit_generator(imageLoader(h5f,100), 
steps_per_epoch=len(h5f['test_class'])/100, epochs=3)

  File "/home/saror/.pyenv/versions/anaconda3-4.4.0/envs/py3dl2/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)

  File "/home/saror/.pyenv/versions/anaconda3-4.4.0/envs/py3dl2/lib/python3.6/site-packages/keras/models.py", line 1156, in fit_generator
initial_epoch=initial_epoch)

  File "/home/saror/.pyenv/versions/anaconda3-4.4.0/envs/py3dl2/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)

  File "/home/saror/.pyenv/versions/anaconda3-4.4.0/envs/py3dl2/lib/python3.6/site-packages/keras/engine/training.py", line 2046, in fit_generator
generator_output = next(output_generator)

StopIteration   

谁能告诉上面的代码有什么问题？如何高效读取大数据进行模型训练？

提前致谢！

标签： pythonkerashdf5large-data