how to read batches in one hdf5 data file for training?
how to read batches in one hdf5 data file for training? I have a hdf5 training dataset with size (21760, 1, 33, 33) . 21760 is the whole number of training samples. I want to use the mini-batch training data with the size 128 to train the network. (21760, 1, 33, 33) 21760 128 I want to ask: How to feed 128 mini-batch training data from the whole dataset with tensorflow each time? 128 3 Answers 3 You can read the hdf5 dataset into a numpy array, and feed slices of the numpy array to the TensorFlow model. Pseudo code like the following would work : import numpy, h5py f = h5py.File('somefile.h5','r') data = f.get('path/to/my/dataset') data_as_array = numpy.array(data) for i in range(0, 21760, 128): sess.run(train_op, feed_dict={input:data_as_array[i:i+128, :, :, :]}) Thank you. But when the number of training iterations i is large, e.g. 100000, ...