tensorflow - GPU中的TensorFlow内存分配
问题描述
使用 TensorFlow 1.9,我正在用一些玩具数据训练一个简单的神经网络,试图了解 TensorFlow 如何在 GPU 中分配内存。我的显卡是 NVIDIA GeForce GTX 780 Ti,它有 3GB 的 GPU 内存。
在我的代码中,我创建了数据并设置了批处理大小,使得一个批处理占用的内存量为 4GB。这在代码中通过打印出包含此数据的 NumPy 数组的字节数来验证。
当我运行此代码时,我收到以下警告消息,该消息在每批中打印 3 次:
2018-08-03 14:24:50.021264: W tensorflow/core/framework/allocator.cc:108] Allocation of 4000000000 exceeds 10% of system memory.
由此,我有两个问题:
1)这个警告信息是什么意思?10% 什么内存?GPU内存?
2) 4GB 大小的批次如何能装在 3GB 大小的 GPU 中?. 批次是否分为子批次,每个批次都通过 GPU 独立发送?
如果感兴趣,那么我的完整代码如下:
# Python imports
import numpy as np
# Tensorflow imports
import tensorflow as tf
# Set some parameters
np.random.seed(0)
num_examples = 2000000
input_size = 1000
num_training_examples = int(0.8 * num_examples)
num_validation_examples = int(0.2 * num_examples)
batch_size = 1000000
# Create the data
print('Creating data')
input_data = np.random.rand(num_examples, input_size).astype(np.float32)
label_data = np.random.rand(num_examples, 1).astype(np.float32)
training_input_data = input_data[:num_training_examples]
training_label_data = label_data[:num_training_examples]
validation_input_data = input_data[num_training_examples:]
validation_label_data = label_data[num_training_examples:]
print('Data created')
# Get the memory for the data
data_memory = training_input_data.nbytes + training_label_data.nbytes + validation_input_data.nbytes + validation_label_data.nbytes
data_memory /= 1e6
print('Dataset memory = ' + str(data_memory) + ' MB')
example_memory = training_input_data[0].nbytes + training_label_data[0].nbytes
batch_memory = example_memory * batch_size
batch_memory /= 1e6
print('Batch memory = ' + str(batch_memory) + ' MB')
# Create the placeholders
input_placeholder = tf.placeholder(dtype=np.float32, shape=[None, input_size])
label_placeholder = tf.placeholder(dtype=np.float32, shape=[None, 1])
# Create the network
x = tf.layers.dense(inputs=input_placeholder, units=input_size, activation=tf.nn.relu)
x = tf.layers.dense(inputs=x, units=50, activation=tf.nn.relu)
x = tf.layers.dense(inputs=x, units=50, activation=tf.nn.relu)
x = tf.layers.dense(inputs=x, units=50, activation=tf.nn.relu)
predictions = tf.layers.dense(inputs=x, units=1)
# Define the loss
loss_op = tf.reduce_mean(tf.square(label_placeholder - predictions))
# Define the optimiser
train_op = tf.train.AdamOptimizer(learning_rate=0.001).minimize(loss_op)
# Run a TensorFlow session
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
# Loop over epochs
num_training_batches = int(num_training_examples / batch_size)
num_validation_batches = int(num_validation_examples / batch_size)
training_losses = []
validation_losses = []
for epoch_num in range(1000):
print('epoch ' + str(epoch_num))
# Training
batch_loss_sum = 0
for batch_num in range(num_training_batches):
print('batch ' + str(batch_num))
batch_inputs = training_input_data[batch_num * batch_size: (batch_num + 1) * batch_size]
batch_labels = training_label_data[batch_num * batch_size: (batch_num + 1) * batch_size]
batch_loss, _ = sess.run([loss_op, train_op], feed_dict={input_placeholder: batch_inputs, label_placeholder: batch_labels})
batch_loss_sum += batch_loss
training_loss = batch_loss_sum / num_training_batches
# Validation
batch_loss_sum = 0
for batch_num in range(num_validation_batches):
batch_inputs = validation_input_data[batch_num * batch_size: (batch_num + 1) * batch_size]
batch_labels = validation_label_data[batch_num * batch_size: (batch_num + 1) * batch_size]
batch_loss, _ = sess.run([loss_op, train_op], feed_dict={input_placeholder: batch_inputs, label_placeholder: batch_labels})
batch_loss_sum += batch_loss
validation_loss = batch_loss_sum / num_validation_batches
解决方案
推荐阅读
- reactjs - React-final-form 忽略字段验证属性更改
- java - Lombok 安装程序弹出窗口阻止我运行构建
- c# - 无论如何要为单个 [TestMethod] 覆盖 [TestCleanup]?
- mysql - 如何为自动更新创建触发器
- c# - 如何使用 Google Speech cloud api 识别不同的说话者?
- django - 无法使用扩展用户模型正确保存数据
- mysql - MYSQL group by 用于单独的 id
- python - python的链式赋值不考虑顺序
- python - 提示用户输入字符
- javascript - Android原生分享菜单:检测支持的浏览器