tensorflow - tf.layers.batch_normalization 中“可训练”和“训练”标志的意义
问题描述
tf.layers.batch_normalization 中“trainable”和“training”标志的意义是什么?这两者在训练和预测过程中有何不同?
解决方案
批量规范有两个阶段:
1. Training:
- Normalize layer activations using `moving_avg`, `moving_var`, `beta` and `gamma`
(`training`* should be `True`.)
- update the `moving_avg` and `moving_var` statistics.
(`trainable` should be `True`)
2. Inference:
- Normalize layer activations using `beta` and `gamma`.
(`training` should be `False`)
用于说明少数情况的示例代码:
#random image
img = np.random.randint(0,10,(2,2,4)).astype(np.float32)
# batch norm params initialized
beta = np.ones((4)).astype(np.float32)*1 # all ones
gamma = np.ones((4)).astype(np.float32)*2 # all twos
moving_mean = np.zeros((4)).astype(np.float32) # all zeros
moving_var = np.ones((4)).astype(np.float32) # all ones
#Placeholders for input image
_input = tf.placeholder(tf.float32, shape=(1,2,2,4), name='input')
#batch Norm
out = tf.layers.batch_normalization(
_input,
beta_initializer=tf.constant_initializer(beta),
gamma_initializer=tf.constant_initializer(gamma),
moving_mean_initializer=tf.constant_initializer(moving_mean),
moving_variance_initializer=tf.constant_initializer(moving_var),
training=False, trainable=False)
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
init_op = tf.global_variables_initializer()
## 2. Run the graph in a session
with tf.Session() as sess:
# init the variables
sess.run(init_op)
for i in range(2):
ops, o = sess.run([update_ops, out], feed_dict={_input: np.expand_dims(img, 0)})
print('beta', sess.run('batch_normalization/beta:0'))
print('gamma', sess.run('batch_normalization/gamma:0'))
print('moving_avg',sess.run('batch_normalization/moving_mean:0'))
print('moving_variance',sess.run('batch_normalization/moving_variance:0'))
print('out', np.round(o))
print('')
何时training=False
和trainable=False
:
img = [[[4., 5., 9., 0.]...
out = [[ 9. 11. 19. 1.]...
The activation is scaled/shifted using gamma and beta.
何时training=True
和trainable=False
:
out = [[ 2. 2. 3. -1.] ...
The activation is normalized using `moving_avg`, `moving_var`, `gamma` and `beta`.
The averages are not updated.
何时traning=True
和trainable=True
:
The out is same as above, but the `moving_avg` and `moving_var` gets updated to new values.
moving_avg [0.03249997 0.03499997 0.06499994 0.02749997]
moving_variance [1.0791667 1.1266665 1.0999999 1.0925]
推荐阅读
- javascript - 基于对象响应Angular显示复选框
- python - Numpy 返回“TypeError:'float' 对象不能被解释为整数”
- reactjs - 为嵌套在状态中的值创建排序依据按钮
- css - 正确使用 css 变量
- python - 在 python3 中从 bash 执行 DNS 查询的最佳方法是什么?
- django - ValueError: save() 禁止以防止由于未保存的相关对象表单而导致数据丢失 - 使用表单 obj 作为 fk 创建模型
- angular - How can we use HTTP_INTERCEPTORS to call refresh token, whenever it gets expired
- python - 如何打印不带括号的元组?
- c - 为什么输出 -1.#IND00 会出现此浮点错误?
- javascript - 表单具有先前填充的数据