tensorflow - Tensorflow - 有没有一种简单的方法可以将小批量中损失最大的样本的损失归零？

Question

我正在训练一个用于分类的神经网络。在我的研究中，我想将每个 minibatch 中的 (k) 最高损失归零。如果在某种程度上不依赖 numpy，我想不出一种简单的方法来执行此过程。

我尝试了以下过程：
1. 计算损失数组的 argmax 索引——它返回一个 tf 张量
2. 用索引数组切片损失张量

问题是无法使用 tf Tensor 执行切片。

# losses is tf.Tensor
ind_sorted = tf.argsort(losses)
losses_sorted = losses[ind_sorted] # Error mentioned above
# The issue is that ind_1_sorted depends on the output of the neural network. I couldn't find an equivalent of the detach method in pytorch

k_smallest_losses = losses_sorted[:k] # Keeping only the k smallest losses
loss = tf.sum(k_smallest_losses) # Performing the summation of the k smallest losses

标签： tensorflow

Answer 1

可能您想使用tf.nn.top_k，它会返回 top_k 项的值和索引。（请注意，为了获得最小的损失，我会在您的损失中添加一个负数，并在完成后将其转换回来）。

batch = 2
max_len = 6
losses = tf.random.uniform(shape=[batch, max_len], minval=0, maxval=2, dtype = tf.float32)
bottom_losses_values, bottom_losses_indices = tf.nn.top_k(-losses, k=3)
total = tf.reduce_sum(-bottom_losses_values, axis=-1)
with tf.Session() as sess:
  losses, bottom_losses_values, bottom_losses_indices, total = sess.run([losses, bottom_losses_values, bottom_losses_indices, total])
  print 'original losses\n', losses
  print 'bottom 3 loss values\n', -bottom_losses_values
  print 'bottom 3 loss indices\n', bottom_losses_indices
  print 'total\n', total

结果：

original losses
[[ 1.45301318  1.65069246  1.31003475  1.71488905  1.71400714  0.0543921 ]
 [ 0.09954047  0.12081003  0.24793792  1.51561213  1.73758292  1.43859148]]
bottom 3 loss values
[[ 0.0543921   1.31003475  1.45301318]
 [ 0.09954047  0.12081003  0.24793792]]
bottom 3 loss indices
[[5 2 0]
 [0 1 2]]
total
[ 2.81744003  0.46828842]