python-3.x - InternalError：无效的变量引用。TensorFlow 上的 [Op:ResourceApplyAdam]

Question

我目前正在使用 tensorflow 1.7.0 开发 EagerExecution。

我在使用 GPU 时收到此错误：tensorflow.python.framework.errors_impl.InternalError: Invalid variable reference。[操作：ResourceApplyAdam]

不幸的是，我无法隔离错误，所以我无法给出可以解释的片段。

当我在 CPU 上工作时不会发生错误。在最近更新之前，我的代码在 GPU 上运行良好。我认为它与机器无关，因为它发生在不同的机器上。

我无法找到相关的内容，因此如果您对可能导致此错误的原因有任何提示，请告诉我。

完整跟踪：

2018-07-19 17:52:32.393711: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at training_ops.cc:2507 : Internal: Invalid variable reference.
Traceback (most recent call last):
  File "debugging_jules_usage.py", line 391, in <module>
    mainLoop()
  File "debugging_jules_usage.py", line 370, in mainLoop
    raise e
  File "debugging_jules_usage.py", line 330, in mainLoop
    Kn.fit(train)
  File "/home/jbayet/xai-link-prediction/xai_lp/temporal/models_temporal.py", line 707, in fit
    self._train_one_batch(X_bis, i)
  File "/home/jbayet/xai-link-prediction/xai_lp/temporal/models_temporal.py", line 639, in _train_one_batch
    self.optimizer.minimize(batch_model_loss, global_step=tf.train.get_global_step())
  File "/home/jbayet/miniconda3/envs/xai/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 409, in minimize
    name=name)
  File "/home/jbayet/miniconda3/envs/xai/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 564, in apply_gradients
    update_ops.append(processor.update_op(self, grad))
  File "/home/jbayet/miniconda3/envs/xai/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 161, in update_op
    update_op = optimizer._resource_apply_dense(g, self._v)
  File "/home/jbayet/miniconda3/envs/xai/lib/python3.6/site-packages/tensorflow/python/training/adam.py", line 166, in _resource_apply_dense
    grad, use_locking=self._use_locking)
  File "/home/jbayet/miniconda3/envs/xai/lib/python3.6/site-packages/tensorflow/python/training/gen_training_ops.py", line 1105, in resource_apply_adam
    _six.raise_from(_core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: Invalid variable reference. [Op:ResourceApplyAdam]

标签： python-3.xtensorflow