tensorflow - 导出 TensorFlow 模型 - AssertionError:未指定检查点(save_path=None);什么都没有恢复
问题描述
我在 Ubuntu 机器上使用 google colab 和 tensorflow 2.3.0,并从这里完成示例: Tensorlow2 Training Custom Model
这是我的代码:
!python exporter_main_v2.py --input_type image_tensor --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config --trained_checkpoint_dir=/models/my_ssd_resnet50_v1_fpn --output_directory=exported-models/my_model/
我收到以下错误:
2020-09-06 08:03:23.830447: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-09-06 08:03:25.844063: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-09-06 08:03:25.879149: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:25.879813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.73GiB deviceMemoryBandwidth: 298.08GiB/s
2020-09-06 08:03:25.879853: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-09-06 08:03:25.881273: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-09-06 08:03:25.882999: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-09-06 08:03:25.883384: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-09-06 08:03:25.885102: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-09-06 08:03:25.886330: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-09-06 08:03:25.889988: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-09-06 08:03:25.890105: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:25.891047: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:25.891854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-09-06 08:03:25.901457: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2200000000 Hz
2020-09-06 08:03:25.901653: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2cdd480 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-06 08:03:25.901678: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-09-06 08:03:26.012959: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:26.013665: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2cdd640 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-06 08:03:26.013697: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2020-09-06 08:03:26.013935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:26.014510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.73GiB deviceMemoryBandwidth: 298.08GiB/s
2020-09-06 08:03:26.014556: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-09-06 08:03:26.014600: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-09-06 08:03:26.014625: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-09-06 08:03:26.014647: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-09-06 08:03:26.014667: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-09-06 08:03:26.014689: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-09-06 08:03:26.014712: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-09-06 08:03:26.014784: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:26.015364: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:26.015875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-09-06 08:03:26.015919: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-09-06 08:03:26.651590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-06 08:03:26.651650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2020-09-06 08:03:26.651663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2020-09-06 08:03:26.651874: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:26.652564: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-06 08:03:26.653153: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-09-06 08:03:26.653195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13962 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
Traceback (most recent call last):
File "exporter_main_v2.py", line 159, in <module>
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "exporter_main_v2.py", line 155, in main
FLAGS.side_input_types, FLAGS.side_input_names)
File "/usr/local/lib/python3.6/dist-packages/object_detection-0.1-py3.6.egg/object_detection/exporter_lib_v2.py", line 260, in export_inference_graph
status.assert_existing_objects_matched()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/util.py", line 885, in assert_existing_objects_matched
"No checkpoint specified (save_path=None); nothing is being restored.")
AssertionError: No checkpoint specified (save_path=None); nothing is being restored.
我已经使用 Tensorflow1 完成了一个不同的示例并遇到了同样的问题(我认为)并在这里寻求帮助: Stackovefflow question
指定目录中有多个检查点文件。训练似乎如期进行。我真的很难过。请问有人可以帮忙吗?
解决方案
中应该只有一个检查点trained_checkpoint_dir
。删除不必要的检查点。
推荐阅读
- python - 使用 Python 登录网站
- php - php 查询不使用 UTF-8 字符集
- php - WordPress有什么方法或任何片段来最小化标签'style amp-custom'中指定的样式表的渲染大小?
- git - 无法通过 GoDaddy 的 SMTP 服务器使用 Git 命令发送电子邮件
- java - 在 Android 中读取内部存储文件花费了太多时间
- html - 单选按钮上的 Angular2+(单击)事件在 Safari/iOS 上不起作用
- java - 如何将不同的图像放入回收站视图中的每个卡片视图中?
- extjs - 包含 ExtReact 主题的问题
- google-app-engine - AppEngine appcfg.py 抱怨版本错误
- python - 有没有像 Clojure 的入门之类的 python 函数?