tensorflow - 使用 tensorflow 处理大尺寸视频时内存耗尽

Question

我正在使用一个简单的语义分割网络，我使用预训练模型来预测图像上的类。一切都适用于图像甚至小视频。但是，我想通过网络并输出处理后的视频的视频文件大约为 900MB。每当我尝试使用它时，它也可以处理大约 200 帧，然后一切都崩溃并出现以下错误：

(0) Resource exhausted:  OOM when allocating tensor with shape[327635100] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc    
         [[{{node StatefulPartitionedCall/StatefulPartitionedCall/while/body/_628/while/cond_1/else/_924/while/cond_1/norm/Sum}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

我正在使用 opencv 来阅读视频：

cap = cv2.VideoCapture(filename)

# Define the codec and create VideoWriter object
fourcc = cv2.VideoWriter_fourcc(*'MP42')
out = cv2.VideoWriter('output.avi', fourcc, 20.0, (1656, 949))
count = 0
while(cap.isOpened()):
    ret, image = cap.read()
    # plt.imshow(image)
    if ret:
      print(image.shape)
      plt.imshow(image)
      image = cv2.resize(image, (1656, 949))
      image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)  
      print(image.shape)
      plt.imshow(image)
    
      output = LOADED_MODEL(tf.cast(image, tf.uint8))
      panoptic_map, used_colors = color_panoptic_map(output['panoptic_pred'][0], DATASET_INFO, 60)
      dst = cv2.addWeighted(image, 0.5, panoptic_map, 1,1)
      print('Read a new frame: ', ret)
      count += 1
      #plt.imshow(dst)
      dst = cv2.resize(dst, (1656, 949))
      out.write(dst)
    
     
out.release()
cap.release()

在小尺寸视频上，一切都很完美。当我检查 GPU 内存使用情况时，它肯定已经耗尽了。除了 32GB RAM 外，我还有 1080 Ti 和 8GB 专用内存。

无论如何，我可以通过将部分保存在磁盘上并稍后合并它们来处理这个视频吗？或者任何其他可以避免内存溢出的方法？

谢谢！

编辑：目前，我正在尝试将视频处理逻辑强制为 cpu 而不是 gpu。强制使用 cpuwith tf.device("/cpu:0"):也不起作用。我仍然得到同样的错误：

2021-09-08 09:44:02.367723: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 2772297216 totalling 2.58GiB
2021-09-08 09:44:02.367855: I tensorflow/core/common_runtime/bfc_allocator.cc:1058] Sum Total of in-use chunks: 3.99GiB
2021-09-08 09:44:02.368003: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] total_region_allocated_bytes_: 6930300928 memory_limit_: 6930300928 available bytes: 0 curr_region_allocation_bytes_: 13860601856
2021-09-08 09:44:02.368153: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Stats:
Limit:                      6930300928
InUse:                      4285491712
MaxInUse:                   6300585472
NumAllocs:                      618063
MaxAllocSize:               3192342016
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2021-09-08 09:44:02.369071: W tensorflow/core/common_runtime/bfc_allocator.cc:467] **************__**********************************************________________****__________________
2021-09-08 09:44:02.369234: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at reduction_ops_common.h:186 : Resource exhausted: OOM when allocating tensor with shape[346537125] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):

标签： tensorflowopencvnvidiasemantic-segmentation