python - reprojectImageto3d 后左右图像点重投影不一致

问题描述

每个人。我试图在涉及两个摄像机的设置中对位于平面上的一些点（密集重建）进行三角测量。【参考图片】：https ://imgur.com/gOps4vP和【其他图片】：https ://imgur.com/VIiH9Rv

首先，我在基本矩阵估计的未失真点上使用 5pts 算法解决了相对位姿问题，我恢复了位姿。我正在使用 RANSAC。

然后，我以通常的方式纠正立体声对。

R1, R2, Pn1, Pn2, Q, _, _ = cv2.stereoRectify(K1, dcoeffs1, K2, dcoeffs2, 
                                                  img1.shape[::-1], R, t, 
                                                  flags=cv2.CALIB_ZERO_DISPARITY, 
                                                  alpha=-1)

    # Compute the rigid transform that OpenCV apply to world points (USEFUL LATER)
    # in order for the rectified reference camera to be K_new[I|0]
    tn_1 = np.zeros((3,1)) # Cameras are never translated in the rectification


    G1_rect = np.block([[R1, tn_1], [np.zeros((1,3)), 1.0]])
    maps1   =   cv2.initUndistortRectifyMap(K1, dcoeffs1, R1, Pn1, (1920,1080), cv2.CV_32FC1)
    maps2   =   cv2.initUndistortRectifyMap(K2, dcoeffs2, R2, Pn2, (1920,1080), cv2.CV_32FC1)
    img1_remap = cv2.remap(img1, maps1[0], maps1[1], cv2.INTER_LANCZOS4)
    img2_remap = cv2.remap(img2, maps2[0], maps2[1], cv2.INTER_LANCZOS4)

整改结果：【整改参考图】https://drive.google.com/open?id=10VfgXrXFO3_lYqtO9qJXr17Dc6F1PuXU 【另一张已整改】https://drive.google.com/open?id=13ZkeMiF5xEovGmX13LSQVaJ237hoJLX0

现在我调用一个函数来识别图像中的已知对象（目标）。

#Now call a function that recognize a known object in the images (target)
# Find target
target_corners, _ = dt.detectTarget(img_scene1, img_target, 0.5) # return 4 corners of the detected polygon
target_corners = target_corners[:,0,:]
# Compute mask for the target cutout:
target_mask = mp.maskPolygon(target_corners, img_scene1.shape[::-1]) # Output: mask of same dimension of the image

找到目标（请注意突出显示的角落）：[找到目标] https://imgur.com/QjYV8tp

然后我使用 StereoSGBM 计算视差图。我只对目标视差的计算感兴趣（我将掩盖所有其他点）。获得视差图并使用 stereoRectify 给出的 4x4 投影矩阵 Q，我执行视差图的 3d 重投影。

    # Compute disparity map
    # https://docs.opencv.org/3.3.1/d2/d85/classcv_1_1StereoSGBM.html
    window_size = 5
    min_disp = 16
    max_disp = 1024
    num_disp = max_disp-min_disp # Deve essere divisibile per 16!

    stereo = cv2.StereoSGBM_create(minDisparity = min_disp,
        numDisparities = num_disp,
        blockSize = window_size,
        P1 = 8*3*window_size**2,
        P2 = 32*3*window_size**2,
        disp12MaxDiff = 1,
        uniquenessRatio = 10,
        speckleWindowSize = 150,
        speckleRange = 2
    )

    print('Calcolo SGBM della disparità...')
    disp = stereo.compute(img_scene1, img_scene2).astype(np.float32) / 16.0

    target_disparity = target_mask*disp

    points = cv2.reprojectImageTo3D(target_disparity, Q)


    # DEBUG:
    cv2.namedWindow('scene1', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('scene1', 800,450)
    cv2.imshow('scene1', img_scene1)
    cv2.namedWindow('disparity', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('disparity', 800,450)
    cv2.imshow('disparity', (disp-min_disp)/num_disp)
    cv2.namedWindow('target_disparity', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('target_disparity', 800,450)
    cv2.imshow('target_disparity', target_mask*(disp-min_disp)/num_disp)
    cv2.waitKey()
    cv2.destroyAllWindows()


    # Obtain matrix of the target 3D points starting from disparity image obtained from reprojectImageTo3D()
    mask_disp = disp > disp.min()
    mask_inf = ~(np.isinf(points[:,:,0]) | np.isinf(points[:,:,1]) | np.isinf(points[:,:,2]))
    mask_nan = ~(np.isnan(points[:,:,0]) | np.isnan(points[:,:,1]) | np.isnan(points[:,:,2]))

    mask = mask_disp & mask_inf & mask_nan

    pts3D = points[mask]

现在，我已经 3d 重建了与目标相对应的图像区域。我注意到 OpenCv 在相机校正期间对世界点应用刚性变换，使得参考原始相机和新的（校正后的）参考相机具有相同的外在参数（R=eye(3) 和 t=[0,0,0 ]'）。事实上，在校正过程中，两个相机都必须旋转，我认为 OpenCV 只是将新相机带回一个新的参考，这样参考校正相机具有与原始相机相同的外部特性。但这意味着重建的 3d 点将在不是原始相机的世界参考的世界参考中表达！

因此，将逆刚性变换应用于 pts3D，我们在原始参考相机帧中获得了重建。（见代码）。

target3Dpts_hom = cv2.convertPointsToHomogeneous(target3Dpts)[:,0,:].T
    target3Dpts_hom = G.T @ target3Dpts_hom
    new_target3Dpts = cv2.convertPointsFromHomogeneous(target3Dpts_hom.T[:,np.newaxis,:])[:,0,:]

请注意，如果我不执行此操作，则通过其投影矩阵在原始相机上重新投影的 pt3D 将与目标点不对应！

通过重投影检查重建；现在，我可以重新投影 new_target3Dpts：让我介绍一下我调用的投影函数：

    def proj_dist(P, dcoeffs, M):

    import numpy as np
    import cv2

    K, R, t,_,_,_,_ = cv2.decomposeProjectionMatrix(P)
    rotv, _ = cv2.Rodrigues(R)

    # Projection. Returns a (N,2) shaped array
    m,_ = cv2.projectPoints(M,rotv,t[0:-1],K,dcoeffs)
    m = m.squeeze()

    return m

最后，重新投影：

    #P_kin = K_kin[eye(3),0] # Originals MPPs of two cameras
    #P_rpi = K_rpi[R,t]

    m0 = proj.proj_dist(P_kin,dcoeffs_kin,new_points).astype('int32')

    for (x, y) in m0:

        x = int(x)
        y= int(y)

        cv2.circle(img_kin, (x, y), 2, (255, 255, 0), 4)

    cv2.namedWindow('frame1', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('frame1', 800,450)
    cv2.imshow('frame1',img_kin)
    cv2.waitKey(0)

    m1 = proj.proj_dist(P_rpi,dcoeffs_rpi,new_points).astype('int32')
    img_rpi1 = img_rpi.copy()
    for (x, y) in m1:
        x = int(x)
        y = int(y)

    cv2.circle(img_rpi1, (x, y), 2, (255, 255, 0), 4)

    cv2.namedWindow('frame2', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('frame2', 800,450)
    cv2.imshow('frame2',img_rpi1)
    cv2.waitKey(0)

但是，虽然原始参考相机上的重投影点是正确的，但对于第二个来说，情况并非如此......这些点只是简单地翻译了，但我无法解释原因。

结果：[第一帧repj] https://imgur.com/S4lo9Wz [第二帧repj。错误] https://imgur.com/y4igaEI

有任何想法吗？我现在将包含所有代码。谢谢你。

标签： pythonopencv

python - reprojectImageto3d 后左右图像点重投影不一致

问题描述

解决方案

推荐阅读