opencv - 如何将 2D 面部地标转换为 3D 世界坐标?
问题描述
第一次在这里发帖,我会尽力描述问题。我正在尝试使用类似于此软件等精美应用程序的 opencv、dlib 和网络摄像头数据实时为 3D 虚拟角色的面部设置动画
按照此处的示例并使用网络摄像头,我可以获得屏幕空间中的实时 2D 面部地标数据以及头部姿势的 3D 平移数据的估计。但我真正想知道的是面部地标的估计 3D 世界坐标。例如,当头部向一侧倾斜 30 度时,嘴巴看起来就像演员用嘴巴发出“w”音一样。
谁能告诉我使用估计的头部姿势将这些 2D 面部标志转换为 3D 数据的策略是什么,或者这是否是这些实时面部动画应用程序如此昂贵的原因?
std::vector<cv::Point3d> object_pts;
std::vector<cv::Point2d> image_pts;
std::vector<cv::Point3d> reprojectsrc;
std::vector<cv::Point2d> reprojectdst;
cv::Mat rotation_vec;
cv::Mat rotation_mat;
cv::Mat translation_vec;
cv::Mat pose_mat;
cv::Mat euler_angle;
cv::Mat cam_matrix;
cv::Mat dist_coeffs;
cv::Mat out_intrinsics;
cv::Mat out_rotation;
cv::Mat out_translation;
int UpdateFace(dlib::cv_image<dlib::bgr_pixel> cimg, dlib::rectangle face)
{
// get facial landmarks
shape = predictor(cimg, face);
// draw facial landmarks
for (unsigned int i = 0; i < 68; ++i)
{
cv::circle(frame, cv::Point(shape.part(i).x(), shape.part(i).y()), 2, cv::Scalar(0, 0, 255), -1);
}
// clear data and fill in 2D ref points
image_pts.clear();
//#17 left brow left corner
image_pts.push_back(cv::Point2d(shape.part(17).x(), shape.part(17).y()));
//#21 left brow right corner
image_pts.push_back(cv::Point2d(shape.part(21).x(), shape.part(21).y()));
//#22 right brow left corner
image_pts.push_back(cv::Point2d(shape.part(22).x(), shape.part(22).y()));
//#26 right brow right corner
image_pts.push_back(cv::Point2d(shape.part(26).x(), shape.part(26).y()));
//#36 left eye left corner
image_pts.push_back(cv::Point2d(shape.part(36).x(), shape.part(36).y()));
//#39 left eye right corner
image_pts.push_back(cv::Point2d(shape.part(39).x(), shape.part(39).y()));
//#42 right eye left corner
image_pts.push_back(cv::Point2d(shape.part(42).x(), shape.part(42).y()));
//#45 right eye right corner
image_pts.push_back(cv::Point2d(shape.part(45).x(), shape.part(45).y()));
//#31 nose left corner
image_pts.push_back(cv::Point2d(shape.part(31).x(), shape.part(31).y()));
//#35 nose right corner
image_pts.push_back(cv::Point2d(shape.part(35).x(), shape.part(35).y()));
//#48 mouth left corner
image_pts.push_back(cv::Point2d(shape.part(48).x(), shape.part(48).y()));
//#54 mouth right corner
image_pts.push_back(cv::Point2d(shape.part(54).x(), shape.part(54).y()));
//#57 mouth central bottom corner
image_pts.push_back(cv::Point2d(shape.part(57).x(), shape.part(57).y()));
//#8 chin corner
image_pts.push_back(cv::Point2d(shape.part(8).x(), shape.part(8).y()));
// calculate the head pose
cv::solvePnP(object_pts, image_pts, cam_matrix, dist_coeffs, rotation_vec, translation_vec);
// reproject
cv::projectPoints(reprojectsrc, rotation_vec, translation_vec, cam_matrix, dist_coeffs, reprojectdst);
//draw 3d box around the actors head
cv::line(frame, reprojectdst[0], reprojectdst[1], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[1], reprojectdst[2], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[2], reprojectdst[3], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[3], reprojectdst[0], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[4], reprojectdst[5], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[5], reprojectdst[6], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[6], reprojectdst[7], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[7], reprojectdst[4], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[0], reprojectdst[4], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[1], reprojectdst[5], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[2], reprojectdst[6], cv::Scalar(0, 0, 255));
cv::line(frame, reprojectdst[3], reprojectdst[7], cv::Scalar(0, 0, 255));
//calculate euler angle to move avatars head
cv::Rodrigues(rotation_vec, rotation_mat);
cv::hconcat(rotation_mat, translation_vec, pose_mat);
cv::decomposeProjectionMatrix(pose_mat, out_intrinsics, out_rotation,
out_translation, cv::noArray(), cv::noArray(), cv::noArray(), euler_angle);
// calculate 3D transformation of facial landmarks to animate avatars facial features
// code needed here ...
}
解决方案
这不是一个容易解决的问题。
您可以将 3D 可变形模型作为起点。
https://cvssp.org/faceweb/3dmm/
然后,您可以在 Google Scholar 上查看所有最近引用该论文的论文,以找到最先进的技术。
希望有帮助。
推荐阅读
- reactjs - react-native 的 react-navigation:SwitchNavigator 在哪里?
- c# - 从 VB.NET 转换为 C# 后比较空值和可空值
- python - 使用 pathlib 将一个目录破坏性地重命名为另一个现有目录
- r - 更改 geom_text 的文本颜色
- laravel - Laravel - GroupBy 回调后的排序日期
- google-app-engine - 我最近使用谷歌应用引擎(灵活)部署了一个“你好,单词”烧瓶应用程序,它非常慢
- swift - Swift 操作员身份检查
- c# - 实体框架尝试添加已在数据库中的表的迁移
- ruby-on-rails - 将 JSON 转换为 jbuilder 语法,其中包含冒号的键
- haskell - 存在类型作为返回值