ios - 将 ARFrame#capturedImage 转换为视图大小
问题描述
在 ARKit 中使用ARSessionDelegate
处理原始相机图像时...
func session(_ session: ARSession, didUpdate frame: ARFrame) {
guard let currentFrame = session.currentFrame else { return }
let capturedImage = currentFrame.capturedImage
debugPrint("Display size", UIScreen.main.bounds.size)
debugPrint("Camera frame resolution", CVPixelBufferGetWidth(capturedImage), CVPixelBufferGetHeight(capturedImage))
// ...
}
...如文件所述,相机图像数据与屏幕尺寸不匹配,例如,在 iPhone XI 上获取:
- 显示尺寸:375x812pt
- 相机分辨率:1920x1440px
现在有displayTransform(for:viewportSize:) API 将相机坐标转换为视图坐标。像这样使用 API 时:
let ciimage = CIImage(cvImageBuffer: capturedImage)
let transform = currentFrame.displayTransform(for: .portrait, viewportSize: UIScreen.main.bounds.size)
var transformedImage = ciimage.transformed(by: transform)
debugPrint("Transformed size", transformedImage.extent.size)
我得到的大小为 2340x1920,这似乎不正确,结果的纵横比应为 375:812(~0.46)。我在这里想念什么/使用此 API 将相机图像转换为“由 ARSCNView 显示”的图像的正确方法是什么?
(示例项目:ARKitCameraImage)
解决方案
事实证明这非常复杂,因为displayTransform(for:viewportSize)
需要标准化的图像坐标,看来您只能在纵向模式下翻转坐标,并且图像不仅需要转换,还需要裁剪。下面的代码对我有用。建议如何改进这一点将不胜感激。
guard let frame = session.currentFrame else { return }
let imageBuffer = frame.capturedImage
let imageSize = CGSize(width: CVPixelBufferGetWidth(imageBuffer), height: CVPixelBufferGetHeight(imageBuffer))
let viewPort = sceneView.bounds
let viewPortSize = sceneView.bounds.size
let interfaceOrientation : UIInterfaceOrientation
if #available(iOS 13.0, *) {
interfaceOrientation = self.sceneView.window!.windowScene!.interfaceOrientation
} else {
interfaceOrientation = UIApplication.shared.statusBarOrientation
}
let image = CIImage(cvImageBuffer: imageBuffer)
// The camera image doesn't match the view rotation and aspect ratio
// Transform the image:
// 1) Convert to "normalized image coordinates"
let normalizeTransform = CGAffineTransform(scaleX: 1.0/imageSize.width, y: 1.0/imageSize.height)
// 2) Flip the Y axis (for some mysterious reason this is only necessary in portrait mode)
let flipTransform = (interfaceOrientation.isPortrait) ? CGAffineTransform(scaleX: -1, y: -1).translatedBy(x: -1, y: -1) : .identity
// 3) Apply the transformation provided by ARFrame
// This transformation converts:
// - From Normalized image coordinates (Normalized image coordinates range from (0,0) in the upper left corner of the image to (1,1) in the lower right corner)
// - To view coordinates ("a coordinate space appropriate for rendering the camera image onscreen")
// See also: https://developer.apple.com/documentation/arkit/arframe/2923543-displaytransform
let displayTransform = frame.displayTransform(for: interfaceOrientation, viewportSize: viewPortSize)
// 4) Convert to view size
let toViewPortTransform = CGAffineTransform(scaleX: viewPortSize.width, y: viewPortSize.height)
// Transform the image and crop it to the viewport
let transformedImage = image.transformed(by: normalizeTransform.concatenating(flipTransform).concatenating(displayTransform).concatenating(toViewPortTransform)).cropped(to: viewPort)
推荐阅读
- assembly - 汇编器 - 将两个大的十六进制数相乘
- java - 将 For-Each-Loop 中来自 Retrofit2 调用的服务器响应保存到文件、数据库或新 List / ArrayList
- java - MediaPlayer - onCompletion 问题
- flutter - 在 webrtc 视频通话中响铃(颤振 webrtc)
- javascript - 取决于 URL 的条件渲染
- python - Pygame 告诉我图像不是文件对象
- firebase - 允许谷歌云计算访问firestore
- google-cloud-platform - GCP Terraform - 在没有 sudo 的情况下导入 ssh-keys
- c# - 项目出现延迟
- javascript - 无法将 FixedSizeList (react-window) 组件与 antd 传输组件集成