swift - 当 AVCaptureConnection videoOrientation 设置为纵向时，MLKit 人脸跟踪无法在视频流中找到人脸

Question

我正在尝试使用Metal / Scenekit渲染器进行一些前置摄像头面部跟踪（是的，重新发明 snapchat）

我Firebase用于面部跟踪

我有一个问题，设置AVCaptureConnectionvideoOrientation 导致没有检测到面部

public func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    do {
        var textures: [MTLTexture]!
        delegate?.metalCameraSession(self, didSendRawSampleBufferForFaceDetection: sampleBuffer)
        connection.videoOrientation = .portrait//portrait causes no faces to be detected
        connection.isVideoMirrored = true
        guard let imageBuffer = sampleBuffer.imageBuffer else {
            throw MetalCameraSessionError.failedToGetImageBuffer
        }
        switch pixelFormat {
        case .rgb:
            let textureRGB = try texture(sampleBuffer: imageBuffer, textureCache: textureCache)
            textures = [textureRGB]
        case .yCbCr:
            let textureY = try texture(sampleBuffer: imageBuffer, textureCache: textureCache, planeIndex: 0, pixelFormat: .r8Unorm)
            let textureCbCr = try texture(sampleBuffer: imageBuffer, textureCache: textureCache, planeIndex: 1, pixelFormat: .rg8Unorm)
            textures = [textureY, textureCbCr]
        }

        let timestamp = try self.timestamp(sampleBuffer: sampleBuffer)

        delegate?.metalCameraSession(self, didReceiveFrameAsTextures: textures, withTimestamp: timestamp, andImageBuffer :sampleBuffer)

    }
    catch let error as MetalCameraSessionError {
        self.handleError(error)
    }
    catch {
        /**
         * We only throw `MetalCameraSessionError` errors.
         */
    }
}

如果我不将其设置为纵向，一切都很好，但我的 MTLTexture 会旋转和拉伸

纹理创建如下：

private func texture(sampleBuffer: CVImageBuffer?, textureCache: CVMetalTextureCache?, planeIndex: Int = 0, pixelFormat: MTLPixelFormat = .bgra8Unorm) throws -> MTLTexture {
    guard let sampleBuffer = sampleBuffer else {
        throw MetalCameraSessionError.missingSampleBuffer
    }
    guard let textureCache = textureCache else {
        throw MetalCameraSessionError.failedToCreateTextureCache
    }


    let isPlanar = CVPixelBufferIsPlanar(sampleBuffer)
    let width = isPlanar ? CVPixelBufferGetWidthOfPlane(sampleBuffer, planeIndex) : CVPixelBufferGetWidth(sampleBuffer)
    let height = isPlanar ? CVPixelBufferGetHeightOfPlane(sampleBuffer, planeIndex) : CVPixelBufferGetHeight(sampleBuffer)

    var imageTexture: CVMetalTexture?

    let result = CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, sampleBuffer, nil, pixelFormat, width, height, planeIndex, &imageTexture)

    guard
        let unwrappedImageTexture = imageTexture,
        let texture = CVMetalTextureGetTexture(unwrappedImageTexture),
        result == kCVReturnSuccess
        else {
            throw MetalCameraSessionError.failedToCreateTextureFromImage
    }

    return texture
}

即使我在更改 videoOrientation 和创建金属纹理之前将样本缓冲区发送到人脸检测器，情况仍然如此（没有人脸跟踪）

标签： swiftavfoundationscenekitmetalfirebase-mlkit