swift - 为什么 iPhone XS 在现场使用相机时 CPU 性能比 iPhone 6S Plus 差？

问题描述

我正在使用实时相机输出来更新 MTKView 上的 CIImage。我的主要问题是我有一个很大的负面性能差异，即旧 iPhone 比新 iPhone 获得更好的 CPU 性能，尽管我遇到的所有设置都是相同的。

这是一篇很长的文章，但我决定包含这些细节，因为它们可能对导致此问题的原因很重要。请让我知道我还可以包括什么。

下面，我的 captureOutput 函数带有两个调试布尔值，我可以在运行时打开和关闭它们。我用它来试图确定我的问题的原因。

applyLiveFilter - bool 是否使用 CIFilter 操作 CIImage。

updateMetalView - bool 是否更新 MTKView 的 CIImage。

// live output from camera
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection){

    /*              
     Create CIImage from camera.
     Here I save a few percent of CPU by using a function 
     to convert a sampleBuffer to a Metal texture, but
     whether I use this or the commented out code 
     (without captureOutputMTLOptions) does not have 
     significant impact. 
    */

    guard let texture:MTLTexture = convertToMTLTexture(sampleBuffer: sampleBuffer) else{
        return
    }

    var cameraImage:CIImage = CIImage(mtlTexture: texture, options: captureOutputMTLOptions)!

    var transform: CGAffineTransform = .identity

    transform = transform.scaledBy(x: 1, y: -1)

    transform = transform.translatedBy(x: 0, y: -cameraImage.extent.height)

    cameraImage = cameraImage.transformed(by: transform)

    /*
    // old non-Metal way of getting the ciimage from the cvPixelBuffer
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else
    {
        return
    }

    var cameraImage:CIImage = CIImage(cvPixelBuffer: pixelBuffer)
    */

    var orientation = UIImage.Orientation.right

    if(isFrontCamera){
        orientation = UIImage.Orientation.leftMirrored
    }

    // apply filter to camera image
    if debug_applyLiveFilter {
        cameraImage = self.applyFilterAndReturnImage(ciImage: cameraImage, orientation: orientation, currentCameraRes:currentCameraRes!)
    }

    DispatchQueue.main.async(){
        if debug_updateMetalView {
            self.MTLCaptureView!.image = cameraImage
        }
    }

}

下面是两部手机切换上述不同布尔组合的结果图表：

即使没有 Metal view 的 CIIMage 更新，也没有应用过滤器，iPhone XS 的 CPU 比 iPhone 6S Plus 高 2%，这不是很大的开销，但让我怀疑相机捕捉的方式在设备之间是不同的.

我的 AVCaptureSession 的预设在两部手机之间设置相同（AVCaptureSession.Preset.hd1280x720）
从 captureOutput 创建的 CIImage 在两部手机之间的大小（范围）相同。

是否需要在这两部手机 AVCaptureDevice 的设置（包括 activeFormat 属性）之间手动设置任何设置，以使它们在设备之间相同？

我现在的设置是：

if let captureDevice = AVCaptureDevice.default(for:AVMediaType.video) {
    do {
        try captureDevice.lockForConfiguration()
            captureDevice.isSubjectAreaChangeMonitoringEnabled = true
            captureDevice.focusMode = AVCaptureDevice.FocusMode.continuousAutoFocus
            captureDevice.exposureMode = AVCaptureDevice.ExposureMode.continuousAutoExposure
        captureDevice.unlockForConfiguration()

    } catch {
        // Handle errors here
        print("There was an error focusing the device's camera")
    }
}

我的 MTKView 是基于 Simon Gladman 编写的代码，对性能进行了一些编辑，并在使用 Apple 建议的 Core Animation 将渲染放大到屏幕宽度之前对其进行了缩放。

class MetalImageView: MTKView
{
    let colorSpace = CGColorSpaceCreateDeviceRGB()

    var textureCache: CVMetalTextureCache?

    var sourceTexture: MTLTexture!

    lazy var commandQueue: MTLCommandQueue =
        {
            [unowned self] in

            return self.device!.makeCommandQueue()
            }()!

    lazy var ciContext: CIContext =
        {
            [unowned self] in

            return CIContext(mtlDevice: self.device!)
            }()


    override init(frame frameRect: CGRect, device: MTLDevice?)
    {
        super.init(frame: frameRect,
                   device: device ?? MTLCreateSystemDefaultDevice())



        if super.device == nil
        {
            fatalError("Device doesn't support Metal")
        }

        CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, self.device!, nil, &textureCache)

        framebufferOnly = false

        enableSetNeedsDisplay = true

        isPaused = true

        preferredFramesPerSecond = 30
    }

    required init(coder: NSCoder)
    {
        fatalError("init(coder:) has not been implemented")
    }

    // The image to display
    var image: CIImage?
    {
        didSet
        {
            setNeedsDisplay()
        }
    }

    override func draw(_ rect: CGRect)
    {
        guard var
            image = image,
            let targetTexture:MTLTexture = currentDrawable?.texture else
        {
            return
        }

        let commandBuffer = commandQueue.makeCommandBuffer()

        let customDrawableSize:CGSize = drawableSize

        let bounds = CGRect(origin: CGPoint.zero, size: customDrawableSize)

        let originX = image.extent.origin.x
        let originY = image.extent.origin.y

        let scaleX = customDrawableSize.width / image.extent.width
        let scaleY = customDrawableSize.height / image.extent.height

        let scale = min(scaleX*IVScaleFactor, scaleY*IVScaleFactor)
        image = image
            .transformed(by: CGAffineTransform(translationX: -originX, y: -originY))
            .transformed(by: CGAffineTransform(scaleX: scale, y: scale))


        ciContext.render(image,
                         to: targetTexture,
                         commandBuffer: commandBuffer,
                         bounds: bounds,
                         colorSpace: colorSpace)


        commandBuffer?.present(currentDrawable!)

        commandBuffer?.commit()

    }

}

我的 AVCaptureSession (captureSession) 和 AVCaptureVideoDataOutput (videoOutput) 设置如下：

func setupCameraAndMic(){
    let backCamera = AVCaptureDevice.default(for:AVMediaType.video)

    var error: NSError?
    var videoInput: AVCaptureDeviceInput!
    do {
        videoInput = try AVCaptureDeviceInput(device: backCamera!)
    } catch let error1 as NSError {
        error = error1
        videoInput = nil
        print(error!.localizedDescription)
    }

    if error == nil &&
        captureSession!.canAddInput(videoInput) {

        guard CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, MetalDevice, nil, &textureCache) == kCVReturnSuccess else {
            print("Error: could not create a texture cache")
            return
        }

        captureSession!.addInput(videoInput)            

        setDeviceFrameRateForCurrentFilter(device:backCamera)

        stillImageOutput = AVCapturePhotoOutput()

        if captureSession!.canAddOutput(stillImageOutput!) {
            captureSession!.addOutput(stillImageOutput!)

            let q = DispatchQueue(label: "sample buffer delegate", qos: .default)
            videoOutput.setSampleBufferDelegate(self, queue: q)
            videoOutput.videoSettings = [
                kCVPixelBufferPixelFormatTypeKey as AnyHashable as! String: NSNumber(value: kCVPixelFormatType_32BGRA),
                kCVPixelBufferMetalCompatibilityKey as String: true
            ]
            videoOutput.alwaysDiscardsLateVideoFrames = true

            if captureSession!.canAddOutput(videoOutput){
                captureSession!.addOutput(videoOutput)
            }

            captureSession!.startRunning()

        }

    }

    setDefaultFocusAndExposure()
}

视频和麦克风记录在两个单独的流中。由于我的重点是实时摄像机输出的性能，因此省略了有关麦克风和录制视频的详细信息。

更新- 我在 GitHub 上有一个简化的测试项目，可以更轻松地测试我遇到的问题：https ://github.com/PunchyBass/Live-Filter-test-project

标签： swiftiphoneperformancemetalavcapturesession

swift - 为什么 iPhone XS 在现场使用相机时 CPU 性能比 iPhone 6S Plus 差？

问题描述

解决方案

推荐阅读