ios - Swift - 使用 AVPlayerItemVideoOutput 访问解码帧：不调用 outputMediaDataWillChange
问题描述
我有一个应用程序可以播放从用户库中选择的视频。我打算让应用程序最终能够做的是在视频上渲染一个叠加层（当它正在播放时），然后将结果输出到一个新的媒体文件。为此，我需要捕获解码的帧，以便在视频播放结束后渲染此叠加层并输出到文件。
这是我第一个使用 AVFoundation 的应用程序，我花了一两天时间试图通过 google 和 Apple 文档找出如何实现这一点，我认为我在AVPlayerItemVideoOutput对象中有一些东西。但是，委托回调永远不会执行。
我发现AVPlayerItemVideoOutput必须在is in status之后创建。因此，在我的初始化程序中，我向 AVPlayerItem 添加了一个观察者以观察其状态。AVPlayerItemreadyToPlayPlayerUIView
init(frame: CGRect, url: Binding<URL?>) {
_url = url
// Setup the player
player = AVPlayer(url: url.wrappedValue!)
super.init(frame: frame)
playerLayer.player = player
playerLayer.videoGravity = .resizeAspect
layer.addSublayer(playerLayer)
//displayLink = CADisplayLink()
// Setup looping
player.actionAtItemEnd = .none
NotificationCenter.default.addObserver(self,
selector: #selector(playerItemDidReachEnd(notification:)),
name: .AVPlayerItemDidPlayToEndTime,
object: player.currentItem)
player.currentItem?.addObserver(self, forKeyPath: #keyPath(AVPlayerItem.status), options: [.old, .new], context: nil)
// Start the movie
player.play()
}
我在中间创建了一个 CADisplayLink - 注释掉 - 因为我看到它可以以某种方式用于此目的，但不完全确定它应该如何或做什么。还担心它从显示的视频中获取帧而不是从我想要的实际解码的视频帧中获取帧的名称。
当状态第一次设置readyToPlay为时，我创建并添加AVPlayerItemVideoOutput.
override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) {
if let item = object as? AVPlayerItem {
if item.status == AVPlayerItem.Status.readyToPlay && item.outputs.count == 0 {
let settings = [ String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_24RGB ]
let output = AVPlayerItemVideoOutput(pixelBufferAttributes: settings)
output.setDelegate(PlayerOutput(output: output), queue: DispatchQueue(label: ""))
player.currentItem?.add(output)
}
}
}
在委托上PlayerOutput，我希望在新框架可用时收到通知。此时我将访问AVPlayerItemVideoOutput对象以访问像素缓冲区。
class PlayerOutput : NSObject, AVPlayerItemOutputPullDelegate {
func outputMediaDataWillChange(_ sender: AVPlayerItemOutput) {
let videoOutput = sender as! AVPlayerItemVideoOutput
let newPixelBuff = videoOutput.hasNewPixelBuffer(forItemTime: CMTime(seconds: 1, preferredTimescale: 10))
}
}
但是，永远不会进行此回调。我在代码中设置了一个断点，它永远不会被命中。根据 AVFoundation 中其他地方的命名和类似代码，我假设每个新帧都会被命中，因此我可以访问缓冲区中的帧，但我没有看到任何事情发生。我有什么遗漏或做错了吗？
我有一种感觉，我并没有完全正确地使用/理解这些类以及它们的用途，但它在命名法上与AVCaptureVideoDataOutput我设法在应用程序的其他地方成功实现的等类相似，它们似乎并不工作完全一样。很难找到任何我想用 AVPlayer 做的例子。
编辑：当前代码的工作示例：
import SwiftUI
import AVFoundation
struct CustomCameraPhotoView: View {
@State private var image: Image?
@State private var showingCustomCamera = false
@State private var showImagePicker = false
@State private var inputImage: UIImage?
@State private var url: URL?
var body: some View {
ZStack {
if url != nil
{
PlayerView(url: $url)
}
else
{
Button(action: {
self.showImagePicker = true
}) {
Text("Select a Video").foregroundColor(.white).font(.headline)
}
}
}.edgesIgnoringSafeArea(.all)
.sheet(isPresented: $showImagePicker,
onDismiss: loadImage) {
PhotoCaptureView(showImagePicker: self.$showImagePicker, image: self.$image, url: self.$url)
}.edgesIgnoringSafeArea(.leading).edgesIgnoringSafeArea(.trailing)
}
func loadImage() {
guard let inputImage = inputImage else { return }
image = Image(uiImage: inputImage)
}
}
struct PlayerView: UIViewControllerRepresentable {
@Binding var url: URL?
func updateUIViewController(_ uiView: UIViewController, context: UIViewControllerRepresentableContext<PlayerView>) {
}
func makeCoordinator() -> PlayerCoordinator{
//Make Coordinator which will commnicate with the ImagePickerViewController
PlayerCoordinator()
}
func makeUIViewController(context: Context) -> UIViewController {
let view = PlayerUIView(frame: .zero, url: $url)
let controller = PlayerController()
controller.view = view
return controller
}
}
class PlayerCoordinator : NSObject, UINavigationControllerDelegate {
}
class PlayerController: UIViewController {
override var shouldAutorotate: Bool {
return false
}
override var supportedInterfaceOrientations: UIInterfaceOrientationMask {
return .all
}
}
class PlayerUIView: UIView {
private let playerLayer = AVPlayerLayer()
private var playerOutput = PlayerOutput()
private let _myVideoOutputQueue = DispatchQueue(label: "VideoFrames", qos: .background, attributes: .concurrent, autoreleaseFrequency: .workItem, target: nil)
var displayLink: CADisplayLink?
var player: AVPlayer
@Binding var url: URL?
required init?(coder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
init(frame: CGRect, url: Binding<URL?>) {
_url = url
// Setup the player
player = AVPlayer(url: url.wrappedValue!)
super.init(frame: frame)
playerLayer.player = player
playerLayer.videoGravity = .resizeAspect
layer.addSublayer(playerLayer)
let settings = [ String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_32BGRA ]
let output = AVPlayerItemVideoOutput(pixelBufferAttributes: settings)
output.setDelegate(self.playerOutput, queue: self._myVideoOutputQueue)
player.currentItem?.add(output)
//displayLink = CADisplayLink()
// Setup looping
player.actionAtItemEnd = .none
NotificationCenter.default.addObserver(self,
selector: #selector(playerItemDidReachEnd(notification:)),
name: .AVPlayerItemDidPlayToEndTime,
object: player.currentItem)
player.currentItem?.addObserver(self, forKeyPath: #keyPath(AVPlayerItem.status), options: [.old, .new], context: nil)
// Start the movie
player.play()
}
override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) {
if let item = object as? AVPlayerItem {
if item.status == AVPlayerItem.Status.readyToPlay && item.outputs.count == 0 {
}
}
}
@objc
func playerItemDidReachEnd(notification: Notification) {
self.url = nil
}
override func layoutSubviews() {
super.layoutSubviews()
playerLayer.frame = bounds
}
class PlayerOutput : NSObject, AVPlayerItemOutputPullDelegate {
func outputMediaDataWillChange(_ sender: AVPlayerItemOutput) {
let videoOutput = sender as! AVPlayerItemVideoOutput
let newPixelBuff = videoOutput.hasNewPixelBuffer(forItemTime: CMTime(seconds: 1, preferredTimescale: 10))
}
}
}
struct ImagePicker : UIViewControllerRepresentable {
@Binding var isShown : Bool
@Binding var image : Image?
@Binding var url : URL?
func updateUIViewController(_ uiViewController: UIImagePickerController, context: UIViewControllerRepresentableContext<ImagePicker>)
{
//Update UIViewcontrolleer Method
}
func makeCoordinator() -> ImagePickerCoordinator{
//Make Coordinator which will commnicate with the ImagePickerViewController
ImagePickerCoordinator(isShown: $isShown, image: $image, url: $url)
}
func makeUIViewController(context: UIViewControllerRepresentableContext<ImagePicker>) -> UIImagePickerController
{
let picker = UIImagePickerController()
picker.sourceType = .photoLibrary
picker.delegate = context.coordinator
picker.mediaTypes = ["public.movie"]
picker.videoQuality = .typeHigh
return picker
}
}
class ImagePickerCoordinator : NSObject, UINavigationControllerDelegate, UIImagePickerControllerDelegate{
@Binding var isShown : Bool
@Binding var image : Image?
@Binding var url: URL?
init(isShown : Binding<Bool>, image: Binding<Image?>, url: Binding<URL?>) {
_isShown = isShown
_image = image
_url = url
}
//Selected Image
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) {
let uiImage = info[UIImagePickerController.InfoKey.mediaURL] as! URL
url = uiImage
//image = Image(uiImage: uiImage)
isShown = false
}
//Image selection got cancel
func imagePickerControllerDidCancel(_ picker: UIImagePickerController) {
isShown = false
}
}
struct PhotoCaptureView: View {
@Binding var showImagePicker : Bool
@Binding var image : Image?
@Binding var url : URL?
var body: some View {
ImagePicker(isShown: $showImagePicker, image: $image, url: $url)
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
CustomCameraPhotoView()
}
}
解决方案
编辑：你猜对了，outputMediaDataWillChange它是一个完全不同的野兽AVCaptureVideoDataOutput。它不是每帧调用的，而是表示一些玩家状态的变化（我有点不清楚究竟是什么情况）。（至少）有两个选项用于读取和修改视频输出。
- 您可以添加一个
videoComposition到当前playerItem您可以使用CoreImage 工具来调整视频（例如添加一个覆盖）。在这个例子中，我对输出应用了一个简单的模糊过滤器，但 CoreImage 允许你做更复杂的事情。
let blurFilter = CIFilter(name: "CIGaussianBlur")
if let playerItem = player.currentItem {
let asset = playerItem.asset
playerItem.videoComposition = AVMutableVideoComposition(asset: asset) { (filteringRequest) in
let source = filteringRequest.sourceImage
blurFilter?.setValue(source, forKey: kCIInputImageKey)
filteringRequest.finish(with: blurFilter?.outputImage ?? source, context: nil)
}
}
- 如果您需要实际的像素缓冲区，那么您在正确的轨道上使用
CADisplayLink. 显示链接允许您将操作与显示刷新率同步。您可以通过以下方式从视频输出中抓取帧：
lazy var displayLink: CADisplayLink = CADisplayLink(target: self, selector: #selector(displayLinkDidRefresh(link:)))
init(frame: CGRect, url: Binding<URL?>) {
...
// activate the displayLink
displayLink.add(to: .main, forMode: .common)
...
}
@objc func displayLinkDidRefresh(link: CADisplayLink) {
guard let videoOutput = self.videoOutput else { return }
let itemTime = player.currentTime()
if videoOutput.hasNewPixelBuffer(forItemTime: itemTime) {
var presentationItemTime: CMTime = .zero
if let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: itemTime, itemTimeForDisplay: &presentationItemTime) {
// process the pixelbuffer here
}
}
}
完整的最小示例：
import SwiftUI
import AVFoundation
struct CustomCameraPhotoView: View {
@State private var image: Image?
@State private var showingCustomCamera = false
@State private var showImagePicker = false
@State private var inputImage: UIImage?
@State private var url: URL?
var body: some View {
ZStack {
if url != nil {
PlayerView(url: $url)
} else {
Button(action: {
self.showImagePicker = true
}) {
Text("Select a Video")
.font(.headline)
}
}
} .edgesIgnoringSafeArea(.all)
.sheet(isPresented: $showImagePicker,
onDismiss: loadImage) {
PhotoCaptureView(showImagePicker: self.$showImagePicker, image: self.$image, url: self.$url)
}
.edgesIgnoringSafeArea(.leading).edgesIgnoringSafeArea(.trailing)
}
func loadImage() {
guard let inputImage = inputImage else { return }
image = Image(uiImage: inputImage)
}
}
struct PlayerView: UIViewControllerRepresentable {
@Binding var url: URL?
func updateUIViewController(_ uiView: UIViewController, context: UIViewControllerRepresentableContext<PlayerView>) {
}
func makeUIViewController(context: Context) -> UIViewController {
let view = PlayerUIView(frame: .zero, url: $url)
let controller = PlayerController()
controller.view = view
return controller
}
}
class PlayerController: UIViewController {
override var shouldAutorotate: Bool { false }
override var supportedInterfaceOrientations: UIInterfaceOrientationMask { .all }
}
class PlayerUIView: UIView {
private let playerLayer = AVPlayerLayer()
private let _myVideoOutputQueue = DispatchQueue(label: "VideoFrames", qos: .background, attributes: .concurrent, autoreleaseFrequency: .workItem, target: nil)
lazy var displayLink: CADisplayLink = CADisplayLink(target: self, selector: #selector(displayLinkDidRefresh(link:)))
var player: AVPlayer
var videoOutput: AVPlayerItemVideoOutput
@Binding var url: URL?
required init?(coder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
init(frame: CGRect, url: Binding<URL?>) {
_url = url
// Setup the player
player = AVPlayer(url: url.wrappedValue!)
let settings = [ String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_32BGRA ]
let output = AVPlayerItemVideoOutput(pixelBufferAttributes: settings)
self.videoOutput = output
super.init(frame: frame)
playerLayer.player = player
playerLayer.videoGravity = .resizeAspect
layer.addSublayer(playerLayer)
attachVideoComposition()
player.currentItem?.add(output)
displayLink.add(to: .main, forMode: .common)
// Start the movie
player.play()
}
private func attachVideoComposition() {
let blurFilter = CIFilter(name: "CIGaussianBlur")
if let playerItem = player.currentItem {
let asset = playerItem.asset
playerItem.videoComposition = AVMutableVideoComposition(asset: asset) { (filteringRequest) in
let source = filteringRequest.sourceImage
blurFilter?.setValue(source, forKey: kCIInputImageKey)
// Apply CoreImage provessing here
filteringRequest.finish(with: blurFilter?.outputImage ?? source, context: nil)
}
}
}
override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) {
if let item = object as? AVPlayerItem {
if item.status == AVPlayerItem.Status.readyToPlay && item.outputs.count == 0 {
}
}
}
override func layoutSubviews() {
super.layoutSubviews()
playerLayer.frame = bounds
}
@objc func displayLinkDidRefresh(link: CADisplayLink) {
let itemTime = player.currentTime()
if videoOutput.hasNewPixelBuffer(forItemTime: itemTime) {
var presentationItemTime: CMTime = .zero
if let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: itemTime, itemTimeForDisplay: &presentationItemTime) {
// process the pixelbuffer here
print(pixelBuffer)
}
}
}
}
struct ImagePicker : UIViewControllerRepresentable {
@Binding var isShown : Bool
@Binding var image : Image?
@Binding var url : URL?
func updateUIViewController(_ uiViewController: UIImagePickerController, context: UIViewControllerRepresentableContext<ImagePicker>) {
//Update UIViewcontrolleer Method
}
func makeCoordinator() -> ImagePickerCoordinator{
ImagePickerCoordinator(isShown: $isShown, image: $image, url: $url)
}
func makeUIViewController(context: UIViewControllerRepresentableContext<ImagePicker>) -> UIImagePickerController {
let picker = UIImagePickerController()
picker.sourceType = .photoLibrary
picker.delegate = context.coordinator
picker.mediaTypes = ["public.movie"]
picker.videoQuality = .typeHigh
return picker
}
}
class ImagePickerCoordinator : NSObject, UINavigationControllerDelegate, UIImagePickerControllerDelegate {
@Binding var isShown : Bool
@Binding var image : Image?
@Binding var url: URL?
init(isShown : Binding<Bool>, image: Binding<Image?>, url: Binding<URL?>) {
_isShown = isShown
_image = image
_url = url
}
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) {
url = info[UIImagePickerController.InfoKey.mediaURL] as? URL
isShown = false
}
func imagePickerControllerDidCancel(_ picker: UIImagePickerController) {
isShown = false
}
}
struct PhotoCaptureView: View {
@Binding var showImagePicker : Bool
@Binding var image : Image?
@Binding var url : URL?
var body: some View {
ImagePicker(isShown: $showImagePicker, image: $image, url: $url)
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
CustomCameraPhotoView()
}
}
原始答案：
代表总是（？）定义为弱成员。请参阅文档。在调用委托之前，您的PlayerOutput对象将退出范围。使PlayerOutput某些对象的成员在播放期间处于活动状态，并且您的代码应该按原样工作。
推荐阅读
- php - 每页的WordPress自定义链接并显示它
- php - 上交时的PERMISSION_DENIED课堂api，即使课程作业也是使用课堂api创建的
- javascript - 从 spring 控制器动态创建 HTML 文本框
- express - 如何使用车把编写带有条件的 {{#each}} 语句，是否可以
- html - 尝试在 Angular 中显示后端发送的 SVG
- asp.net-core - Cookie 不会从前端发送到后端
- java - 使用 Java/Spring 连接到 Azure 事件网格和服务总线
- python - 该程序没有正确比较两个数字
- python - SQLite3 和多线程？最好的行动方案是什么？
- c# - EventCounters 是从 .NET 库中公开指标的标准方法吗？