objective-c - 将多通道音频缓冲区合并到一个 CMSampleBuffer

Question

我正在使用FFmpeg在我的macOS应用程序中访问RTSP流。

达到的目标：我创建了一个音调发生器，它创建单声道音频并返回一个 CMSampleBuffer。当视频的 fps 和音频采样率发生变化时，音调发生器用于测试我的音频管道。

目标：目标是将多通道音频缓冲区合并到单个 CMSampleBuffer 中。

音频数据生命周期：

AVCodecContext* audioContext = self.rtspStreamProvider.audioCodecContext;
        if (!audioContext) { return; }

        // Getting audio settings from FFmpegs audio context (AVCodecContext).
        int samplesPerChannel = audioContext->frame_size;
        int frameNumber = audioContext->frame_number;
        int sampleRate = audioContext->sample_rate;
        int fps = [self.rtspStreamProvider fps];

        int calculatedSampleRate = sampleRate / fps;

        // NSLog(@"\nSamples per channel = %i, frames = %i.\nSample rate = %i, fps = %i.\ncalculatedSampleRate = %i.", samplesPerChannel, frameNumber, sampleRate, fps, calculatedSampleRate);

        // Decoding the audio data from a encoded AVPacket into a AVFrame.
        AVFrame* audioFrame = [self.rtspStreamProvider readDecodedAudioFrame];
        if (!audioFrame) { return; }

        // Extracting my audio buffers from FFmpegs AVFrame.
        uint8_t* leftChannelAudioBufRef = audioFrame->data[0];
        uint8_t* rightChannelAudioBufRef = audioFrame->data[1];

        // Creating the CMSampleBuffer with audio data.
        CMSampleBufferRef leftSampleBuffer = [CMSampleBufferFactory createAudioSampleBufferUsingData:leftChannelAudioBufRef channelCount:1 framesCount:samplesPerChannel sampleRate:sampleRate];
//      CMSampleBufferRef rightSampleBuffer = [CMSampleBufferFactory createAudioSampleBufferUsingData:packet->data[1] channelCount:1 framesCount:samplesPerChannel sampleRate:sampleRate];

        if (!leftSampleBuffer) { return; }
        if (!self.audioQueue) { return; }
        if (!self.audioDelegates) { return; }

        // All audio consumers will receive audio samples via delegation. 
        dispatch_sync(self.audioQueue, ^{
            NSHashTable *audioDelegates = self.audioDelegates;
            for (id<AudioDataProviderDelegate> audioDelegate in audioDelegates)
            {
                [audioDelegate provider:self didOutputAudioSampleBuffer:leftSampleBuffer];
                // [audioDelegate provider:self didOutputAudioSampleBuffer:rightSampleBuffer];
            }
        });

CMSampleBuffer 包含音频数据创建：

import Foundation
import CoreMedia

@objc class CMSampleBufferFactory: NSObject
{

    @objc static func createAudioSampleBufferUsing(data: UnsafeMutablePointer<UInt8> ,
                                             channelCount: UInt32,
                                             framesCount: CMItemCount,
                                             sampleRate: Double) -> CMSampleBuffer? {

        /* Prepare for sample Buffer creation */
        var sampleBuffer: CMSampleBuffer! = nil
        var osStatus: OSStatus = -1
        var audioFormatDescription: CMFormatDescription! = nil

        var absd: AudioStreamBasicDescription! = nil
        let sampleDuration = CMTimeMake(value: 1, timescale: Int32(sampleRate))
        let presentationTimeStamp = CMTimeMake(value: 0, timescale: Int32(sampleRate))

        // NOTE: Change bytesPerFrame if you change the block buffer value types. Currently we are using double.
        let bytesPerFrame: UInt32 = UInt32(MemoryLayout<Float32>.size) * channelCount
        let memoryBlockByteLength = framesCount * Int(bytesPerFrame)

//      var acl = AudioChannelLayout()
//      acl.mChannelLayoutTag = kAudioChannelLayoutTag_Stereo

        /* Sample Buffer Block buffer creation */
        var blockBuffer: CMBlockBuffer?

        osStatus = CMBlockBufferCreateWithMemoryBlock(
            allocator: kCFAllocatorDefault,
            memoryBlock: nil,
            blockLength: memoryBlockByteLength,
            blockAllocator: nil,
            customBlockSource: nil,
            offsetToData: 0,
            dataLength: memoryBlockByteLength,
            flags: 0,
            blockBufferOut: &blockBuffer
        )

        assert(osStatus == kCMBlockBufferNoErr)

        guard let eBlock = blockBuffer else { return nil }

        osStatus = CMBlockBufferFillDataBytes(with: 0, blockBuffer: eBlock, offsetIntoDestination: 0, dataLength: memoryBlockByteLength)
        assert(osStatus == kCMBlockBufferNoErr)

        TVBlockBufferHelper.fillAudioBlockBuffer(blockBuffer,
                                                 audioData: data,
                                                 frames: Int32(framesCount))
        /* Audio description creations */

        absd = AudioStreamBasicDescription(
            mSampleRate: sampleRate,
            mFormatID: kAudioFormatLinearPCM,
            mFormatFlags: kLinearPCMFormatFlagIsPacked | kLinearPCMFormatFlagIsFloat,
            mBytesPerPacket: bytesPerFrame,
            mFramesPerPacket: 1,
            mBytesPerFrame: bytesPerFrame,
            mChannelsPerFrame: channelCount,
            mBitsPerChannel: 32,
            mReserved: 0
        )

        guard absd != nil else {
            print("\nCreating AudioStreamBasicDescription Failed.")
            return nil
        }

        osStatus = CMAudioFormatDescriptionCreate(allocator: kCFAllocatorDefault,
                                                  asbd: &absd,
                                                  layoutSize: 0,
                                                  layout: nil,
//                                                layoutSize: MemoryLayout<AudioChannelLayout>.size,
//                                                layout: &acl,
                                                  magicCookieSize: 0,
                                                  magicCookie: nil,
                                                  extensions: nil,
                                                  formatDescriptionOut: &audioFormatDescription)

        guard osStatus == noErr else {
            print("\nCreating CMFormatDescription Failed.")
            return nil
        }

        /* Create sample Buffer */
        var timmingInfo = CMSampleTimingInfo(duration: sampleDuration, presentationTimeStamp: presentationTimeStamp, decodeTimeStamp: .invalid)

        osStatus = CMSampleBufferCreate(allocator: kCFAllocatorDefault,
                                        dataBuffer: eBlock,
                                        dataReady: true,
                                        makeDataReadyCallback: nil,
                                        refcon: nil,
                                        formatDescription: audioFormatDescription,
                                        sampleCount: framesCount,
                                        sampleTimingEntryCount: 1,
                                        sampleTimingArray: &timmingInfo,
                                        sampleSizeEntryCount: 0, // Must be 0, 1, or numSamples.
            sampleSizeArray: nil, // Pointer ot Int. Don't know the size. Don't know if its bytes or bits?
            sampleBufferOut: &sampleBuffer)
        return sampleBuffer
    }

}

CMSampleBuffer 填充了来自 FFmpeg 数据的原始音频数据：

@import Foundation;
@import CoreMedia;

@interface BlockBufferHelper : NSObject

+(void)fillAudioBlockBuffer:(CMBlockBufferRef)blockBuffer
                  audioData:(uint8_t *)data
                     frames:(int)framesCount;



@end

#import "TVBlockBufferHelper.h"

@implementation BlockBufferHelper

+(void)fillAudioBlockBuffer:(CMBlockBufferRef)blockBuffer
                  audioData:(uint8_t *)data
                     frames:(int)framesCount
{
    // Possibly dev error.
    if (framesCount == 0) {
        NSAssert(false, @"\nfillAudioBlockBuffer/audioData/frames will not be able to fill an blockBuffer which has no frames.");
        return;
    }

    char *rawBuffer = NULL;

    size_t size = 0;

    OSStatus status = CMBlockBufferGetDataPointer(blockBuffer, 0, &size, NULL, &rawBuffer);
    if(status != noErr)
    {
        return;
    }

    memcpy(rawBuffer, data, framesCount);
}

@end

LEARNING Core AudioChris Adamson/Kevin Avila的书将我引向了多通道混音器。多通道混音器应具有 2-n 个输入和 1 个输出。我假设输出可能是缓冲区或可以放入以CMSampleBuffer供进一步使用的东西。

这个方向应该引导我去AudioUnits，AUGraph和AudioToolbox。我不了解所有这些课程以及它们如何协同工作。我在 SO 上找到了一些可以帮助我的代码片段，但它们中的大多数都使用AudioToolBox类并且没有使用CMSampleBuffers我需要的那么多。

还有另一种方法可以将音频缓冲区合并到一个新的缓冲区中吗？

使用 AudioToolBox 创建多声道混音器是正确的方向吗？

标签： objective-cmacosaudioffmpeg