ios - 音频检测器可以在设备上工作，但不能在模拟器上工作......和准确性

Question

嗨，萨雷姆

背景

我有一个应用程序可以检测到有人说“Hi Sarem”是一种电子锁。我想做一些像“Hi Siri”这样的东西，但既然是这样，我就去做了一些不同的事情，比如“Hi Sarem”。

执行

该代码从麦克风采样音频，拟合 FFT，然后检查三个连续频率，因此如果您例如吹口哨或在钢琴上弹奏正确的三个音符，则可以触发它。这些频率需要在一定时间内相互触发，并且可以使用滑块进行配置。该代码包含设置时间和公差等所需的参数。三个滑块代表“Hi-Sa-rem”中的三个“音符”。

用户界面

此处的图像给出了 UI 的概念。当检测到相关频率时，子弹会变成红色，一旦检测到整个序列，大的就会变成红色。顶部的滑块充当监视器，持续监视“听到”的频率，因此您可以使用它来校准音符。

问题

我对此有一些问题。准确性是一个重要因素，但不是主要因素。（我想如果我有一个更可怕的妈妈，这可能会更准确，也可以在午餐时间完成，但那是另一个故事......）

所以这里是 - 主要问题。

这在设备上运行良好，但在模拟器上我在日志中得到以下内容

2020-07-26 18:47:13.543219+0200 HiSarem[68826:1238118] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x600000788320> F8BB1C28-BAE8-11D6-9C31-00039315CD46
2020-07-26 18:47:13.575866+0200 HiSarem[68826:1238118] No exclusivity (null)

我怀疑这与访问权限有关，但我不确定。我到处找我知道的，但对我来说，错误会抱怨工厂没有注册是没有意义的。另外，为什么它在设备上而不是在模拟器上工作？现在我确实打印出我无法获得对设备的独占访问权限，但即使没有请求或锁定麦克风，我仍然会遇到问题。

代码

这来自单个视图应用程序ViewController将提供的默认值，我确实描述了 UI 是如何连接到它的。因此，您应该能够将其简单地粘贴到项目中并在需要时运行它。这是一个测试项目，并不完善，但本着 MRE 的精神，您拥有所有代码。

#import <AVKit/AVKit.h>
#import <Accelerate/Accelerate.h>

#import "ViewController.h"

// Amplitute threshold
#define THRESHOLD    500

// Maximum frequency
#define MAXFREQ     7000

// Tolerance (% so 0.1 is 10%)
#define TOL          0.1

// Reset if no match within so many millis
#define RESETMIL    1500
#define BIGRESETMIL 5000

@interface ViewController () < AVCaptureAudioDataOutputSampleBufferDelegate >

@property (weak, nonatomic) IBOutlet UISlider  * monitorSlider;
@property (weak, nonatomic) IBOutlet UISlider  * phrase1Slider;
@property (weak, nonatomic) IBOutlet UISlider  * phrase2Slider;
@property (weak, nonatomic) IBOutlet UISlider  * phrase3Slider;

@property (weak, nonatomic) IBOutlet UILabel   * phrase1Label;
@property (weak, nonatomic) IBOutlet UILabel   * phrase2Label;
@property (weak, nonatomic) IBOutlet UILabel   * phrase3Label;
@property (weak, nonatomic) IBOutlet UILabel   * successLabel;

@property (nonatomic)         BOOL               busy;
@property (nonatomic, strong) AVCaptureSession * avSession;
@property (nonatomic, strong) AVCaptureInput   * avInput;
@property (nonatomic, strong) AVCaptureDevice  * avDevice;
@property (nonatomic, strong) AVCaptureOutput  * avOutput;

@property (nonatomic) double   prevF;
@property (nonatomic) NSDate * prevTime;

@end

@implementation ViewController

+ ( NSString * ) offText
{
    return @"⚫️";
}

+ ( NSString * ) onText
{
    return @"";
}

// See if we can turn on for a given frequency
- ( BOOL ) turnOn:( double ) f
         want:( double ) w
{
    double wLo = w * ( 1 - TOL );
    double wHi = w * ( 1 + TOL );

    return self.prevF < wLo && f >= wLo && f <= wHi;
}

// Update the value
- ( void ) measure:( int    ) s
         n:( int    ) n
{
    // Convert
    double f = 44100.0 * s / n;

    if ( f <= MAXFREQ )
    {
        self.monitorSlider.value = f;

        // See where we are with the sliders
        if ( [self.phrase1Label.text isEqualToString:ViewController.offText] )
        {
            // See if we can turn on 1
            if ( [self turnOn:f want:self.phrase1Slider.value] )
            {
                self.phrase1Label.text = ViewController.onText;

                // Match
                self.prevTime = NSDate.date;
            }
        }
        else if ( [self.phrase2Label.text isEqualToString:ViewController.offText] )
        {
            // See if we can turn on 2
            if ( [self turnOn:f want:self.phrase2Slider.value] )
            {
                self.phrase2Label.text = ViewController.onText;

                // Match
                self.prevTime = NSDate.date;
            }
        }
        else if ( [self.phrase3Label.text isEqualToString:ViewController.offText] )
        {
            // See if we can turn on 3
            if ( [self turnOn:f want:self.phrase3Slider.value] )
            {
                self.phrase3Label.text = ViewController.onText;
                self.successLabel.text = ViewController.onText;

                // Big match
                self.prevTime = NSDate.date;
            }
        }
    }

    // Reset if we do not get a match fast enough
    if ( self.prevTime )
    {
        NSTimeInterval d = [NSDate.date timeIntervalSinceDate:self.prevTime] * 1000;

        if ( d > RESETMIL )
        {
            self.phrase1Label.text = ViewController.offText;
            self.phrase2Label.text = ViewController.offText;
            self.phrase3Label.text = ViewController.offText;
        }
        if ( d > BIGRESETMIL )
        {
            self.successLabel.text = ViewController.offText;
        }
    }
}

- ( void ) viewDidLoad
{
    super.viewDidLoad;
}

- ( void ) viewDidAppear:(BOOL)animated
{
    [super viewDidAppear:animated];

    if ( self.requestPermission )
    {
        self.startCapture;
    }
}

- ( void ) viewWillDisappear:(BOOL)animated
{
    [super viewWillDisappear:animated];

    if ( self.avSession )
    {
        self.avSession.stopRunning;
        self.avSession = nil;
    }
}

- ( BOOL ) requestPermission
{
    if ( AVAudioSession.sharedInstance.recordPermission == AVAudioSessionRecordPermissionGranted )
    {
        return YES;
    }
    else if ( AVAudioSession.sharedInstance.recordPermission == AVAudioSessionRecordPermissionDenied )
    {
        UIAlertController * alert = [UIAlertController alertControllerWithTitle:@"No ears"
                                        message:@"I can not hear you - please change it quickly"
                                     preferredStyle:UIAlertActionStyleDefault];

        [alert addAction:[UIAlertAction actionWithTitle:@"Apologies"
                              style:UIAlertActionStyleDefault
                            handler:nil]];

        [self presentViewController:alert
                   animated:YES
                 completion:nil];

        return NO;
    }
    else
    {
        [AVAudioSession.sharedInstance requestRecordPermission:^ ( BOOL granted ) {

            if ( granted )
            {
                self.startCapture;
            }
            
        }];

        return NO;
    }
}

- ( void ) startCapture
{
    if ( ! self.busy )
    {
        self.busy = YES;
        
        // Create the capture session.
        NSError          * avErr;
        AVCaptureSession * captureSession = [[AVCaptureSession alloc] init];
        
        // Default anyhow
        captureSession.sessionPreset = AVCaptureSessionPresetHigh;

        // Lookup the default audio device.
        AVCaptureDevice * audioDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeAudio];

        if ( [audioDevice lockForConfiguration: & avErr] )
        {
            // Wrap the audio device in a capture device input.
            AVCaptureDeviceInput * audioInput = [AVCaptureDeviceInput deviceInputWithDevice:audioDevice
                                                  error: & avErr];
            
            audioDevice.unlockForConfiguration;

            if ( audioInput )
            {
                // If the input can be added, add it to the session.
                if ( [captureSession canAddInput:audioInput] )
                {
                    [captureSession addInput:audioInput];
                    
                    AVCaptureAudioDataOutput * audioOutput = [[AVCaptureAudioDataOutput alloc] init];
                    
                    if ( [captureSession canAddOutput:audioOutput] )
                    {
                        [audioOutput setSampleBufferDelegate:self
                                           queue:dispatch_queue_create ( "ears", NULL )];
                        [captureSession addOutput:audioOutput];

                        // Do on background
                        dispatch_async ( dispatch_queue_create ( "spotty", NULL ), ^ {
                            
                            NSLog ( @"Come to papa" );
                            captureSession.startRunning;
                            
                            // Done
                            dispatch_async ( dispatch_get_main_queue (), ^ {
                                
                                self.busy      = NO;
                                self.avSession = captureSession;
                                self.avDevice  = audioDevice;
                                self.avInput   = audioInput;
                                self.avOutput  = audioOutput;
                                
                            } );
                        } );
                    }
                    else
                    {
                        NSLog ( @"Not today : add output" );
                        self.busy = NO;
                    }
                }
                else
                {
                    NSLog( @"Sorry : add input" );
                    self.busy = NO;
                }
            }
            else
            {
                NSLog( @"Ooops %@", avErr );
                self.busy = NO;
            }
        }
        else
        {
            NSLog( @"No exclusivity %@", avErr );
            self.busy = NO;
        }
    }
}

#pragma mark -
#pragma mark Audio capture delegate

- ( void ) captureOutput:( AVCaptureOutput     * ) output
   didOutputSampleBuffer:( CMSampleBufferRef     ) sampleBuffer
      fromConnection:( AVCaptureConnection * ) connection
{
    CMItemCount n = CMSampleBufferGetNumSamples ( sampleBuffer );

    // We have our standards
    if ( n == 1024 )
    {
        AudioBufferList audioBufferList;
        
        CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer (
                                     sampleBuffer,
                                     NULL,
                                     & audioBufferList,
                                     sizeof ( audioBufferList ),
                                     NULL,
                                     NULL,
                                     kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
                                     & sampleBuffer
                                     );
        
        // Loop buffers
        for ( int b = 0; b < audioBufferList.mNumberBuffers; b ++ )
        {
            // Evaluate samples
            [self fft:audioBufferList.mBuffers [ b ].mData];
        }

        // Release the baby ... I mean buffer
        CFRelease ( sampleBuffer );
    }
}

- ( void ) fft:( SInt16 * ) samples
{
    // In place so r and i are both input and output
    COMPLEX_SPLIT c;

    float r [ 512 ];
    float i [ 512 ];

    c.realp = r;
    c.imagp = i;

    // Load it and calculate maximum amplitute along the way
    int amp = 0;

    for ( int s = 0; s < 512; s ++ )
    {
        SInt16 ev = samples [ s * 2     ];
        SInt16 od = samples [ s * 2 + 1 ];
    
        // Convert to float
        r [ s ] = ( float ) ev;
        i [ s ] = ( float ) od;

        if ( amp < ev )
        {
            amp = ev;
        }
        if ( amp < od )
        {
            amp = od;
        }
    }

    // Only proceed if we have a big enough amplitute
    if ( amp > THRESHOLD )
    {
        FFTSetup fft = vDSP_create_fftsetup ( 10, kFFTRadix2 );
        
        if ( fft )
        {
            // FFT!
            vDSP_fft_zrip ( fft, & c, 1, 10, FFT_FORWARD );
            
            // Get frequency
            int   maxS = 0;
            float maxF = 0;

            for ( int s = 1; s < 512; s ++ )
            {
                float f = r [ s ] * r [ s ] + i [ s ] * i [ s ];
                
                if ( f > maxF )
                {
                    maxF = f;
                    maxS = s;
                }
            }

            // Dealloc
            vDSP_destroy_fftsetup ( fft );

            // Done
            dispatch_async ( dispatch_get_main_queue (), ^ {
                
                [self measure:maxS
                        n:1024];

            } );
        }
    }
}

@end

为什么这在设备上运行良好但在模拟器上拒绝？

然后，第二个问题，因为我确实在这里提供了所有细节，关于如何提高准确性的任何想法，或者只能通过使用更多频率触发器来实现？

TIA

标签： iosobjective-c

Answer 1

欢迎来到仅使用真实设备进行调试的世界，因为涉及到音频并且模拟器可能对此很挑剔。

请记住，nil/NULL在为它们分配任何东西之前，您希望将 AVCaptureXYZ 指针设置为。音频是 C 业务，Objective-C 不是调用快速快速工作的方法的理想语言。即使它有效.. 还没有什么新东西。

此外，您可能在打开任何会话之前需要一个设备，因此 AVCaptureSession 可以在 AVCaptureDevice 启动之后进行。我知道文档告诉对方。但是当没有设备时你不需要会话，对吧？:)

写入时dispatch_async(...，self->_busy代替self.busy. 并且dispatch_async(dispatch_get_main_queue(),^{})是线程业务，把它放在它所属的地方，围绕访问 UIKit 的东西。在例子里面-(void)measure:(int)samples n:(int)n。

帮自己一个忙，把objective-C-(void)fft:(SInt16 *)samples; 改成

void fft(SInt16* samples, int *result) {
    //do fast fourier transformation
}

如果您需要在此函数中访问self，那么您实际上正在做一些接近错误的事情。避免在音频线程中使用 ObjC 方法调用。给这个函数一个void*指针变量以使其可以从函数内部访问呢？或者将引用指针传递给函数以更改给定变量的内容。或者让它返回结果。

并忽略这个特定的模拟器警告。这是一个警告，它为工厂添加了一个实例，因为那里还没有那个 CFUUID。这不是一个错误，这是因为你在 OSX 偏离路线的模拟器上运行 AV_XYZ-iOS 的东西。

一些微小的变化..您的浮点转换可能看起来像。

SInt16 amp = 0;
int s=0;
SInt16 evens;
SInt16 odds;
while ( s < 512 ) {
    evens = samples[s * 2    ];
    odds  = samples[s * 2 + 1];
    r[s] = (float)evens;
    i[s] = (float)odds;
    amp = MAX(amp,MAX(odds,evens));
    s++;
}

并在委托方法中-captureOutput:didOutputSampleBuffer:fromConnection:

CMItemCount numSamplesInBuffer = CMSampleBufferGetNumSamples(sampleBuffer);
// works only with 1024 samples
if ( numSamplesInBuffer == 1024 ) {
    AudioBufferList audioBufferList;
    CMBlockBufferRef buffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer,
                                 NULL,
                                 &audioBufferList,
                                 sizeof(audioBufferList),
                                 NULL,
                                 NULL,
                                 kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
                                 &buffer //now its correct pointer
                                 );

    //provide variable for feedback
    int result = 0;

    // Loop buffers
    int b = 0;
    for (; b < audioBufferList.mNumberBuffers; b ++) {
        // Evaluate samples
        // use C if possible, don't call ObjC in functions if possible
        fft(audioBufferList.mBuffers[b].mData, &result);
    }
    // later Release the baby ... I mean buffer <- yes buffer :)
    CFRelease(buffer);
    
    [self measure:result n:1024];
}