deep learning audio plugin Apple Silicon

I created an audio plugin to make predictions about two kinds of sounds with a pre-trained model that I did. It works on audioTestbench, and it's validated by validateaudioplugin, but in VST format, it crashes all my DAWs, or it's unrecognizable. I'm on an M2 Apple Silicon architecture. Is there somebody who has experienced this situation?

Thanks

classdef cnnpredict < audioPlugin

properties

bsize = 1;

end

properties (Access = private)

prediction = 2;

end

properties (Constant)

PluginConfig = audioPluginConfig( ...

'DeepLearningConfig',coder.DeepLearningConfig('none'));

PluginInterface = audioPluginInterface(...

audioPluginParameter('bsize',...

'DisplayName','Buffer size',...

'Mapping',{'lin',0.1,4},...

'Label','seconds'), ...

PluginName="GuessPreparedPiano", ...

VendorName="tommasorosati", ...

VendorVersion="1.1.1", ...

InputChannels=1, ...

OutputChannels=2)

end

properties (Access = private)

Mybuffer

end

methods

function plugin = cnnpredict()

plugin.Mybuffer = dsp.AsyncBuffer(44100*2);

end

function out = process(plugin, in)

fs = getSampleRate(plugin);

% Write signal to buffer

% plugin.Mybuffer=plugin.Mybuffer(1:fs*2);

write(plugin.Mybuffer,in(:,1:size(in,2)));

if plugin.Mybuffer.NumUnreadSamples == (fs*2)

frame = read(plugin.Mybuffer, (fs*2));

plugin.prediction = predictit(frame,fs);

end

if plugin.prediction == 1

out = [in(:,1),in(:,1)*0];

else

out = [in(:,1)*0,in(:,1)];

end

end

end

end

function yy = predictit(x,fs)

persistent trainedNet;

if isempty(trainedNet)

trainedNet = coder.loadDeepLearningNetwork('trained_network_1.mat');

end

x=resample(x,16e3,fs);

x=x(1:32000,:);

auditorySpectrogram = extractAudioFeatures(x);

auditorySpectrogram = log10(auditorySpectrogram + 1e-6);

yy = classify(trainedNet,auditorySpectrogram);

yy = double(yy == "prepared") + 1;

end

[Jimmy Lapierre]

Maybe a few questions to help figure this out: What version of Xcode do you have? Are there any warnings when generating the plugin? Do you have the same issues with AU, AUv3 or standalone?

[Tommaso]

Xcode 14.3.1

no warnings generating the plugin

AU and AUv3 similar problem. DAW ignore them...

[Jimmy Lapierre]

I don't have the mat file and some of the things required to try your example myself, but I wonder if you used the DL config. You could also try the standalone exe (which is actually a AUv3) to see if that works outside the DAW and outside MATLAB (first try running the executable from the MATLAB file browser, then from Finder).

acfg = audioPluginConfig('DeepLearningConfig', coder.DeepLearningConfig('none'));

generateAudioPlugin -audioconfig acfg -exe cnnpredict

Let me know how this goes.

[Tommaso]

I attached the dependencies to allow you to try my plugin.

Meanwhile I'm workinkg on your your suggestion.

Thanks in advance!

[Tommaso]

I tried the standalone as you suggest. No problem in the exporting and launching the plugin. But it seems not working (no output from the standalone plugin).

[Jimmy Lapierre]

With the standalone, there's a menu to select the audio device.

I was not able to reproduce any failure in Reaper on the M1 or M2, so I attached the plugin I built from your sources. I would unzip this and copy it in the VST folder, for example: cp -R cnnpredict.vst ~/Library/Audio/Plug-Ins/VST/

Let me know if that runs on your machine. I'm trying to figure out what might be different (Xcode version, MATLAB version, etc.). Can you tell me what release you are using?

[Tommaso]

I tried your vst but I still got "reaper_host_x86_64 quit unexpectedly." when I try to load it.

And "“cnnpredict.vst” is damaged and can’t be opened. You should move it to the Bin." when I try to open it in MaxMSP 8

Versions:

Matlab R2023a Update 5

Xcode 14.3.1

[Jimmy Lapierre]

Sounds like the MaxMSP issue might be caused by the file being downloaded, I wonder if #4 in this article would help.

For Reaper, with the VST I build, it shouldn't matter which version of MATLAB or Xcode you have. I tried it on a fresh download of Reaper on a M2 with Sonoma and it was running fine (playing one side or the other).

Is it crashing on loading the plugin or when playing back? Maybe there's an issue with the specific media file or project you have? I tried with the RockGuitar 16bit 44.1kHz stereo example file in the audio toolbox samples folder.