You are now following this question
- You will see updates in your followed content feed.
- You may receive emails, depending on your communication preferences.
How to Filter a Real-Time Signal With Rapidly Changing Frequency? (Possible?)
11 views (last 30 days)
Show older comments
Hi everyone,
I need to precisely filter a real-time signal with rapid amplitude and frequency changes (from 20kHz to less than 1Hz).
Changes in the frequency are fast to the extent that FFT methods can not be used.
For detecting a 1Hz frequency there should be a 1s window of data.
The sweep between 20 kHz and 1 Hz occurs in a few milliseconds.
Besides the system requires one microsecond updating.
The latency of the system is extremely important. Because it is a real-time system and I have to implement it in an FPGA device.
Further Information:
Indeed, there are disturbances in the system that cause instantaneous variation in frequency and amplitude of the signal compared to the steady-state.
I have to detect these abrupt changes in the real-time process and change the filter coefficient accordingly to be able to provide proper output (smooth as much as possible) that can represent actual system behavior.
If I use a constant coefficient filter, which is designed for steady-state, for whole the process it causes a significant error and malfunctioning of the system.
An example from the beginning of the signal; there are many ones in the middle of the signal.
The Yellow Signal is the respons of the lowpass filter at main frequency in which the system produces in the steady-state (I mean without disturbances).
Whenever a disturbance occurs, the system always brings about transitions to high frequencies and then back to the main frequency.
...
36 Comments
Mathieu NOE
on 21 Sep 2021
hello (again !)
what is the intention ? you want to make a bandpass filter that tracks the signal frequency ? (seems this is a fast sine sweep)
you can use a trigger and a counter to estimate the signal "almost" instantaneous frequency and tune the filter coefficients on the fly
Minerva Bionic
on 21 Sep 2021
Edited: Minerva Bionic
on 21 Sep 2021
thank you for your reply.
I'm tunning the filter coefficients dynamically. However there are many rapid changes in the signal in which I can not estimate the frequency to tune the filter coefficients.
"use a trigger and a counter to estimate the signal 'almost' instantaneous frequency"
I don't understan what you mentioned. May I ask you to explain more?
An example from the beginning of the signal. There are many unpredictable ones in the system process.
Mathieu NOE
on 21 Sep 2021
hello Minerva
so this plot is quite different from the first one ! or is it just a zoom on the start ?
does it has to do with your previous post regarding PID control with filter in the loop ?
what is the target in terms of data filtering ?
Minerva Bionic
on 21 Sep 2021
Edited: Minerva Bionic
on 21 Sep 2021
It is a zoom on the begining of the signal.
It's a complete different problem.
The raw signal introduces excessive noises into the system that cause malfunctioning. However, inappropriately filtered signals also cause malfunctioning. Unfortunately, these changes occur exactly in critical points for the system.
Minerva Bionic
on 21 Sep 2021
Edited: Minerva Bionic
on 21 Sep 2021
...
Let me present you the difference between the required filtering at the beginning and the destination filter (frequency).
Mathieu NOE
on 21 Sep 2021
ok
I see 3 curves , the raw noisy data, then a bit of smoothing (with smoothdata ?) then a response through a very low fc lowpass filter
so what is the target ? can you elaborate ?
can you also share the data if we want to try on my side ?
Mathieu NOE
on 21 Sep 2021
also can you explain what you mean by "the destination filter (frequency)." ?
tx
Minerva Bionic
on 21 Sep 2021
Edited: Minerva Bionic
on 21 Sep 2021
The destination frequency is the main frequency in which the system produces in the steady-state (I mean without disturbances).
Whenever a disturbance occurs, the system always brings about transitions to high frequencies and then back to the main (destination) frequency.
Sorry for the weird naming :-)
Mathieu NOE
on 21 Sep 2021
ok , so the y data of this plot is a frequency ?
so, again, what do you expect for the "ideal" filter effect ? can you simply draw it on top of your data ?
Minerva Bionic
on 21 Sep 2021
The first is the raw signal.
The second one is the response of a lowpass filter (passband frequency: 2kHz)
The third is the response of a lowpass filter with the passband frequency of 5Hz.
I want to implement a digital filtering system on an FPGA that its output be smooth and as close as possible to the raw signal.
I will upload the signal in few minutes.
Thank you.
Mathieu NOE
on 21 Sep 2021
ok
do you look for a "real time" solution or it's to post process once the data are stored ?
can you share the data ?
Mathieu NOE
on 21 Sep 2021
sorry , I didn't see you already posted the data
still the question : do you look for a "real time" solution or it's to post process once the data are stored ?
Minerva Bionic
on 21 Sep 2021
I'm looking for real time solution and I have to implement it on an FPGA device.
And the latency is critical to the system.
Mathieu NOE
on 21 Sep 2021
do you know how to implement a IIR buterworth filter ?
seems to do the job quite well :
and maybe you could do the filtering in the analog section before it enters the FPGA
clc
clearvars
load('Data1_Sept_21.mat');
y = Data;
Fs = 1e6;
dt = 1/Fs;
t = (0:length(y)-1)*dt;
%%%%%%%%%%%%%%%%
NN = 2;
Wn = 0.0002;
[B,A] = butter(NN,Wn);
figure(1)
ys = filter(B,A,y);
plot(t,y,t,ys);legend('Raw','Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with butterworth LP' ]);
grid on
Minerva Bionic
on 21 Sep 2021
Edited: Minerva Bionic
on 21 Sep 2021
Dear Mathieu,
Thank you in advance.
Both of those filter responses that I posted previously are the output of IIR Butterworth filters.
You filtered the signal at 100Hz . Let me zoom in to see what can come out of it.
as you can see, there is a considerable difference between the 100Hz output with the raw signal.
additionally, these small variations in the signal have to be omitted. Because they cause a considerable error in the system (the acceptable filtering frequency for the steady state signal is at least 5Hz).
Minerva Bionic
on 21 Sep 2021
Edited: Minerva Bionic
on 21 Sep 2021
Let me show you the error at the out put of system with 100Hz and 5Hz lowpass filters respectively:
Mathieu NOE
on 22 Sep 2021
well ! the second plot is even more disturbing to me... I don't know what is your "process" , but I would not feel ok with such spikes in the output. I understand your search for a better input filter but maybe there is also something to do between the input and output (your process) . What creates those spikes ? if that is your main issue
all the best
Minerva Bionic
on 22 Sep 2021
Edited: Minerva Bionic
on 22 Sep 2021
Thank you for your reply.
Fortunately, hose outliers (spikes) in the second plot can be easily handled within the signal processing system by adding a few mathematics. But this just works fine in steady state.
The signal processing solution that I'm going to implement, must work in every condition.
Unfortunately, there is not any way to change the main process.
It is not a simple case at all, at least for me.
Mathieu NOE
on 22 Sep 2021
hello Minerva
tried today another approach with time varying smoothing factor.... low at the begining then increasing up to the max allowed value. This makes little effect of the initial peak, but after a while the smooting effect is much stronger. So very little noise in the steady state time frame. You cannot avoid the time lag as we ask for a very strong low pass filter.
plot
code
clc
clearvars
load('Data1_Sept_21.mat');
y = Data;
Fs = 1e6;
samples = length(y);
dt = 1/Fs;
t = (0:samples-1)*dt;
ind = find(t<0.4);
t = t(ind);
y = y(ind);
samples = length(y);
%%%%%%%%%%%%%%%%
% "smart filter" : simple first order low pass filter (2 stages in series = 2nd order filter),
% the smoothing factor varies with time (low at the beginning , then
% increases to reach max allowed value
alpha = 0.99; % initial value
alpha_max = 0.999975; % final (max) value - must always be slightly below 1
%% main loop
flag = 0;
ys = zeros(size(y));
ys2 = zeros(size(y));
ys(1) = y(1);
ys2(1) = ys(1);
alphaplot = zeros(size(y));
alphaplot(1) = alpha;
for ci = 2:samples
if y(ci) >1 % start time to modify (increase alpha)
flag = 1;
end
if flag == 1 % increase progressively alpha but saturate to alpha_max value
alpha = alpha + 1.5*1e-7;
if alpha>alpha_max
alpha=alpha_max;
end
end
% two first order low pass IIR filter recursion
ys(ci) = alpha.*ys(ci-1) + (1-alpha).*y(ci);
ys2(ci) = alpha.*ys2(ci-1) + (1-alpha).*ys(ci);
alphaplot(ci) = alpha; % just for plotting purposes
end
clear ys
figure(1)
subplot(211),plot(t,y,t,ys2);legend('Raw','Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with home made filter' ]);
subplot(212),plot(t,alphaplot);legend('alpha');
Minerva Bionic
on 23 Sep 2021
Undoubtedly, the solution you provided is intelligent and amazing.
But there are still missing parts, about how can I detect the beginning of the variation in coefficient and the initial coefficient.
The time of occurring disturbances in the system is unpredictable as well as the frequency it sweeps to.
However, I really appreciate your intelligent solution.
Mathieu NOE
on 23 Sep 2021
hello Minerva
if we make the coefficient of smoothing dependant of the signal characteristics (amplitude and or frequency) may be we can still using this king of approach . We only have to know when we are in steady state vs transient. That should be managable ? maybe you can tell more about this than I can on my side.
another trial of the same code on your second data file (Data2_Sept_21.mat)
Minerva Bionic
on 23 Sep 2021
Edited: Minerva Bionic
on 23 Sep 2021
Thanks a lot for your reply.
Unfortunately, all information that we can use to implement this stand-alone signal processing system are these signals.
If we need more information. we have to extract it from the signal.
Without considering hardware and latency limitations, the best solution could be using FFT to extract the main frequency and then manipulate the filter coefficient.
For detecting a 20kHz frequency, we need at least a 40k sampling rate, and the minimum required number of points is 40k/5 = 8k (5 is the minimum frequency). It means That we have to implement 8k points FFT and then cope with implementing another algorithm to find the mean frequency.
Due to the hardware and latency limitations, it is unacceptable.
I will upload another data set as soon as possible.
Thank you.
Mathieu NOE
on 23 Sep 2021
hello
do I have to understand that all your focus is to know wheter there is a 20 kHz strong signal or not ? the first data record is not really showing this behaviour spike followed by a 3 Hz rumble + noise) , the second shows a strong 300 hz tone + lot of spurious frequencies above 3 kHz . But you wanted also the first one to be heavily smoothed (once the peak is gone) so I am a bit lost about what is the target or criteria that drives if I have to apply a strong or weak filter ...
I stay at your disposal if there is more I can do
Data1_Sept_21 FFT plot
Data2_Sept_21 FFT plot
Minerva Bionic
on 23 Sep 2021
Edited: Minerva Bionic
on 23 Sep 2021
There are disturbances in the system that cause instantaneous variation in frequency and amplitude of the signal compared to the steady-state.
I have to detect these abrupt changes in the real-time process and change the filter coefficient accordingly to be able to provide proper output (smooth as much as possible) that can represent actual system behaviour.
In other words, I need to know the main frequency at each time.
If I use a constant filter for whole the process it causes a significant error and malfunctioning of the system.
I have uploaded a new data set and I will upload another one with 20kHz frequency as soon as possible.
Sorry for poor description.
Mathieu NOE
on 23 Sep 2021
hello again
so for the second and third data set , could you draw the "ideal" signal as you did on the first one :
Minerva Bionic
on 23 Sep 2021
Edited: Minerva Bionic
on 23 Sep 2021
Here you are.
I'm not good at all in painting.
However, I think that we should look for a way to detect the main frequency at each time.
Kind regards.
Mathieu NOE
on 23 Sep 2021
Hmmm , not sure if it's so easy to decide based only on frequency info. In the first data , there is this rumble at 3 Hz after the spike. do we consider this frequency as "noisy" or not ? what are we supposed to do ? it was decided in the beginning that the sharp peak should be lightly filtered then after the decay we need strong filtering (exactly where we have this3 Hz rumble).
according to your paintings (they are pretty , believe me !) , I was more thinking about taking small buffers of data and analysing the mean value , deviation (will increase with strong noise) and average slope (like a sharp peak should show high slope condition)
my 2 cents for this evening (in France)
Minerva Bionic
on 23 Sep 2021
Edited: Minerva Bionic
on 24 Sep 2021
In the first data set, there is an impulse at the frequency of 5KHz that gradually sweeps to the steady-state frequency of 3Hz ( the steady-state frequency also varies between 0 and 500 Hz, but we know the steady-state frequency).
I will try to implement your valuable suggestions.
Thank you.
Mathieu NOE
on 24 Sep 2021
Ok , I'll be glad to help further if I can; this is a tricky but interesting challenge !
have a good day
Minerva Bionic
on 24 Sep 2021
Edited: Minerva Bionic
on 24 Sep 2021
Of course, it is.
Especially, if we look for a general solution.
I'm trying different methods.
Thank you, dear Mathieu.
Mathieu NOE
on 29 Sep 2021
hello Minerva
so how are you doing today ?
did you manage to find the "ideal" filter by now ?
all the best
Minerva Bionic
on 29 Sep 2021
Edited: Minerva Bionic
on 29 Sep 2021
I hope you have a great time in beautiful France.
Talking about the problem, I think it is not just about filtering, but it is about "how" and "when" using a proper filter.
As I mentioned before it is not a simple case at all, at least for me.
Thank you.
Kind regards.
Mathieu NOE
on 29 Sep 2021
hello again
thank you , I hope you have a good time also at your location
yes, your problem is not simple , I don't know if I will be of any help from now - let me know
keep the faith !
all the best
Minerva Bionic
on 29 Sep 2021
Of course,
I will update you about this case with my findings or questions.
Thanks a lot.
Answers (0)
See Also
Categories
Find more on Signal Generation and Preprocessing in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!An Error Occurred
Unable to complete the action because of changes made to the page. Reload the page to see its updated state.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)
Asia Pacific
- Australia (English)
- India (English)
- New Zealand (English)
- 中国
- 日本Japanese (日本語)
- 한국Korean (한국어)