How to store machine learning models for multiple users - Thingspeak & Matlab

Hi all!

I have been using Matlab and Thingspeak for several years. What began as a hobby for personal use is turning into a potential business but I am running into questions about how to scale my application in Thingspeak and Matlab. For example, I am collecting data from a series of Arduino devices, each of which is assigned to a different customer. I need to train a machine learning model for each customer and store the trained weights for each customer - and then apply the weights in real time as new data arrives in Thingspeak.

I have done this successfully for one user - by storing the weights in the Metadata field in a Thingspeak channel. This feels ineffiient however and I'm hoping there is a better way in Thingspeak - an application I am already so familiar and comfortable with.

Also, I have done a lot of experimenting with Amazon AWS but I find it extremely complicated and frankly potentiall quite expensive - so I'm hoping someone else may have tackled this in a similar problem.

Thank you!

Ben

[Vinod]

Ben,

That's great to hear that your hobby is becoming a business.

Coming to your question, some follow on questions for me to point you in the suggested direction:

• Are you setting up/programming the devices individually using the Arduino IDE or using MATLAB?
• Are the trained weights different for each channel/device, or the same across all channels?
• Are you training the models from MATLAB on your desktop?
• How often does a retraining happen that triggers an update to the channel metadata?

Vinod

Answer 1

[Ben Alimansky]

Hi Vinod -

Thank you so much for your note back. I deeply appreciate your advice and help here. Here are my responses and thoughts to your questions:

1. I've programmed the devices using the Arduino IDE directly.
2. The trained weights are different for each channel. I four channels that are specific to the device itself. Then each device has its own weights too. For example, if I have three devices, I'd have 12 trained models in total. I then associate the models to their respective devices and channels.
3. In terms of training - it's a good question. Right now I am manually training, but I could imagine having an app (say iPhone or Android) that "calls" a cloud based Matlab script to run a classification exercise and store the trained weights. I already have a perspective on how i'd do that.
4. Retraining is not frequent at all. It's basically a one-time event for the device, but I could imagine a user wanting to do it occassionally. It would just be idea to automate the process of retraining - rather than for me to do it manually.

Thank you!!!

Ben

[Vinod]

Hi Ben,

based on what you say, I figure you have the embedded software that goes on the Arduino figured out and your questions are about scaling the MATLAB aspects.

Assuming you are doing the training of the machine learning algorithm and extraction of the weights in MATLAB code, I'd recommend that you modify the same script to use the Channel API to find all all your channels. Then you can iterate over each, pull down data, calculate the weights and upload it to the channel's metadata using the Channel API. So every time you add new devices and calculate weights, your metadata is up to date across all your channels.

As to how to now apply these weights to your data as it comes in, you can use the React App or the TimeControl App on ThingSpeak to execute MATLAB code. If you have multiple channels, I'd recommend the TimeControl app schedule MATLAB code to run at a fixed schedule, maybe every hour, that pulls the raw data for the last hour from the channels the device is sending to, apply the weights and write the "corrected" data to a new channel. Using the Channel API you can iterate over any number of channels you have.

Using the above described approach, you can add and remove devices at will and not have to modify your MATLAB code for the TimeControl or the machine learning.

As for scaling this among many users, you can share the code that you use for both the machine learning and the TimeControl on GitHub (with your user API key replaced by a placeholder, of course) and have each of your users follow a simple copy-paste and change the user API key to reproduce what you have.

Does that address your use case?

-Vinod

Answer 2

[Ben Alimansky]

Hi Vinod -

Is there a way for me to dynamically create and delete channels as needed? I'm thinking about your suggestion and it might work in an even cleaner way if I can simply create a channel for each device that I have. For example, say I have one customer - they would sign into the app to set up the device and I would then establish a channel with their own metadata specific to them. Then if another client comes along, I would do the same for them. This would work cleanly. Right now I've been dealing with how to merge multiple clients into one channel which is too difficult - especially because I have metadata for each that I have to track.

If this is possible I'd just need some help to figure out the code to programmatically both create and delete channels - set them up with the right fields, etc.

Thanks!

Ben

[Vinod]

Yes, there is a way to programmatically create and delete channels. Refer the API documentation here: https://www.mathworks.com/help/thingspeak/rest-api.html

Specifically, see:

https://www.mathworks.com/help/thingspeak/createchannel.html

https://www.mathworks.com/help/thingspeak/deletechannel.html

As of now, there is no documented API to create the MATLAB code for a user and the associated TimeControl or React. We're looking into that. If that is the only thing blocking you, we may be able to discuss workarounds.

Answer 3

[Ben Alimansky]

That's awesome! I'd love to explore a workaround for doing this via MATLAB. I think that basically what you have illustrated may now allow me to run my entire business from Thingspeak - it's very clever! I deeply appreciate your continued support here.