How to partially delete data in ThingSpeak channel?

One of my channels ( https://thingspeak.com/channels/124005 ) somehow got data with invalid date (in future). Probably something went wrong in the code of my IoT device.

The device resumed normal functionality after reboot, and now it is posting data correctly. I will review my code and implement the protection to avoid such incidents in future.

However the channel is hardly usable now - it always shows last invalid data from the future :(

How can I clean it up? Like remove individual data points?

Such topic was already mentioned before ( see https://www.mathworks.com/matlabcentral/discussions/thingspeak/135318-i-can-get-data-from-a-channel-but-cannot-delete-the-data and https://www.mathworks.com/matlabcentral/discussions/thingspeak/133992-how-to-remove-some-single-wrong-values ), but the answer was "no, it wasn't possible".

Are there any plans to implement the functionality for partial deletion of data?

BTW I'm somewhat surprised that the production-ready system doesn't allow data deletion.

Which other options may I consider?

I'd tend to avoid channel recreation, because it looks like an overkill.

Any other ideas?

Answer 1

[Christopher Stapels]

A ThingSpeak channel is supposed to act a bit like a lab notebook. As such, specific deletion has been passed in favor of other features. There are presently not plans to add sepcific deletion, though we are aware of the request, and it may be re-evaluated int he future. I definitely feel your frustration, as I have made the same mistake before as well.

Fortunately, we do allow you to download your complete data set using the web interface. If you want to re-upload the cleaned data, there is a small bit of effort to rewrite the timestamps, feel free to post another question if you need help with that.

I also really appreciate that you did the research looking for related topics, and even shared them. Thank you. I hope you dont find the solution too painful in the end.

[Anton K]

@Christopher Stapels , thanks for the reply.

My channel has ~574160 entries (collected for ~6 years). If I try to recreate it with bulk upload, it will take... 574160 / 960 * 15 / 3600 = ~2.5 hours to fill the channel with same data. Well, I can leave the script running over night and the issue will be "fixed" :)

REF: Bulk upload issue - HTTP Error code 502

[Christopher Stapels]

I reccomend you wait a bit more than 15 seconds to be safe. If you happen to hit a point where the server is really busy, it is possible it might miss your request or think it came too fast (server time vs client time etc)

I'd put some try catch in your code in case something does happen, you dont want to have to repeat the process.

If you are doing it ovenight, you could even put a minute in, and then you will get more sleep (~10 hours)!

[Anton K]

According to the page with License Options:

ThingSpeak is available as a free service for non-commercial small projects (<3 million messages/year or ~8 200 messages/day)

With my bulk import (~500K messages) I definitely fit into the yearly limit, but the daily limit is too small for such a task. Can you please clarify, whether the daily limit (8200 msgs/day) applicable to the bulk import as well?

If so, it will take ~70 days (more than 2 months!) to import my historical data into the recreated channel :(

Even the paid Home license with the limit ~90000 msgs/day won't allow me to finish the bulk import in a day.

[Christopher Stapels]

The 8200 per day is the division of 3 million per year into days. That limit is not specifically enforced. You can choose to use up your messages faster than that if you like.

Since the free account is limited to 4 channels, and the limit is 15 seconds per update, the most a free user can do in a day is 23040 unless they use the bulk update. Then I calculate they can do 960*5760 = 5,529,600 per day. Of course you would exhaust your free messages before that.

Another option is to purchase a paid license. That will give you 33 million messages per year, 1 second updates per channel, and longer MATLAB code execution. You can do 14.4 k bulk updates per 15 seconds. Your job would take a little over ten minutes then. Though we have awesome solar panel installations here, your purchase will help us keep the lights on.

Either way, I appreciate your questions. If you feel inclined to tell us more about your project, we'd love to hear about it in the discussions section of the ThingSpeak community.