ThingSpeak


Disappearing data: downloaded excel data that was posted by python script is missing and not visible in widget

Mark Ennamorato on 24 Jul 2021 (Edited on 24 Jul 2021)
Latest activity Edit by Vinod on 26 Jul 2021

Hi,

I get 200 after a Python post to a channel like this from my Python script:

Ambient light: 13.3 lux Mass is 943.6 g 10:37 that is today, 7/24/2021 Pacific time <Response [200]

Yet the data is not showing up in the widget after 03:07 today and when I export the data it agrees with that:

2021-07-24 02:57:17 MDT 4123 947.4597598 0.833333333

2021-07-24 03:07:18 MDT 4124 947.5867868 0.833333333

here is a code snippet: note it posts every 10 minutes

-----------

request_headers = {'Content-Type':'application/json'}
          try:
             response = requests.post('http://api.thingspeak.com/update?api_key=' + API_KEY,
             json=planta_json, headers=request_headers)
             print(response)
          except:
             print("did not post for some reason")

------------

i never get a "did not post for some reason" debug message from the Python script. its like the data just disappears for some reason.

is there something else in the response i should look at to see why this happens? it will work fine for days then do this.

thanks

5 Comments

Mark Ennamorato
Mark Ennamorato on 24 Jul 2021

and there is no Averaging or other things going on in the widget

Vinod
Vinod on 26 Jul 2021

I believe it has to do with the code sending data to ThingSpeak. Clearly one of your other channels is working: https://thingspeak.com/channels/1412172

Maybe you need to check the difference between the code posting to this channel vs the code posting to this channel: https://thingspeak.com/channels/1316073

Mark Ennamorato
Mark Ennamorato on 26 Jul 2021

Hi Vinod ok thanks and yes i thought so as well but here is the deal and I think its a code problem - maybe. All of the fields 1,2,3,4,5,6 in this channel https://thingspeak.com/channels/1412172 are bring driven from 3 different devices using the same exact Python code that I copied and pasted above. Notice that Fields 1 and 2, which is a device in Oregon, has never failed completely (there is missing data due to WiFi but otherwise it has been solid for almost a month). Fields 3 and 4 are the ones that stop posting for some reason but that device is in San Diego. So I did an experiment with a 4th test device here are home with the same exact Python code posting to Fields 7 and 8. That is where I saw the problem again with missing data even though my debug message gave me the status OK after posting data.

I have another channel using Arduino code / Arduino c++ API that has never failed: https://thingspeak.com/channels/798053, but that is only 1 device posting ( at the moment... my pool sensor died due to overheating in the Arizona heatwave a few weeks ago.;-)

Therefore suspecting the Python code in the trouble channels I changed the 4th test device using Python code posting to the above field 7 and 8 but different code:

# trying the new way  to post
      params = urllib.parse.urlencode({'field7':str(weight), 'field8':str(ambient_lux),'key':API_KEY})
      headers = {"Content-type": "application/x-www-form-urlencoded","Accept": "text/plain"}
      conn = http.client.HTTPConnection("api.thingspeak.com:80")

        try:
           conn.request("POST", "/update", params, headers)
           response = conn.getresponse()
           print(datetime.now().strftime("%H:%M"))
           print(response.status, response.reason)
           conn.close()
        except:
           print("did not post for some reason, connection failed")
        finally:
           time.sleep(SLEEP_TIME)

So now I wonder ... what is the 'best' Python API usage? Notice before I never checked the channel connection nor did I close the channel connection. Now I do. Is there a 'really good' Python example you can show me ?

thanks ! Mark

Vinod
Vinod on 26 Jul 2021 (Edited on 26 Jul 2021)

If you have 4 devices posting to the same channel, how are you keeping their clocks synchronized?

Note that a channel can be updated at most every 15 seconds with a free license and at most once a second with a paid license. If you have multiple devices posting to the same channel, and their updates overlap within window I just mentioned, the second device will just get a 200 HTTP status code with a "0" in the response.

The mental model to use is each channel can be updated by one device/MCU with up to 8 sensors (i.e. fields on ThingSpeak) per device. If you have multiple devices, they should really be posting to multiple channels.

Mark Ennamorato
Mark Ennamorato on 26 Jul 2021 (Edited on 26 Jul 2021)

Vinod AH.. ok so that is something I never knew! that explains the missing data holes from time to time but since these things are not time sync'd in any way I would be surprised if they hit the 15 second window for days at a time..but maybe it is possible. Ok so I will next time set up one channel per device. thanks for the explanation.

So.. given these are basically Raspberry Pi Zero W devices their on board clock is not very good.. and they all started at random times by different people so I find it hard to believe they could line up posting times that closely, as matter of fact you can see in the data they are usually minutes apart... but maybe

Tags

No tags entered yet.