1

I am developing a project with the esp32, that is measuring sensor data, saving this data to an sd and also publishes the data to the web (to Google cloud IOT core). The measuring and saving of the data works already very well, but I am facing some issues regarding the sending to the web.

To measure the data (max frequency should be 500 Hz), I am using a timer, that measures the data periodically and sends the measured data to a queue. A separate task reads the data from the queue and then saves this data formatted to the sd card. So far so good.

I am using esp-idf for the development and therefore implemented the web framework esp-google-iot. Connecting to the cloud worked very well and I am able to receive commands already (in the first approach just a command to stop or start with measuring). Now I tried to create another queue, where the data, that should be published is hold. Then I created a delayed publish task, that should send all the data, stored in the queue every second to the cloud. Unfortunately I am always receiving the error "heap out of memory".

As I thought this could appear due to to big data amount, I removed the second queue and just sent the message "dummy" every second. This worked perfectly.

Does anyone know, how to handle these amount of data or had similar issues?

Probably I am also doing the completely wrong approach.... Probably sending more than just one value per publish, so that I do not have to send 500 requests per second?

Any help is appreciated

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
ledtom
  • 19
  • 1
  • Could you help us by posting some code which you use? Ideally, this should be a minimal working example, it doesn't need have real data, though. Yes, probably sending 500 requests/s may not be a good idea. Please have a look here if you need more information on your device's heap usage: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/mem_alloc.html?highlight=heap%20debugging – StrawHat Jun 30 '20 at 06:15

1 Answers1

0

Depending on what you're doing on IoT Core, you could send the whole queue in one JSON string and use a Cloud function to parse through the JSON and move the data to BigQuery. I think this might solve the problem that you're having although i suggest just publishing the the data whenever your device has read it.

Also you might want to space out your publishing to more then every second since as StrawHat pointed out sending 500 request/s is not a good idea and would probably cause IoT Core to reject the connection.

Gal Zahavi
  • 51
  • 2