ESPAsyncWebServer serve large array from RAM

Question

I am trying to serve a large float array with 8192 values from the ESP32 Heap with the ESPAsyncWebServer library for the ArduinoIDE. The µC is a ESP32 devkit c and I want to access the array with a browser. Here is the code for the array:

#include "AsyncJson.h"
#include "ArduinoJson.h"
#include <AsyncTCP.h>
#include <ESPAsyncWebServer.h>

#define SAMPLES 8192

static float * vReal;

void setup() {
  vReal = (float *) malloc(SAMPLES * sizeof(float));
  assert(vReal != NULL);
}

void loop() {
  //Put something into the array
  for (int i = 0; i < SAMPLES; i++)
  {
    vReal[i] = 1.1;
  }
}

At the moment I use the "ArduinoJson Basic Response" and send the large array in parts of 512 values. With 1024 values I get a stack overflow in task async_tcp, if I try to access the array with the browser, so I set it to 512 values. Here is the code for this:

server.on("/array1", HTTP_GET, [](AsyncWebServerRequest * request) {
AsyncResponseStream *response = request->beginResponseStream("application/json");
const size_t CAPACITY = JSON_ARRAY_SIZE(512); //Compute size of array
StaticJsonDocument<CAPACITY> vRealPart;
JsonArray array = vRealPart.to<JsonArray>();
for (int i = 0; i < 512; i++)
{
  vRealPart.add(vReal[i]);
}
serializeJson(vRealPart, *response); //Print to HTML
request->send(response);
});

I do this 16 times to serve the whole array. Later, I call the path's "/array1", "/array2", "/array3"... with JavaScript and parse the JSON. This is the output, if one of the the path's is called in a webbrowser:

[0.334593,0.427480,0.181299,0.066654,0.271184,0.356220,0.374454,0.235625,...]

This works so far for the most of the time, but I think it is very long-winded. It would be nice, if there is only one path with the whole array. It works with a static file from SPIFFS, for example:

server.serveStatic("/jsonArray1", SPIFFS, "/jsonArray1");

But it takes to much time to write the whole array to flash. Although the reading is realy fast.

I also tried the "ArduinoJson Advanced Response", but I could not get it to run with a JsonArray. Unfortunatly the examples on the GitHub page from ESPAsyncWebServer for ArduinoJson are deprecated, because they changed the syntax a bit in the new version (v6).

In a nutshell: What is the best way to serve such large arrays from the ESP32 Heap with the ESPAsyncWebServer library? The goal is to process the array later with JavaScript in a webbrowser.

Thanks for your help!

Bentiedem

PS: If it helps: I am doing a FFT with the library arduinoFFT from a motor current. Therefore I do a ADC and save the 16384 values from the analog to digital converter in an array. This array is passed to the FFT library. The output is an array with 8192 values. This result is in the heap and should be transferred to my webinterface to display the result. I want to keep the arrays in the RAM for speed. For every measurment you get a result array with 8192 values.

Answer 1

Here a proposal how you could use arrays compiled to flash:

const uint16_t SAMPLESPERSEND = 8196;
const uint8_t SAMPLESNUMSIZE = 8; // as your nums are 0.271184 so 8 chars

float myMeasureVals [SAMPLESPERSEND];
char myValueArray[SAMPLESPERSEND * (SAMPLESNUMSIZE + 1) + 2 + 1] = {'\0'};
char numBufferArray[SAMPLESNUMSIZE + 1] = {'\0'}; // Converter helper array

then you add with that code values

server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request) {
AsyncResponseStream *response = request->beginResponseStream("application/json");
  strcpy (myValueArray, "[");
  for (uint16_t i = 0; i < SAMPLESPERSEND; i++) {
    if (i!=0) strcat (myValueArray, ",");
    // float myTempValue = function2GetValues(); // direct froma function
    float myTempValue = myMeasureVals [i];
    // dtostrf(floatvar, StringLengthIncDecimalPoint, numVarsAfterDecimal, charbuf);
    dtostrf(myTempValue, 2, 6, numBufferArray);
    strcat (myValueArray, numBufferArray);
  }
  strcat (myValueArray, "]");
  myValueArray [strlen(myValueArray)] = '\0'; // terminate correct
  Serial.println(myValueArray);  // Only debug
 request->send(myValueArray);
  myValueArray[0] = '\0';  // empty array for next part
});

Code compiles, have not tested sending could be processed that way or the chunked method has to be used. Normally I only use such large arrays only in setup and read from FileSystem. Handling here is the "same" as ArduinoJson does, but with just a few lines of code and no heap fragmentation if you need long and stable running app.
EDIT
My prefered solution would be to write constantly to a file in SPIFFS (see SD example data logger or this post) and serve the file. For ~10.000 values this is the cleanest method. How to add the brackets and comas is shown in the code above so the content of that file is one large JSON which can be quickly processed via javascript. And by using eg 10 different log files you have the possibility to recover in case of network errors or similar

Answer 2

I finally found a solution now which uses the chunked response. Also ArduinoJson is not used any more. I had to shrink the buffer to get it stable (maxLen = maxLen >> 1;). The library gives me a wrong max. buffer length with maxLen. This could be bug.
It is much more faster than my previous solution and runs without any crashes of the µC. An array with 16384 values gets transfered in 422 ms from the µC to the webbrowser in one piece. Here is the code for the server response:

server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request)
  {
    indexvReal = 0;
    AsyncWebServerResponse* response = request->beginChunkedResponse(contentType,
                                       [](uint8_t* buffer, size_t maxLen, size_t index)
    {
      maxLen = maxLen >> 1;
      size_t len = 0;
      if (indexvReal == 0)
      {
        len += sprintf(((char *)buffer), "[%g", vReal[indexvReal]);
        indexvReal++;
      }
      while (len < (maxLen - FLOATSIZE) && indexvReal < LEN(vReal))
      {
        len += sprintf(((char *)buffer + len), ",%g", vReal[indexvReal]);
        indexvReal++;
      }
      if (indexvReal == LEN(vReal))
      {
        len += sprintf(((char *)buffer + len), "]");
        indexvReal++;
      }
      return len;
    });
    request->send(response);
  });

Here is a complete example for testing purposes:

#include <WiFi.h>
#include <AsyncTCP.h>
#include <ESPAsyncWebServer.h>

const char *ssid = "...";
const char *password = "...";
int counter_wifi = 0;
AsyncWebServer server(80);

#define LEN(arr) ((int) (sizeof (arr) / sizeof (arr)[0]))
#define SAMPLES 16384
static float vReal[SAMPLES]; //Stack
uint16_t indexvReal = 0;

#define FLOATSIZE 20
const char *contentType = "application/json";
//const char *contentType = "text/plain";

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.println();
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(500);
    Serial.print(".");
    counter_wifi++;
    if (counter_wifi >= 10) { //after 5 seconds timeout - reset board (on unsucessful connection)
      ESP.restart();
    }
  }
  Serial.println("WiFi connected.");
  Serial.println("IP Address: ");
  Serial.println(WiFi.localIP());

  for (int i = 0; i < SAMPLES; i++)
  {
    vReal[i] = random(20) * 3.2357911;
  }

  server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request)
  {
    indexvReal = 0;
    AsyncWebServerResponse* response = request->beginChunkedResponse(contentType,
                                       [](uint8_t* buffer, size_t maxLen, size_t index)
    {
      maxLen = maxLen >> 1;
      size_t len = 0;
      if (indexvReal == 0)
      {
        len += sprintf(((char *)buffer), "[%g", vReal[indexvReal]);
        indexvReal++;
      }
      while (len < (maxLen - FLOATSIZE) && indexvReal < LEN(vReal))
      {
        len += sprintf(((char *)buffer + len), ",%g", vReal[indexvReal]);
        indexvReal++;
      }
      if (indexvReal == LEN(vReal))
      {
        len += sprintf(((char *)buffer + len), "]");
        indexvReal++;
      }
      return len;
    });
    request->send(response);
  });

  DefaultHeaders::Instance().addHeader("Access-Control-Allow-Origin", "*");
  server.begin();
  Serial.println("HTTP server started");
}

void loop() {
  // put your main code here, to run repeatedly:

}