0

I am running a webserver on an ESP32 chip using the libwebsocket library. The server files are in a ROMFS partition on the ESP32. Currently I am in the process of trying to improve the loading time by concatenating, minifying and compressing the javascript, html and css files. The concatenation and minification worked properly, I now only have a concatenated.js and concatenated.css file in my website. But the issue came when I tried to get the compression working.

Initially, I thought my server would compress the files by itself before sending them, however when I looked at the server file transfer using Chrome developper extension, I found out that the javascript file GET request was returned with "content-type: text/javascript".

I tried several solutions I could think of, but none seem to work:

  • gzip the file before creating the romfs (ie there is now only a concatenated.js.gz in my ROMFS file system)
    • The server returns 404 when trying to access "concatenated.js"
  • gzip the file before creating the romfs and make it live alongside the original file (I was thinking maybe libwebsocket would be able to see they were both there and pick the most efficient one)
    • The server only returns the js file, and never the gz file

Does anybody knows how to enable the gzip compression in libwebsocket ? I am guessing there must be some options I don't have enabled, but it has been hard finding resources on the web. Most of them only discuss about the ability of the libwebsocket to get gzip from a zipped file.

Regards,

Tom
  • 1
  • 2
  • Hi @Tom, welcome to Stack Overflow. I'm confused by your questions. Websockets have little to do with running web servers. Websockets start their initial protocol over HTTP but then do their own thing. They provide a bidirectional data stream between the two endpoints. They have nothing to do with serving files as you're describing or gzipping them. You appear to be describing a problem with a web browser fetching compressed files from an ESP32-based web server - what does "libwebsocket" have to do with that? – romkey Jul 01 '22 at 16:22
  • No problem, I am getting confused myself and fairly new to webdevelopment. The libwebsocket library is not just taking care of websocket, it is also the library in charge of responding to HTTP GET request on the ESP32 and serving the files. From my understanding, it is the one that then should find/compress/send the files requested by the browser back to it. Does that help ? – Tom Jul 01 '22 at 16:32
  • 1
    Hi @Tom - Generally on the ESP32 you'd use [ESP-IDF's http server](https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_http_server.html), Arduino's [WebServer](https://github.com/espressif/arduino-esp32/tree/master/libraries/WebServer) or one of the Pythons' HTTP servers to serve HTTP files, depending on your environment. You might have an easier time with one of those as they're much more commonly used. – romkey Jul 01 '22 at 16:55
  • 1
    You're unlikely to want to compress files on demand as that would kill the ESP32's performance. It's more a matter of respecting HTTP negotiation on compression, mapping a URL to the compressed file and setting the Content-Type correctly. You can certainly use pre-compressed files to make that work, and you're much better off doing that on a small CPU like the ESP32. – romkey Jul 01 '22 at 16:56
  • Hi @romkey, unfortunately the project is now fairly advanced and I don't think I will have the opportunity to do a change of that scope. – Tom Jul 01 '22 at 16:57
  • ROMFS is also not something that is normally used on an ESP32. – romkey Jul 01 '22 at 16:58
  • I agree, it would be much better to precompress the file and be able to serve them or the non-compressed one as needed. I am not sure how to do what you described though... ^^ – Tom Jul 01 '22 at 17:00
  • If you don't get an answer here research how HTTP compression is done. I told you most of the parts in the previous comment (negotiation, Content-Type, URL mapping). – romkey Jul 01 '22 at 17:01

1 Answers1

0

The issue ended up coming directly from the libwebsocket code.

When opening a file from the ESP32, there was no logic in place to look for a file with the same name and ".gz" at the end. The logic to look for such a file if the browser accepted gzip file needed to be added to the function.

This change was done on an older version of the libwebsocket, and as such may not apply to the latest version (for anybody looking at this modification). Also, I needed to include <string.h> to have access to the string manipulation functions:

libwebsocket/lib/plate/freertos/esp32/esp32-helpers.c -> function esp32_lws_fops_open

Replace

f->i = romfs_get_info(lws_esp32_romfs, filename, &len, &csum);

By

// check for the gzip file if gzip is allowed by the browser
f->i = NULL;
if((*flags & LWS_FOP_FLAG_COMPR_ACCEPTABLE_GZIP) == LWS_FOP_FLAG_COMPR_ACCEPTABLE_GZIP)
{
  char *filename_gz = malloc(strlen(filename) + 3 + 1); // add space for ".gz" and null termination
  sprintf(filename_gz, "%s.gz", filename);
  f->i = romfs_get_info(lws_esp32_romfs, filename_gz, &len, &csum);
}

// if we haven't found a gz file (not allowed or no gzip), search for the regular file
if(!f->i)
{
  f->i = romfs_get_info(lws_esp32_romfs, filename, &len, &csum);
}
// otherwise, add the flags to let the library knows the file transfered is a gzip file
else
{
  *flags |= LWS_FOP_FLAG_COMPR_IS_GZIP;
}
Tom
  • 1
  • 2