0

The R rgl package exports an HTML widget with the rglwidget() function, built using the htmlwidgets package. Often the data for this widget is quite large, and Pandoc and webshot2 choke on it.

I would like to try compressing the data when the HTML page is created, and uncompressing it in Javascript before display. I can see that there's a Javascript package pako that appears to do what I want, and it can be "browserified", but I can't see how to make it available to rglwidget(). Can anyone describe what's necessary?

Edited to add some more detail in response to a comment: This needs to happen on the server (or actually, even before the file gets to the server). htmlwidgets produces output in a Markdown document that is converted to HTML by Pandoc, and that step is failing because Pandoc chokes on the large JSON datasets. I'm not sure if it's just the size, or the complexity (I think Pandoc parses the JSON, and appears to blow up to a huge memory footprint before it crashes). I'm hoping that by using a base64 blob Pandoc will handle it better. webshot2 also has problems that may be the same.

2nd Edit: I've got some evidence that it's the size that matters, not the complexity. I used base64 encoding on the JSON to make it simpler (just one long string), but 33% bigger, and things were worse. So I'm back to thinking that compression would help.

user2554330
  • 37,248
  • 4
  • 43
  • 90
  • Do you care about file size at the server or transmission speed? For the latter: HTTP has gzip compression built in... – danlooo Feb 18 '22 at 12:54
  • @danlooo: See additional info I just added. – user2554330 Feb 18 '22 at 13:16
  • maybe you can do the compression with a pandoc filter: https://pandoc.org/filters.html – danlooo Feb 18 '22 at 13:26
  • @danlooo: I think I know how to do the compression, it's the decompression that's currently problematic. But maybe a Pandoc filter could still help if I could tell Pandoc to ignore the whole JSON script element. I think for `webshot2` it should be feasible to put the JSON data in a separate file. This could help Pandoc too, but in the end I want a self-contained file. – user2554330 Feb 18 '22 at 14:13
  • Actually, Pandoc filters are applied after the file is tokenized, and I suspect that's too late. – user2554330 Feb 18 '22 at 16:56
  • You mentioned `webshot2` this I assume you're fine with a static image of the rendered stuff (e.g. just include a png of the scene in the Rmarkdown.). `webshot` is based on Chrome which is famous for wanting to have lots of memory. I suggest letting `RSelenium` with Firefox to shoot the picture. The widget standalone in the browser works, right? – danlooo Feb 18 '22 at 17:31
  • This needs to produce different things in different circumstances. Sometimes a static PNG is what the user wants, sometimes they want a dynamic display in the browser. `webshot2` is used for static displays, Pandoc is involved in producing the dynamic ones. Both have problems on the same scenes (ones with multi-megabyte JSON data), hence my guess that compression might help both. – user2554330 Feb 18 '22 at 17:41
  • You can also use `htmlwidgets::saveWidget` for your rgl widget and knit the rest without the widget using pandoc. Then you can insert the widget afterwards into the output of pandoc e.g. as an `IFrame`. – danlooo Feb 18 '22 at 17:46
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/242176/discussion-between-user2554330-and-danlooo). – user2554330 Feb 19 '22 at 16:11

0 Answers0