0

I am trying to parse xbrl file (1.35Gb) via arelle. During debug I spot that execution holds on line ModelDocument.py:157. It holds more than 30 minutes. Python process take about 8Gb RAM and slowly increases memory consuming:

enter image description here

It looks like python parses xml with 20-50Kb/s speed which is extremelly slow. Especially if we take into account that python have C optimization code. Note also that I got 1 core loaded 100% so CPU does some heavy work (but what exactly?)

Any ideas how xbrl parsing can be speeded up?

System: Windows 10, Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05)

Cherry
  • 31,309
  • 66
  • 224
  • 364

1 Answers1

0

Maybe my answer will be more relevant, in the long term, to developers of XBRL processors, but I would encourage taking a look at what makes an instance streaming-friendly, notably, the following candidate recommendation by XBRL international:

https://specifications.xbrl.org/work-product-index-streaming-extensions-streaming-extensions-1.0.html

Producing and consuming large XBRL instances in a streaming fashion helps avoiding the issue of being stuck at the parsing call, i.e., instead of loading and parsing the entire instance in a bulk, streaming reduces pressure on memory, as the facts can be converted on the fly to the processor's internal memory structure.

In general, just streaming through 1-2 GB of data doing simple things takes much less than a minute. If it takes 30 minutes, it seems that there is optimization potential for the implementation of a processor. I do not think that this is an issue only with Arelle, and I think that with more users opening larger files, implementors will at some point start looking into this.

Ghislain Fourny
  • 6,971
  • 1
  • 30
  • 37