"Data element too large" error when compiling+assembling with fpc (on OSX)

Question

How can I troubleshoot and fix a fatal Data element too large error when compiling and assembling with fpc (Free Pascal compiler) in a 64-bit Mac/OSX environment?

It seems that the cause of the error is probably an array[0..MaxInt] call (see below for more details), but if so, I don’t to how to fix it or work around it.

Details: I’m getting that error when running the build.sh script from https://github.com/whatwg/wattsi using fpc 3.0.0-rc1 installed from ftp://freepascal.stack.nl/pub/fpc/beta/3.0.0-rc1/i386-macosx/ (from the pc-3.0.0rc1.intel-macosx.dmg image). (Note: The README.md file says I need to use 3.0.0-rc1 specifially—instead of, I guess, v2.6.4, the latest stable?).

The build runs fine until, after compiling the https://github.com/whatwg/wattsi/blob/master/src/html/htmlparser.pas source, it fails with that Data element too large error when trying to assemble that source.

Specifically, it logs this: htmlparser.pas(336,42) Error: Data element too large. And lines 333 to 336 of that htmlparser.pas file look like this:

type
    TBlob = Pointer;
    PBlobArray = ^TBlobArray;
    TBlobArray = array[0..MaxInt] of TBlob;

…so I suspect that Data element too large error is caused by it running into a system limit of some kind, due to the array[0..MaxInt]?

The extent of my troubleshooting attempts so far is that in the https://github.com/whatwg/wattsi/blob/master/src/lib/compile.sh file that gets sourced by the build, there’s a line with ulimit -v 800000 which I thought could possibly be over-restricting the memory resources—so I removed that line and re-ran the build script, but I still got the Data element too large error at exactly the same point.

Answer 1

More recent FPC's do a form of compiletime rangechecking. This rangechecking determines compiletime that the type is too big, even though it is never actually allocated (always used using pointers, older Delphi versions couldn't overindex pointers, so this construct was used)

That is a known limitation of the technique, and strictly the original source is buggy.

Since the type is probably never allocated simply readjust the bounderies to fit, e.g.

 TBlobArray = array[0..MaxInt div sizeof(tblob)] of TBlob;

(maybe subtract a few elements to be sure, I don't know if 2GB-2 is the real limit, IIRC old delphis had lower limits)

or even simply

 TBlobArray = array[0..0] of TBlob;

And disable runtime checks where you access it. The high upper bound is an artificial construct to not run into runtime checks

Answer 2

Try to change the DEFINES line in src/build.sh from

DEFINES="-dUSEROPES -dLINES -dPARSEERROR"

to

DEFINES="-dUSEROPES -dLINES -dPARSEERROR -Px86_64"

this is actually a fresh note added to the repo origin README.md