Please remember, HTMLHelp (CHM's) is nearly 20 years old and Microsoft's Windows based help system is in "maintenance mode" only. No new features and support are expected. As far as I know there is no new help systems being developed at Microsoft.
So, the CHM Compiler (written in C++ 1996 by Ralph Walden) has some limits - it's a HH compiler problem not a Doxygen error.
See also: https://superuser.com/questions/330418/file-size-limits-for-chm-html-help-format-files
Please try to compile without 'Index', try to reduce the number of topics, try to compile on a single PC (I think you are working in a server Environment).
I see a small chance for a solution only because there are about 1106 files in your snap above.