I have also encountered similar problems when reading large Dymola simulation results in MAT format to Python (even though I've used modelicares instead of DyMat, see http://kdavies4.github.io/ModelicaRes/ ). For me, the problem seems to occur for results with many time steps, even when very few variables are written to file.
One workaround that has worked so far is this: Even though the large MAT file cannot be loaded in Python, it has worked for me to load the result file back into Dymola and from there export the results as SDF or CSV. Downsides are of course that the resulting SDF or CSV files are even larger than the original MAT file, and that this involves another manual step, but from there, I was always able to load the SDF or CSV data into Python, e.g. using the sdf
Python package. If this would occur more often, it would probably also be possible to write a MOS-Script to automate the process of loading the MAT file into Dymola and export it to SDF or CSV.
This solution is not ideal, but maybe helpful as a workaround. If there are better solutions, I'd also be happy to hear about them.