Not sure if you're still looking for a solution here, but I found your question when trying to avoid overlap myself.
In case you didn't know, you can experiment with some of the attributes for the organic layouts. For example, you can do the following:
mxFastOrganicLayout layout = new mxFastOrganicLayout(adapter);
layout.setForceConstant(150);
The force constant has the following description in the docs:
The force constant by which the attractive forces are divided and the replusive forces are multiple by the square of.
For me, I played around with the force constant until I got a layout with far fewer overlaps. There are a number of other attributes that you can play around with as well to see if this helps your overlapping issues. The ones that made the most difference for me are as follows:
layout.setMinDistanceLimit(5);
Increasing your minDistanceLimit
should allow your vertices to spread out more.
Since fastOrganicLayout also works through simulated annealing, you can actually specify the maximum amount of iterations to run the layout through. If your map is particularly complex (with lots of vertices) it might also help to increase this.
layout.getMaxIterations(2000);
For me, changing the force constant made the most difference but your mileage may vary.
I've not used hierarchical layout before but there seem to be similar attributes that you can change which may lessen your chance of overlap.
If you came up with a more solid solution than this then it would be great if you could share it.