Most of my 3D modeling is Optical Coherence Tomography data that tends to be rather large. My experience has been a little backwards. The project started as a C++ project using x64 VTK and Qt. Folks in the lab weren't C++ people, so it got converted to Java which is used through Matlab, which the lab post-docs know.
Java VTK is really easy to use and has a lot of features, but I'm having some trouble with registering callback events for buttons and such with will be a deal breaker if it can't get resolved. This problem might dramatically effect how the UI is done, so keep that in mind.
Most of the C++ examples tend to convert to Java really quickly (except callbacks >.< ) so there is a plethora of examples. If you are going to use VTK, I highly recommend "The VTK User's Guide" by Kitware, Inc (ISBN: 978-1-930934-23-8), as it is an excellent overview of the API.
Good luck!