A few tips that may or may not help.
Contrary to common advice, load Angular in head of document. That way when you hit an ng-if during page load, the browser knows what to do with it. If it's in footer ng-if is meaningless until page is loaded and thus elements flash up and then disappear during page load
Use class .ng-cloak as you are doing already.
Use booleans in controller to dictate when to display elements. E.g.
<section ng-if='controller.loading === false'>
you could place some of or your entire markup for a view in such a div to remove elements until they are ready for display
Play with ng-if ng-show and ng-hide, ng-if actually removes elements from the DOM whereas the other two just show or hide. They can produce very different results in terms of smoothness of page loading.
Where you use an expressions on an ng-if, be careful to ensure the expression is very accurate. Don't be sloppy with the expression. Consider what would happen if the var in the expression was undefined. Would your element show when you don't want it to?
Whilst I haven't directly dealt with unloading these same concepts can be applied.