The $templateCache is an applicative cache: angular stores the template in a JavaScript object. As soon as the application restarts because, for example, you refresh the page, this cache disappears and is recreated in the new application.
The browser cache has nothing to do with $templateCache. The browser is responsible for this cache, and populates it based on cache headers sent (or not) by the server. This is what prevents the browser for completely reloading pages, images and other resources that don't change often when using the back and forward buttons, or simply when navigating between pages.
So what you're seeing is completely normal.