EDIT
According to the spec, @font-face fonts should only be downloaded when they are required to render the document: http://www.w3.org/TR/css3-fonts/#font-face-loading
The @font-face rule is designed to allow lazy loading of font
resources that are only downloaded when used within a document. A
stylesheet can include @font-face rules for a library of fonts of
which only a select set are used; user agents must only download those
fonts that are referred to within the style rules applicable to a
given page. User agents that download all fonts defined in @font-face
rules without considering whether those fonts are in fact used within
a page are considered non-conformant. In cases where a font might be
downloaded in character fallback cases, user agents may download a
font if it's contained within the computed value of ‘font-family’ for
a given text run.
NOTE:
At the time of writing this answer, different versions of browsers on various systems seem to implement this behaviour wildly differently. All we can do is hope that they will eventually conform to the spec.
Chrome and Firefox (testing others in progress) will only download fonts that are required to render the content that is being displayed, regardless as to what's been specified via @font-face. More importantly, if you specify an @font-face font and then don't use it, e.g.
.css-rule { font-family: Helvetica,'My at-font-face-font',sans-serif; }
Then your custom font face will not be downloaded because it does not need to be.
To examine what fonts are being download and when, open up your network tab during a page load.
EDIT
It appears that the behavior of browsers varies greatly on this issue and I encourage more looking into your individual circumstances.
I can guarantee one thing, however, and that is not all fonts are downloaded all the time in all browsers, just because you've declared them via @font-face
.