Since in general this is meant as a feature, you'd like to just block one or several specific attachments from common view.
If this is true, TracFineGrainedPermissions is for you. Then you'll be able to have something like
[wiki:Users/KilldaclickHome@*/attachment/supersecret.file]
killdaclick = ATTACHMENT_VIEW
* = !ATTACHMENT_VIEW
to just deny access to the specific file to anyone but yourself. Have a look at the FineGrainedPageAuthzEditorPlugin for a alternative way to define these permission (from the web-UI instead of direct access to the authz file).
You'll certainly think twice about granting permission 'TRAC_ADMIN' - just a hint that these user can edit the file and possibly circumvent any restriction. But 'TRAC_ADMIN' will have 'ATTACHMENT_VIEW' anyway.