I added a themes.xml in /res/values-v11 and made sure that my Themes defined there extend "@android:style/Theme.Holo.Light" or "@android:style/Theme.Holo". When running my app on the Galaxy Nexus everything looks fine.
When running it on an Emulator with Android 4.0 or 4.0.3, the new Holo-Theme does not seem to take effect; the ActionBar Buttons are not displayed, etc.