You can't simulate the home button being pressed, but you can force the app to exit as if it were. Calling [[UIApplication sharedApplication] terminate];
will terminate or (in the case of iOS 4.x) background the app. You will not, however, be able to relaunch the application using UIAutomation.
You could try UIAutomation in combination with a screen recording script that allows you to replay mouse movement and clicks. This would allow you to interact with the simulator directly for things like home button clicks and app icon clicks.
Alternately you can get "good enough" testing using the UIATarget
class. Per the docs,
The UIATarget class represents high-level user interface elements of
the system under test (SUT)—that is, your application, the iOS, and
the connected device on which they’re running. Your test scripts,
written in JavaScript and running in conjunction with the UI
Automation instrument, use this class and related UIA classes to
exercise the SUT and log results.
Using UIATarget.localTarget().deactivateAppForDuration(seconds);
you can background your app for n seconds.
Use this method to test shifting your application to and from the
background execution context. Note that applications built using iOS
SDK 4.0 or later and running in iOS 4.0 and later aren’t necessarily
terminated when the user presses the Home button. See iOS Application
Programming Guide for details of multitasking and background execution
context.