After a fair amount of struggling I was able to successfully deploy a WCF 4.0 REST web service on XP/IIS5. For the benefit of you and others in the same situation, here's what I did (let me know if you need more detail):
Using VS 2010, build a deployment
package for the project and then
install it on the target machine
(your web server). The following
link will show you how to do this: http://vishaljoshi.blogspot.com/2009/03/web-packaging-installing-web-packages.html
Make sure that your virtual directory in IIS is configured to use .NET 4 (right click on virtual directory > properties > ASP.NET tab > ASP.NET Version)
Configure your virtual directory to support extensionless requests (i.e. your REST API) by creating a wildcard mapping (right click on virtual directory > properties > configuration > mappings tab) with the following properties:
Executable: C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\aspnet_isapi.dll
Extension: .*
Check that file exists: unchecked
(Note: watch out for the bug that keeps the "OK" button on this dialog grayed out. To get around this just click in the executable textbox after entering your extension and the "OK" button should get enabled).
If you get an authentication error when accessing the service then turn off integrated authentication on the virtual directory. This should fix the problem if you have used the defaults in the generated web.config of your service. If you have explicitly modified the authentication mode then you will have to adapt accordingly.