How do I programmatically get the previous version of an item in a TFS changeset?

Question

Using the TFS APIs it is trivial to loop through changes in a changeset and then get the files using the ServerItem and ChangesetId.

But suppose I want to diff the modified files, how can I get the VersionSpec for the previous versions?

The trick of subtracting one from the ChangesetId seems to break if the file was renamed or branched?