Browser will definitely cache the 301 redirect because it's a permanent redirect. Basically you're telling the browser never to look at the original URL and to automatically go to the second URL. Same thing happens for any search engines. If you ever plan to use that original URL again, make sure you use a temporary redirect.
Specifically, after visiting the first URL, will some browsers start
performing a 301 redirect for the id=2, or are redirects based on the
full URL?
No, the browser should see each URI in it's entirety, so page?id=1
is a different URL than Page1?Id=2
and the 301 redirect would only apply to the first url.
Keep in mind that the browser is doing the redirect and implementing its own rules based on its interpretation of the HTTP spec. If there are any bugs in the browser you could get unpredictable results, but the short answer is that they're different requests to different urls and each get their own HTTP status code therefore they can be treated as separate pages.