I frequently need to create branches in my application for new development. This often means that I have changes to my Gemfile in my branch that are not present in Master, and as such I end up with differences in the cached gems in vendor/cache.
I find that it bothers me that I am committing cached gems, but I don't know whether it is indeed wrong. Should I be .gitignore-ing these? And if so, should I git rm the cached gem files?
TL;DR: This is up to you, but I recommend keeping them in git.
What vendor/cache does is allow bundling, especially deployment bundling to skip downloading the gems from rubygems. This significantly reduces reliance on rubygems.
In deployment, you can specify --local to bundler to completely remove all dependence on rubygems.org, but only if these files are checked into git.
This is valuable because while rubygems.org is a fantastic service, it is not immune to outages. Without these cached gems and outage on rubygems.org could mean that you would be incapable of deploying new versions, redeploying, or scaling up to new machines until rubygems is back up.
There is also an argument that vendor/cache can be used as an extension of Gemfile.lock, locking deployment gems to be binary identical to the ones used in development, not just version identical.
So for practical reasons of keeping your ability to function as a practical, working application independent from rubygems.org: keep them in your git repo.