Safety depends very much on the code as you implement it. I prefer to avoid logging in people with Twitter, because it is very easy to create a fake account on Twitter. For now, I am with Facebook and Google, and I've noticed nothing particularly "dangerous" in terms of security.
The odd phenomenon (at least as far as my experience goes) is that, when presented with two options, i.e., the possibility to sign up with a "stardard form" that requires a verification email (long procedure) and the possibility to click a button and login with Google, Yahoo, or Facebook (fast and easy), users prefer providing their information with the form, the good old way. It must be due to rumors about the privacy breach you mention.
I don't think I can dismiss or confirm such myths. The sure thing is that the Google+ button (the +1 button to be more specific), says hi if you visit any site that has it while you're logged in, and greets you by name. Google analytics suggests you change your privacy statement if you decide to track the interests and hobbies of visitors. Facebook has insights too. I don't think that these are all myths.
Privacy also depends on what you do with the data you can collect with Google/Facebook/Live login. I have made it a point not to share, sell, use any of the information gathered - not even emails, not even for newsletters (I don't send newsletters). I do understand that this may be pure idealism, and that it doesn't bring you that far if you have to run a business (not sure!), but so far it's working fine, at least for me, and for my users.