This is indeed a problem (although not one you could solve with program code, so it might be very well off-topic).
Google receives the IP address whenever the tracking pixel is called. In theory this should not be enough to recognise a particular user - after all IP addresses are in many cases dynamically assigned, plus you need added information from the telecommunications provider to connect an IP address to a specific person.
On the other hand lead generation software has successfully demonstrated that you can in many cases deduce a user from an IP address and public available information. Lead generation software is usually used to identify potential business clients (i.e. entities, not persons), but at least in theory the method could be applied to individuals as well. Also in some cases Google is a telecommunications provider and might be able to obtain personal data and correlate that to IP addresses.
As for the question if we simply have to trust Google, that depends on what you mean by that phrase. Google is bound by the law, and I trust Google to respect the law (if I assumed somehow that everybody breaks the law I would not be able to do any business). That means that the way they deal with IP addresses depends to some extent on your local legislation.
For example I live in Germany, and here IP addresses are considered personally identifiable information, so they must not be stored without the visitors consent. In the course of a legal debate that took place some five years ago Google introduced a way to anonymize IP addresses so that they are usable to extract geo information but are permanently stored only in obfuscated form. Of course Google still receives the complete address (the internet would not work without some kind of addressing scheme), but they have agreed to respect your settings when it comes to storing it (probably not just for the Germans sake, but it came in time to bring the debate to a conclusion).
I would not simply trust Google to "do the right thing" all of their own accord, but they are bound by the applicable law and I feel that I (and you) can reasonably trust them to keep the law. If you feel they do anything improper you should take legal action, and if you share the by now common sentiment that "the system" is rigged in favour of the elusive "them" who get away with anything (making legal action pointless) then you have officially left civilisation and nobody will be able to calm your fears.
And if the question was, can you do anything against transmitting IP addresses then, no.