Servers are typically willing to rely upon the IP address of a connection for low-risk traffic because setting up a TCP session requires a three-way handshake. This handshake can only succeed if the IP address in the packets is routable and some machine is prepared to handle the connection. A rogue router could fake IP addresses but in general, it is more difficult to fake connections the further away from either endpoint the router is, so most people are prepared to rely on it for low-risk uses. (DNS spoofing is far more likely way to misrepresent a connection endpoint, for example.)
Higher-risk users must use something more like TLS, IPsec, or CIPSO (rare) to validate the connection end-point, or build user authentication onto the lower layers to authenticate specific connections (OpenSSH).
But the actual contents of the TCP session can be anything and everything -- and a server should not rely upon the contents of the TCP session (such as HTTP headers) to faithfully report IP addresses or anything else vital.