I would like to use graphite to collect metrics from different servers. By default, carbon listens on 2003 on all interfaces, which is fine with me.
Now anyone could theoretically send metric data there. Is there a standard way to prevent this from happening (similar to http base auth ) or do I need to mess with IP based restrictions on the physical interface?
It depends on how much you want to "harden" any Graphite nodes ("Graphite" being whatever mixture topology of carbon-relays, carbon-caches, storage backend, and potentially graphite-web/api).
If you know which hosts in your network should be sending metrics to Graphite (typically relays), you can modify your Graphite host firewall rules to expect either from an explicit list of host IPs or range for the applicable ports. Or you could do something similar at the edge network from firewall or router - I have no advice there as your question doesn't give a fuller scope of what your topology looks like.
An alternative approach would be to use the AMQP support to instead have your nodes publish their metrics to a broker as an authenticated user and then have your Graphite host(s) modify host firewall to only accept TCP 2003 from the broker metrics are being received. The upside here is your Graphite node(s) only need to know what broker metrics will be coming to it from, which drastically simplifies any host firewall rules. Having nodes publish metrics through a lightweight service secures things a bit better in that the "trust" concern you have is taken care of at the top of the flow rather than the eventuality of metrics - legit or not - arriving at your Graphite host(s). RabbitMQ is OSS and pretty simple to get up and going without needing to mess around too much with configuration if you pull in the Management Plugin. Most of its config is opening necessary ports for operation.
However, this makes a simple metric publisher to Graphite topology a bit more complex for the task and firmly establishes a pub/sub model for how your metrics get to Graphite (but does come with a nice side benefit of allowing metrics in transit to not be lost potentially). It also adds yet another host to secure within operational reason.
To go way further, you could implement a log monitoring system to watch carbon-relay's listener.log file as it will write a line for every metric received and processed. At a high level you'd watch that log looking for exceptions to metrics you expect. Like if you have a server.cpu.load metric, you'd expect to see those getting processed, but a metric posted called foo.bar.value is not valid. As a response to such an event, you could simply wipe the corresponding directory Whisper creates for the invalid namespace (if you use Whisper for storage).
Hardening Graphite's carbon-relay and carbon-cache are fine and a smart thing to do, but do not forget it's just as concerning over who can access your Graphite webapp or graphite-api to get those metrics out.
