I use collectd (http://collectd.org/) to monitor bandwidth usage on an interface+direction basis amongst other things and it has a module that can report in a more fine grained manner based on iptables rules (http://collectd.org/wiki/index.php/Plugin:IPTables), so that might be a solution for you.
Setup rules that match what you wish to monitor (i.e. a given IP address destination/source and/or a particular port) that do nothing but match and ask collectd to monitor the packet and byte counts for those new rules. If you already have rules for each interface and/or port (because your firewall rejects by default and only allws specific matches through) just make sure these rules have identifiable comments instead of creating new ones.
Once collectd is monitoring things and populating the rrd files you can use and rrdtool or any wrapper for it to visualise the data. I currently use a slightly modified version of http://haroon.sis.utoronto.ca/rrd/scripts/ though there are many other options out there.