You can block/Allow based on IPs as well. In the man Page http://linux.die.net/man/8/iptables
-s, --source [!] address[/mask]
Source specification. Address can be either a network name, a hostname (please note that specifying any name to be resolved with a
remote query such as DNS is a really bad idea), a network IP address
(with /mask), or a plain IP address. The mask can be either a network
mask or a plain number, specifying the number of 1's at the left side
of the network mask. Thus, a mask of 24 is equivalent to
255.255.255.0. A "!" argument before the address specification inverts the sense of the address. The flag --src is an alias for this option.
-d, --destination [!] address[/mask]
Destination specification. See the description of the -s (source) flag for a detailed description of the syntax. The flag --dst is an
alias for this option.
So you can do something like
sudo iptables -A INPUT -d [serverip]/32 -p tcp --dport 80 -j ACCEPT
will only allow on port 80 to the specified IP or you can block listening on the other ip's
For source address the flag is -s