We run our application on ec2 and use Munin to monitor various metrics. We do a bi-weekly fresh deploy on new ec2 + rds instances. After our most recent fresh deploy, we noticed a 4x reduction in the rate packets are coming through our firewall, but only on one of two servers. The odd thing is that the eth0 traffic bits per second measurement is normal for both machines.
What could cause a drop in the number of packets sent for the same number of bits in/out?
System details: We have two load-balanced application servers in different AWS availability zones, both running Ubuntu 10.04. The majority of our network I/O is caused by search indexing (triggered) via cron, where the app server pulls in a large amount of data from the database.
Here are the graphs for the first app server. The new instance started graphing at ~9:30pm on the 27th, exactly when the firewall throughput drops.
And the unaffected app server for comparison: 
