I'm assuming Linux because of the interface names and the tag of "iproute2".
This is not working, because simply specifying a default route will mean that all outgoing traffic is going to pass through that interface, even responses for requests that have come from the other interface!
In your case, you're saying you have public IP's, yet the two directly connected interfaces are with private IP addresses, so therefore I assume that means you have NAT going on. Whenever you have NAT, you can't use this type of triangular routing, where the incoming and outgoing packets take different paths, because the packets have been changed on the way in by the NAT box, and need to be changed in the same way on the way out. Your requesting client ends up receiving packets from an IP address it's not expecting and does not recognise the response.
What you need to do is to tell the operating system that any packets with IP addresses originating on your interface in 172.31.128.0/24 towards the internet need to hit the gateway at 172.31.128.1, and correspondingly for the 172.31.129.0/24 subnet.
That can be accomplished like this:
ip route add 172.31.129.0/24 dev eth0 src 172.31.129.XXX table T1
ip route add default via 172.31.129.1 table T1
ip route add 172.31.128.0/24 dev eth1 src 172.31.128.XXX table T2
ip route add default via 172.31.128.1 table T2
ip rule add from 172.31.129.XXX table T1
ip rule add from 172.31.128.XXX table T2
You will need to replace XXX with the IP address of your server on those local subnets.
That should get you started. You will find more on this topic in section 4.2 of the Linux Advanced Routing & Traffic Control HOWTO.