This is a routing question. I want to set up a block of IPs (1.1.1.0/24) in the following configuration:
/(eth1)-- (Main (1.1.1.3-1.1.1.253)
Router (1.1.1.1) --(eth0)(Load balancer or Router) (1.1.1.2)
\(eth2)-- (Alt (1.1.1.3-1.1.1.253)
With the Load balancer or Router (1.1.1.2) taking all traffic going to 1.1.1.0/24 and for ports 1-50000 forwarding those packets over eth1 to the main servers. But for requests from 50001-65535 forward that packet to eth2.
The trickiest part of this is the network Main and the Network Alt both need to have the same IP addresses.
The idea being a server in Main with IP (1.1.1.12) and a server in Alt with the same IP (1.1.1.12) can both be running, and make a request out of the network (say using port 3115 and port 53115) to the same target (like a web server). That web server would see two requests come in from the same IP address, just from two different ports, and would respond to both, and those packets would go back, one to each of the two servers (3115 would go to the server on the main network, 53115 response would go to the server on Alt network).
Both servers would need to be run without ever knowing the other exists.
Actually, in a best case scenario I wouldn't even want to need the split to happen based on ports. Ideally you would just have Alt and Main. If a request comes in that is not a response, it gets routed to eth1 (main) (so most traffic). The server Alt though could make a request, and that would go out showing from 1.1.1.12, as that's what the IP is of the machine in ALT. it would get routed back to eth2, the response to the packet, even though there is also a 1.1.1.12 on eth1. I wasn't thinking this situation was possible, which is why I figured that segregating based on port was the best that could be done.