You seem not to have interface routes which are applied when you set IP on your interface.
They should look like this
192.168.4.0/24 dev eth1 proto static src 192.168.4.1 metric xx
192.168.5.0/24 dev eth2 proto static src 192.168.5.1 metric xx
You have two default routes instead. When the packet is created the kernel match packet destination through routing table and take first match. You have only two routes and they both are default (means 0.0.0.0/0), so first route in routing table always wins.
You seem to try diverting traffic by ip rules, but this will not work because source address will be applied after route will be chosen.
Imagine linux trying to send packet to 192.168.5.2. It should pick an outgoing address, but this depends on outgoing interface. To pick outgoing interface it have to consult routing table. It would look in "secondary" because of rule set, but source address is not yet set so it will look in "default". Routing table has only default route and first matches.
These rules can be set for changing routes for packets flowing through, not packets which were originated.