The problem of this model is that some (only some) clients cannot be reached. See the link below for more information about the model which is being described in the upcoming section.
Problem description
Both clients (1.1 and 1.2.3) can connect to the VPN server (1). Server (1) is not using client-to-client declaration in OpenVPN conf file.
Server (1) can ping both clients (1.1 and 1.2.3), both clients (1.1 and 1.2.3) can ping one another and can ping server (1). Local clients (1.1.1 and 1.1.2) behind router (1.1) behind NAT can ping one another and can ping the router (1.1). The same goes for the other netowrk (behind 1.2) where all clients (1.2.1, 1.2.2 and 1.2.3) can ping one another and can also ping the router (1.2). There are no problems so far.
Both routers (1.1 and 1.2) have properly set their static routes. Router in the first network (1.1) has no static routes set, they are pushed from the VPN server. Router in the second network (1.2) is not the VPN gateway therefore his routes are following:
network 192.168.1.0/24 gateway 192.168.2.103
network 192.168.10.0/24 gateway 192.168.2.103
Then the VPN client on the second network (1.2.3) has the routes pushed on him from the server again.
Back to the client stations reaching - now these clients behind NAT cannot be reached from one network to another network; some can, others cannot. To give some examples:
being logged in to a server (1), I may ping 1.1.1 but cannot ping 1.1.2.
being logged in to a client (1.1.1), I may ping 1.2.1 but may not ping 1.2.2.
Temporary fix
For some clients I can see a temporary fix while adding their statis routes, which can be done with Linux machines. Their routing table (for instance machine 1.1.2) may contain information just for network 1.1. Adding the static route for the other network (1.2) does make it work but this cannot be done with all clients.
Another very temporary fix is trying traceroute command from 1.2.2 to 1.1.2 which may actually reach the machine fine and then I can ping it for a few minutes. After a short while, the route is gone however.
None of these are permanent solutions though.
Some questions that arise
I need to note I have just recently exchanged a router 1.2 but all routes are set back as they were with the previous machine.
There are a few other questions that arise:
Is it a DNS problem? If so, why the ping command does not work with an IP, not a domain name?
How could traceroute work and ping not? There is no firewall to block it and even if it were, how could trceroute force the ping command to work for a short while?
Why some clients do not need to have static routes set and others need it for them to be pingable on the network?
May it be related to boot order of devices on the network? I have also tried to reboot/shutdown all of them but it wouldn't do anything. The idea was that the routes in memory would be cleared upon a reboot and new ones would be "taken" from the router.
The goal
The aim is to fix this for all clients on all networks to be reachable with no static routes being placed anywhere besides the router.