Transiting of routed IP traffic across VPCs is not supported by peering connections.
A peering connection does not give you the same level of connectivity that you might expect from a VPN or other network-to-network interconnections you may be familiar with.
Consider this example from the documentation (with minor reformatting for clarity), which -- admittedly -- does not describe your situation precisely, but contributes to an understanding of the larger issue at hand.
You have a VPC peering connection between VPC A and VPC B (pcx-aaaabbbb), and between VPC A and VPC C (pcx-aaaacccc).
There is VPC peering connection between VPC B and VPC C.
You cannot route packets directly from VPC B to VPC C through VPC A.
http://docs.aws.amazon.com/AmazonVPC/latest/PeeringGuide/invalid-peering-configurations.html#transitive-peering
That's part of the design of VPC peering -- it is a feature affecting only the instances in the two VPCs -- they can access each other. Other resources in the VPCs, such as NAT and Internet Gateways, VPC Service Endpoints (for S3), Direct Connect, and Hardware VPN are not exposed via peering.
tl;dr: VPC peering doesn't enable any IP routing or connectivity beyond the EC2 instances in the directly-peered VPCs.
So, the above is not a perfect explanation, since in the documented example, all three VPCs are of course in the same region, but similar principles apply to your configuration.
The issue is not so much that VPC-1 can't reach VPC-4 via the tunnel to VPC-3 (the traffic may or may not arrive), but rather than VPC-4 has absolutely no way to send a reply. VPC-4 can't transit VPC-3 on its way to something external to VPC-3 (VPC-1).
Similarly, this was the motivation for my question about connectivity between VPC-2 and VPC-3. VPC-2 can't transit VPC-1 on its way to accessing addresses in VPC-3.
The incoming traffic bound for the remote network never makes it to the tunnel server, because there is no mechanism for transiting across VPCs, and traffic to the tunnel server -- though perhaps not clearly documented as such -- is still a case of transiting.
You'd have the same issue if you had a VPC Hardware VPN between VPC-1 and an external corporate datacenter. That VPN connection would not provide access from the external data center to anything outside the borders of VPC-1, regardless of any peering. The same is true for AWS Direct Connect.
Analyzing this only from the perspective of the API/Console interface, consider how you send traffic across a peering connection: You create a route sending the foreign IP prefix from one VPC to another by creating a route table entry using peering connection ID as the target.
But what happens when that traffic arrives at the peered VPC? Which route table is applied to it?
That's a trick question, because no exposed route table applies to traffic coming in over a VPC peering connection -- the only routes available to that incoming traffic is the implicit local route that makes instances accessible.
Traffic addressed to something other than an instance (or service running on an instance, like RDS) -- when it arrives at the peered VPC -- has nowhere to go, and is discarded.
The logical workaround would be to try to route traffic across a peering connection to an instance acting as a router... much the same as the way you route traffic to a NAT Instance -- by setting the destination in the route table to the instance-id or its associated Elastic Network Interface (ENI).
At the bottom of the documentation page mentioned above, we find a one-sentence limitation on a practice like this:
Similarly, if VPC A has a NAT device that provides Internet access to instances in private subnets in VPC A, instances in VPC B cannot use the NAT device to access the Internet.
This is probably less clear than it could be to those who are unfamiliar with typical VPC configurations, but this limitation actually explains more precisely why the configuration proposed in the original question doesn't work.
You configure a subnet to use a NAT instance by creating a route in the subnet's route table to send a specific foreign prefix (typically 0.0.0.0/0, but it could be a subset) to the instance itself, or its Elastic Network Interface, by ID.
But within the region, if you try to route traffic to the ENI of an instance in a peered VPC via the route tables, that fails:
Error: route table rtb-xxxxxxxx and interface eni-xxxxxxxx belong to different networks
(Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue;
Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
So, the intuitive workaround is also not available.
The only three solutions I can find are these:
More tunnels, like the you have already built between your VPC-1 and VPC-3. You essentially need a full mesh of OpenVPN (or another tunneling solution) among all the VPCs that need to exchange traffic, other than the ones you have peered. This is a the solution I have implemented. In each availability zone, I have an OpenVPN server that has tunnels to all other availability zones in foreign regions. You wouldn't necessarily need one to each availability zone -- just one for each VPC would satisfy the requirement, but tunnels among each pair of availability zone will provide resiliency against isolation of an entire VPC due to the loss of one availability zone.
Source NAT from the tunnel server. This one is terribly messy. You have a tunnel between VPC-1 and VPC-3, and VPC-3 is peered to VPC-4, so you could route the traffic from VPC-1 to the tunnel server in VPC-1 via the VPC route tables in VPC-1, and then send it across the tunnel to VPC-3, where it could assume the source IP address of the tunnel instance in VPC-3 via iptables configuration. Anything the tunnel machine in VPC-3 could access in VPC-4 would then become accessible to anything in VPC-1 that was allowed to access the tunnel server.
Service proxy or NAT servers in VPC-3, accessible from VPC-1, that either do layer 4 forwarding of TCP connections or both source and destination NAT of traffic between VPC-1 addressess and VPC-4 addresses, so that VPC-4 sees the traffic as coming from a VPC-3 address and VPC-1 sees the traffic is coming from a VPC-3 address. This is similar to the above, but provides better access control via security groups, since it's not all-or-nothing access as would be required by the previous option... but still a messy solution.
The cleanest configuration would be a full-mesh of tunnels among all your VPCs (or all your availability zones, except for availability zones within the same VPC or any that are directly peered).
