It appears that the issue turned out to be DNS related. When I changed the Availability Group Endpoint URL to be the internal IP address instead of the FQDN, the problem was resolved. So for anyone else who hits this, that is something to try (could apply to read only routing as well). A couple SQL statements that might help:
Query Current Settings
SELECT ag.name as "Availability Group", ar.replica_server_name as "When Primary Replica Is", rl.routing_priority as "Routing Priority", ar2.replica_server_name as "RO Routed To", ar.secondary_role_allow_connections_desc, ar2.read_only_routing_url ,*
FROM sys.availability_read_only_routing_lists rl inner join sys.availability_replicas ar on rl.replica_id = ar.replica_id inner join sys.availability_replicas ar2 on rl.read_only_replica_id = ar2.replica_id inner join sys.availability_groups ag on ar.group_id = ag.group_id ORDER BY ag.name, ar.replica_server_name, rl.routing_priority
Update Endpoint URL
ALTER AVAILABILITY GROUP [AvailabilityGroupName]
MODIFY REPLICA ON 'ReplicaName' WITH (ENDPOINT_URL = 'TCP://<Internal IP>:<Port>')
I'm not yet sure why this was intermittent or why my network's DNS wasn't resolving this to an internal address and instead it was being picked up by a wildcard Route53 DNS CNAME record for the domain. If I figure that out, I'll post it just in case it's helpful. Thanks Bruno for the help earlier!