Moqui connection refused on HTTPS behind Nginx

Question

This post refers.

OK, I've spent the last couple of days trying to get an ebextensions config together to configure HTTPS encryption on Nginx for a single EC2 instance with the creation of a LetsEncrypt certificate. That all seems to be working according to the logs (will publish config on other post once fully working). But I'm getting some weird behaviour when trying to access Moqui through HTTPS. HTTP is working fine but as soon as I switch to HTTPS, the correction is refused. I get the following from EB logs:

/var/log/web-1.log
-------------------------------------
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.recover(XaResourceRecoveryManager.java:67) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:451) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.performRecovery(TransactionServiceImp.java:490) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.access$000(TransactionServiceImp.java:56) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp$1.alarm(TransactionServiceImp.java:471) [transactions-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:95) [atomikos-util-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:82) [atomikos-util-4.0.4.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:843) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc3.AbstractJdbc3Connection.createStatement(AbstractJdbc3Connection.java:231) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc2.AbstractJdbc2Connection.createStatement(AbstractJdbc2Connection.java:310) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:341) ~[postgresql-9.3-1102.jdbc41.jar:?]
    ... 12 more
[33m18:36:04.327  WARN   Atomikos:1 atomikos.r.x.XaResourceRecoveryManager[m Error while retrieving xids from resource - will retry later...
org.postgresql.xa.PGXAException: Error during recover
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:368) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at com.atomikos.datasource.xa.RecoveryScan.recoverXids(RecoveryScan.java:32) ~[transactions-jta-4.0.4.jar:?]
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.retrievePreparedXidsFromXaResource(XaResourceRecoveryManager.java:158) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.recover(XaResourceRecoveryManager.java:67) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:451) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.performRecovery(TransactionServiceImp.java:490) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.access$000(TransactionServiceImp.java:56) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp$1.alarm(TransactionServiceImp.java:471) [transactions-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:95) [atomikos-util-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:82) [atomikos-util-4.0.4.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:843) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc3.AbstractJdbc3Connection.createStatement(AbstractJdbc3Connection.java:231) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc2.AbstractJdbc2Connection.createStatement(AbstractJdbc2Connection.java:310) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:341) ~[postgresql-9.3-1102.jdbc41.jar:?]
    ... 12 more
[33m18:36:14.327  WARN   Atomikos:1 atomikos.r.x.XaResourceRecoveryManager[m Error while retrieving xids from resource - will retry later...
org.postgresql.xa.PGXAException: Error during recover
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:368) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at com.atomikos.datasource.xa.RecoveryScan.recoverXids(RecoveryScan.java:32) ~[transactions-jta-4.0.4.jar:?]
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.retrievePreparedXidsFromXaResource(XaResourceRecoveryManager.java:158) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.recover(XaResourceRecoveryManager.java:67) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:451) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.performRecovery(TransactionServiceImp.java:490) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.access$000(TransactionServiceImp.java:56) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp$1.alarm(TransactionServiceImp.java:471) [transactions-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:95) [atomikos-util-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:82) [atomikos-util-4.0.4.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:843) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc3.AbstractJdbc3Connection.createStatement(AbstractJdbc3Connection.java:231) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc2.AbstractJdbc2Connection.createStatement(AbstractJdbc2Connection.java:310) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:341) ~[postgresql-9.3-1102.jdbc41.jar:?]
    ... 12 more
[33m18:36:24.328  WARN   Atomikos:1 atomikos.r.x.XaResourceRecoveryManager[m Error while retrieving xids from resource - will retry later...
org.postgresql.xa.PGXAException: Error during recover
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:368) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at com.atomikos.datasource.xa.RecoveryScan.recoverXids(RecoveryScan.java:32) ~[transactions-jta-4.0.4.jar:?]
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.retrievePreparedXidsFromXaResource(XaResourceRecoveryManager.java:158) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.recover(XaResourceRecoveryManager.java:67) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:451) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.performRecovery(TransactionServiceImp.java:490) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.access$000(TransactionServiceImp.java:56) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp$1.alarm(TransactionServiceImp.java:471) [transactions-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:95) [atomikos-util-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:82) [atomikos-util-4.0.4.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:843) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc3.AbstractJdbc3Connection.createStatement(AbstractJdbc3Connection.java:231) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc2.AbstractJdbc2Connection.createStatement(AbstractJdbc2Connection.java:310) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:341) ~[postgresql-9.3-1102.jdbc41.jar:?]
    ... 12 more
[33m18:36:34.328  WARN   Atomikos:1 atomikos.r.x.XaResourceRecoveryManager[m Error while retrieving xids from resource - will retry later...
org.postgresql.xa.PGXAException: Error during recover
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:368) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at com.atomikos.datasource.xa.RecoveryScan.recoverXids(RecoveryScan.java:32) ~[transactions-jta-4.0.4.jar:?]
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.retrievePreparedXidsFromXaResource(XaResourceRecoveryManager.java:158) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.recovery.xa.XaResourceRecoveryManager.recover(XaResourceRecoveryManager.java:67) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:451) [transactions-jta-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.performRecovery(TransactionServiceImp.java:490) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp.access$000(TransactionServiceImp.java:56) [transactions-4.0.4.jar:?]
    at com.atomikos.icatch.imp.TransactionServiceImp$1.alarm(TransactionServiceImp.java:471) [transactions-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:95) [atomikos-util-4.0.4.jar:?]
    at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:82) [atomikos-util-4.0.4.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:843) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc3.AbstractJdbc3Connection.createStatement(AbstractJdbc3Connection.java:231) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.jdbc2.AbstractJdbc2Connection.createStatement(AbstractJdbc2Connection.java:310) ~[postgresql-9.3-1102.jdbc41.jar:?]
    at org.postgresql.xa.PGXAConnection.recover(PGXAConnection.java:341) ~[postgresql-9.3-1102.jdbc41.jar:?]
    ... 12 more

/var/log/nginx/error.log is empty and this is clearly a Moqui exception so Nginx is presumably receiving traffic correctly. It seems that Postgres is the underlying cause, but as I said, I can access everything fine through HTTP so it looks like Postgres connections are closed in Moqui when the request is coming over HTTPS which seems....unlikely.

Will appreciate any help so that I can figure out how to deploy Moqui on single EC2 instance. Will share the solution on the other post.

score 0 · Answer 1 · answered May 20 '19 at 09:15

One approach is to create a valve in Tomcat /etc/tomcat7/server.xml:

     <Valve className="org.apache.catalina.valves.RemoteIpValve"
            internalProxies=".*"
            remoteIpHeader="x-forwarded-for"
            remoteIpProxiesHeader="x-forwarded-by"
            protocolHeader="x-forwarded-proto" />

And use something like the following in your nginx set up.

location / {
 proxy_set_header Accept-Encoding "";
 proxy_set_header X-Forwarded-For $remote_addr;
 proxy_set_header Host $http_host;
 proxy_set_header X-Forwarded-Host $host;
 proxy_set_header X-Forwarded-Server $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_pass http://xx.xxx.xx.xxx:8080;
 proxy_http_version 1.1;
}

Thanks, Ronan. My custom config was not being loaded by Beanstalk so port 443 was never being configured correctly. — buzzdriving, May 22 '19 at 00:46

score 0 · Answer 2 · answered May 22 '19 at 00:45

Ignore this idiocy. The issue was that Nginx was not accepting my custom Nginx config for a standalone Java SE instance on Elastic Beanstalk so port 443 was never set up correctly in the first place. Still can't figure out why I was getting that log information back since if port 443 was closed, the request would never had reached Moqui.

Anyway, I have almost managed to get the configuration fully working. Will update here when done.

Moqui connection refused on HTTPS behind Nginx

2 Answers2