-1

First of all, apologies if I missed giving more information on the issue. I'm new to pingfederate. I tried to put all the steps followed below.

  1. I have setup a single node k3s cluster on EC2 centos instance, there is also a traefik ingress deployed along with the k3s cluster.
  2. I deployed the pingfederate using the 20-kubernetes/07-license-as-secret since I have an existing pingfederate license.
  3. For testing purpose, I customized the pingidentiy-server-profiles/baseline/pingfederate profile by following how to modify profile using GitHub repository method.
  4. I also created an ingress for the pingfederate service using the 10-ingress/pingfederate-standalone-ingress. I have modified the hostname to the one which I have created.

When I try to access the application using the ingress hostname, I get the internal server error.

Note: I have already tried the same setup on my local k3s cluster but without an ingress and just using port-forward it works, I’m able to login to the console.

Troubleshooting steps done so far:

Enabled logs to DEBUG mode following this link and deployed the pingfederate server. There is a ClosedChannelException in the org.eclipse.jetty.io.WriteFlusher. Below is the server.log leading to the error I mentioned.

2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.AbstractEndPoint] close SocketChannelEndPoint@71385c8f{/10.42.0.1:36694<->/10.42.0.15:9999,OSHUT,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->SslConnection@271ff47b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint@30fe3126{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=-,flush=-,to=4/30000}=>HttpConnection@106b47be[p=HttpParser{s=CLOSE,0 of -1},g=HttpGenerator@662996bf{s=START}]=>HttpChannelOverHttp@2d6a2ba{r=1,c=false,c=false/false,a=IDLE,uri=null,age=0}
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill] EatWhatYouKill@63ccb1b2/SelectorProducer@1165a952/IDLE/p=true/QueuedThreadPool[qtp1642619282]@61e86192{STARTED,1<=6<=10,i=3,r=1,q=1}[ReservedThreadExecutor@7204d98f{s=0/1,p=1}][pc=0,pic=0,pec=24,epc=3744]@2020-12-29T17:43:06.124489Z tryProduce true
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.AbstractEndPoint] close(null) SocketChannelEndPoint@71385c8f{/10.42.0.1:36694<->/10.42.0.15:9999,OSHUT,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->SslConnection@271ff47b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint@30fe3126{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=-,flush=-,to=4/30000}=>HttpConnection@106b47be[p=HttpParser{s=CLOSE,0 of -1},g=HttpGenerator@662996bf{s=START}]=>HttpChannelOverHttp@2d6a2ba{r=1,c=false,c=false/false,a=IDLE,uri=null,age=0}
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.ManagedSelector] updateable 0
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.ManagedSelector] updates 0
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.util.thread.strategy.EatWhatYouKill] EatWhatYouKill@63ccb1b2/SelectorProducer@1165a952/IDLE/p=true/QueuedThreadPool[qtp1642619282]@61e86192{STARTED,1<=6<=10,i=3,r=1,q=1}[ReservedThreadExecutor@7204d98f{s=0/1,p=1}][pc=0,pic=0,pec=24,epc=3744]@2020-12-29T17:43:06.124474Z m=EXECUTE_PRODUCE_CONSUME t=CEP:SocketChannelEndPoint@71385c8f{/10.42.0.1:36694<->/10.42.0.15:9999,OSHUT,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->SslConnection@271ff47b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint@30fe3126{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=-,flush=-,to=4/30000}=>HttpConnection@106b47be[p=HttpParser{s=CLOSE,0 of -1},g=HttpGenerator@662996bf{s=START}]=>HttpChannelOverHttp@2d6a2ba{r=1,c=false,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING/BLOCKING
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.ChannelEndPoint] doClose SocketChannelEndPoint@71385c8f{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=FI,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->SslConnection@271ff47b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint@30fe3126{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=-,flush=-,to=4/30000}=>HttpConnection@106b47be[p=HttpParser{s=CLOSE,0 of -1},g=HttpGenerator@662996bf{s=START}]=>HttpChannelOverHttp@2d6a2ba{r=1,c=false,c=false/false,a=IDLE,uri=null,age=0}
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.ChannelEndPoint] Key interests updated 1 -> 0 on SocketChannelEndPoint@71385c8f{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=FI,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->SslConnection@271ff47b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint@30fe3126{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=-,flush=-,to=4/30000}=>HttpConnection@106b47be[p=HttpParser{s=CLOSE,0 of -1},g=HttpGenerator@662996bf{s=START}]=>HttpChannelOverHttp@2d6a2ba{r=1,c=false,c=false/false,a=IDLE,uri=null,age=0}
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.ManagedSelector] Selector sun.nio.ch.EPollSelectorImpl@aa7427f waiting with 2 keys
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.FillInterest] fillable FillInterest@21d544d7{SSLC.NBReadCB@271ff47b{SslConnection@271ff47b::SocketChannelEndPoint@71385c8f{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=FI,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->SslConnection@271ff47b{NOT_HANDSHAKING,eio=-1/-1,di=-1,fill=IDLE,flush=IDLE}~>DecryptedEndPoint@30fe3126{/10.42.0.1:36694<->/10.42.0.15:9999,CLOSED,fill=-,flush=-,to=4/30000}=>HttpConnection@106b47be[p=HttpParser{s=CLOSE,0 of -1},g=HttpGenerator@662996bf{s=START}]=>HttpChannelOverHttp@2d6a2ba{r=1,c=false,c=false/false,a=IDLE,uri=null,age=0}}}
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.WriteFlusher] ignored: WriteFlusher@655f6d8{IDLE}->null
java.nio.channels.ClosedChannelException: null
    at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:492) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:353) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:215) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:192) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:175) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doClose(SslConnection.java:1197) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:220) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:192) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:175) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractConnection.close(AbstractConnection.java:248) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:254) ~[jetty-server-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.FillInterest] onClose FillInterest@21d544d7{null}
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.ManagedSelector] Wakeup ManagedSelector@1b868ef0{STARTED} id=0 keys=1 selected=0 updates=0
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.util.thread.QueuedThreadPool] queue org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@68bc5d59
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.WriteFlusher] ignored: WriteFlusher@43cde001{IDLE}->null
java.nio.channels.ClosedChannelException: null
    at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:492) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:353) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:192) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:175) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractConnection.close(AbstractConnection.java:248) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:254) ~[jetty-server-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) ~[jetty-util-9.4.18.v20190429patch5.jar:9.4.18.v20190429patch5]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.io.FillInterest] onClose FillInterest@6bc9a8a7{null}
2020-12-29 17:43:06,124  DEBUG [org.eclipse.jetty.server.HttpConnection]

Please shout if more details are required. Any assistance will be much appreciated. Thanks in advance.

Bhagya
  • 1
  • 1
  • Have you configured the server at all? Or is it a raw unzip that has been started? – Andrew K. Dec 28 '20 at 15:12
  • I have configured the server, customized the PingIdentity server profile, and deployed it for the pingfederate server. – Bhagya Dec 28 '20 at 21:15
  • Set the logs to debug in the profile and redeploy. My guess is that there's an SNI issue or some other naming/addressing problem, but you're not giving much info here. May want to open a support case and send in logs and your profile – Andrew K. Dec 28 '20 at 21:27
  • I'm able to make the REST API call now using the IP of the pingfederate service rather than the instance IP. But I still get the `internal server error` when I try to access the pingfederate console. I set the logs to debug mode. I do not have much information from that. I followed this [link](https://support.pingidentity.com/s/article/Configuring-DEBUG-logging-in-PingFederate-8-2-and-later) to enable the logs in DEBUG mode. I have already created a support case. I will attach the logs to the same ticket. – Bhagya Dec 29 '20 at 12:11
  • Either return here and answer your own question with your resolution, or delete the question. – Andrew K. Dec 29 '20 at 13:31
  • @AndrewK. Sorry if you felt the information was not complete. I tried to add more details on the steps followed and the issue now. I hope it helps. – Bhagya Dec 29 '20 at 19:18
  • No need to apologize! Your information is much more complete now, but I'm still not enough of a devops genius to help is more likely the problem. If I had to guess, this issue sounds very much like HTTPS issues like SNI, or similar, and it could be amplified by the ingress. I'll point one of the Ping devops gurus that I know over here. – Andrew K. Dec 29 '20 at 22:08

1 Answers1

0

I ran into a similar issue and what worked for me was to add hostAliases with hostnames that I was publicly exposing, in th YAML defining the pod. Also the certificate provided needed to include the hostnames for the SNI entries.

John DaSilva
  • 1
  • 1
  • The certificate provided includes the hostnames already. I updated the pods to add hostAliases but still no luck :( – Bhagya Jan 02 '21 at 11:18
  • I don't think your log fragment has enough detail as to what is going on with the request. Do you see the request coming into the request log file?This would tell you if you are getting past the controller to the server? – John DaSilva Jan 05 '21 at 13:40