0

I have a BPS process that rapidly performs a lot of calls to an ESB proxy. The ESB proxy calls an ESB API, which calls the API of our own product.

Often, the BPS process pauses for exactly 3 minutes while waiting for a response from the ESB. After the 3 minutes, everything continues as if nothing happened. Using Wireshark and some additional Log mediators, I have discovered the following:

  • The hanging occurs in the ESB proxy, right before the out sequence.
  • According to Wireshark, the ESB API has already sent its response to the ESB proxy when the hanging occurs.

The fact that this hanging takes exactly 3 minutes should be a hint. I searched the config files for anything specifying 180 seconds, but the only ones are the http socket timeouts and the http transportReceiver's threadKeepAliveTime. If I reduce those, the BPS throws a p2p communication error instead of continuing. To me, this indicates that something somewhere in the background is still causing a 3 minute delay for some calls.

By the way, the hanging does not occur for some specific calls only. I have been testing several times, performing the exact same calls each time, and I cannot predict which call will hang or even if any call will hang at all.

Community
  • 1
  • 1
curledUpSheep
  • 403
  • 3
  • 11
  • 1
    Does BPS disconnection after receive response ? – simar Jul 10 '17 at 20:18
  • 1
    What type of requests do you send: PUT, POST? Are you sending them without a request body? – thanuja Jul 11 '17 at 05:00
  • @simar: After the hanging, the BPS receives the correct response for the call and just continues the process as if nothing happened. I have validated this by enabling tracing in the BPS. – curledUpSheep Jul 11 '17 at 08:46
  • @thanuja I am sending POST requests with a body – curledUpSheep Jul 11 '17 at 08:46
  • 1
    Try this if you haven't done already: https://stackoverflow.com/questions/34722299/apache-synapse-wso2-api-no-response – thanuja Jul 11 '17 at 09:24
  • I have tried using a Call mediator and Loopback mediator instead of the Send mediator at the end of my in sequence. Didn't make a difference. Added some additional logging between the Call and Loopback mediator and this confirmed that the hanging occurs in the Call mediator. – curledUpSheep Jul 11 '17 at 10:31
  • Have you found a solution? I Have the same :( – user3345547 Jun 13 '20 at 22:01
  • @user3345547 My response below shows the solution that worked for me – curledUpSheep Jun 15 '20 at 09:40

4 Answers4

1

I would suggest you add at least a <send /> mediator in your faultSequence. Perhaps also add a simple makefault mediator (don't make it too complex just yet - just a static error response will do - at this stage you don't want faults originating within the faultSequence)

I suspect some error from the backend service or in the out sequence or maybe even faultSequence is not being handled - which means instead of sending a response back to your caller, the proxy hangs until time out instead

PS: It really helps if you post the source code of your proxy too

Jang-Vijay Singh
  • 732
  • 3
  • 11
  • I already had a Log mediator in my Fault sequence, now added another Log mediator and a Send mediator but I see nothing in the logs. I also tried enabling tracing for the ESB, but I am not able to reproduce the issue in that case. It seems that the slowdown from the huge amount of logging prevents the issue from occurring. I will need to check if my company allows me to share the actual source code. – curledUpSheep Jul 11 '17 at 10:02
  • 1
    Maybe not the actual source code - you can create a [simple "shell" proxy like this](https://github.com/jvsingh/wso2-toolchain/blob/develop/Issues/Issue0001Fault/Issue0001Fault/src/main/synapse-config/proxy-services/Issue0001Proxy.xml) that only calls your ESB API. See if you can reproduce the issue in that. If not, then the additional code you have in your proxy might have problems.. – Jang-Vijay Singh Jul 11 '17 at 20:44
1

I had the same experience sometimes ago when I tried to call PUT or POST methods without a request body. But, by default, WSO2 ESB expects a message body to send the request to backend server when the http method is POST or PUT.

I had to either send the request with a request body (at least an empty JSON: {}) or use the property FORCE_POST_PUT_NOBODY in your APIs.

Read this post for more details on how to use FORCE_POST_PUT_NOBODY property.

thanuja
  • 546
  • 1
  • 8
  • 22
  • In this case, I am sending POST requests with a body and I actually see the requests being passed on to the backend service and the backend service responding (checked this using Wireshark). So don't think this will be the issue. – curledUpSheep Jul 11 '17 at 08:52
1

If you enable the wire log in WSO2 you will realize NGIX is using HTTP 1.0 client in the request.

In order to fix that change your NGINX mapping to use "proxy_http_version 1.1;" and the problem will be sorted.

Cheers.

Emmerson
  • 61
  • 3
0

I ended up getting WSO2 support involved in this. Apparently, I was hitting some very rare edge case which was very hard for the WSO2 engineers to reproduce.

In the end, the fix turned out to be adding the following property just before the send mediator in the proxy service:

<property name="NO_KEEPALIVE" value="true" scope="axis2"/>
curledUpSheep
  • 403
  • 3
  • 11