Connection timeout error on Mule http outbound end point while connecting to external server

Question

I have 2 mule applications running on a single machine in which the app A sends a get request to app B and app B connects to an external http app to get the info. Basically the app B is a redirection application and it throws the connect timed out exception (java.net.SocketTimeoutException) many times in a day. As per my observation this happens whenever the application(B) is idle for sometime and then if it receives a request it throws the connection time out for the first request while connecting to external server on outbound http endpoint and the subsequent requests work fine.This happens only in one environment where I dont have the access to debug, and I havent encountered this issue in any other env's though there are two other environments with the same exact setup.

Any help/suggestions/guidence is appreciated in debugging the issue. Code

<flow name="Main" >
        <https:inbound-endpoint exchange-pattern="request-response"
            host="${http.listenHost}" port="${support-1.0.https.port}"
            path="${service.root}"  encoding="UTF-8"
             doc:name="HTTP" connector-ref="HTTPSConnector">
            <!-- NB: SECURITY TURNED OFF BY DEFAULT AS YOU WILL NEED TO ADD RECORDS 
                TO THE DATABASE IN ORDER TO USE THIS -->
            <!-- <mule-ss:http-security-filter realm="mule-realm"/> -->
        </https:inbound-endpoint>

        <custom-interceptor class="xyz.BasicLogger">
            <spring:property name="outermost" value="true" />
        </custom-interceptor>
        <set-session-variable variableName="operation" value="#[message.inboundProperties['http.method']]" doc:name="Operation"/>

        <set-variable variableName="_logging.text" value="Incoming Payload" doc:name="Logging" />
        <component doc:name="Payload Logger">
            <spring-object bean="idPayloadLogger" />
        </component>

        <set-session-variable variableName="http.status"
            value="200" doc:name="HTTP Status" />

        <flow-ref name="Cache" doc:name="Cache"/>

        <http:response-builder status="#[sessionVars['http.status']]" contentType="application/json" doc:name="HTTP Response Builder">
            <http:header name="consumerTransId" value="#[sessionVars['tx.id']]"/>
            <http:header name="http.status" value="#[sessionVars['http.status']]"/>
            <http:header name="Content-Type" value="application/json;charset=utf-8"/>
        </http:response-builder>

    </flow>
    <flow name="Cache">
        <expression-transformer expression="#[payload]" doc:name="Create key for cache"/>
        <custom-interceptor doc:name="PayloadCache"    class="xyz.ehcache.PayloadCache"/>
        <flow-ref name="GetResult" doc:name="GetResult"/>
    </flow>

    <sub-flow name="GetResult">
        <choice doc:name="Choice">
            <when expression="sessionVars.operation =='GET'">
                <processor-chain doc:name="GET">
                    <set-variable variableName="_logging.calloutName " value="Registry GET" doc:name="_logging.calloutName "/>
                    <component doc:name="Java"><spring-object bean="idTimerLogger"/></component>
                    <http:outbound-endpoint exchange-pattern="request-response" host="${registry.host}" port="${registry.port}"
                     path="${registry.path}/#[message.inboundProperties.'http.relative.path']?#[message.inboundProperties.'http.query.string']" method="GET" doc:name="HTTP" responseTimeout="10000"/>
                    <component doc:name="Java"><spring-object bean="idTimerLogger"/></component>
                </processor-chain>

            </when>
            <when expression="sessionVars.operation =='DELETE'">
                <processor-chain doc:name="DELETE">
                    <set-variable variableName="_logging.calloutName " value="Registry DELETE" doc:name="_logging.calloutName "/>
                    <component doc:name="Java"><spring-object bean="idTimerLogger"/></component>
                    <http:outbound-endpoint exchange-pattern="request-response" host="${registry.host}" port="${registry.port}" path="${registry.path}/#[message.inboundProperties.'http.relative.path']?#[message.inboundProperties.'http.query.string']"
                     method="DELETE" doc:name="HTTP"/>
                    <component doc:name="Java"><spring-object bean="idTimerLogger"/></component>
                </processor-chain>
            </when>
            <otherwise>
                <processor-chain doc:name="Unsupported Verb">
                    <set-variable variableName="_error.code" value="2" doc:name="Error code"/>
                    <custom-transformer class="xyz.eai.utility.transformation.ErrorMessageTransformer" doc:name="Java"/>
                    <json:object-to-json-transformer doc:name="Object to JSON"/>
                </processor-chain>
            </otherwise>
        </choice>
        <object-to-string-transformer doc:name="Object to String"/>
    </sub-flow>

</mule>

and the error is `

Exception stack is:
1. connect timed out (java.net.SocketTimeoutException)
  java.net.PlainSocketImpl:-2 (null)
2. The host did not accept the connection within timeout of 2000 ms (org.apache.commons.httpclient.ConnectTimeoutException)
  org.apache.commons.httpclient.protocol.ReflectionSocketFactory:155 (null)
3. Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=http://xyz.ca.com:8080/search?param1=xyz&param2=true, connector=HttpConnector
{
  name=connector.http.mule.default
  lifecycle=start
  this=59c02565
  numberOfConcurrentTransactedReceivers=4
  createMultipleTransactedReceivers=true
  connected=true
  supportedProtocols=[http]
  serviceOverrides=<none>
}
,  name='endpoint.http.xyz.com.8080.search.param1.xyz.param2.true', mep=REQUEST_RESPONSE, properties={isActive=true, http.method=GET, exceptionOnMessageError=true, param1=xyz}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, timeout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=10000, endpointEncoding=UTF-8, disableTransportTransformer=false}. Message payload is of type: GetMethod (org.mule.api.transport.DispatchException)
  org.mule.transport.http.HttpClientMessageDispatcher:151 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/transport/DispatchException.html)`

################3

Added a until-successful loop to the outbound http endpoint in my app B and is working if I am accessing it separately. But when the app A tries to communicate with app B , even before the app B tried to connect to the external server and returns the response , the http outbound endpoint on app A is throwing org.mule.api.transport.DispatchException with message : Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=https://application-b/xyz, connector=HttpsConnector and when the app B completes its retries and tries to send back the response to app A its getting the error : org.mule.transport.http.HttpsMessageProcessTemplate: Exception sending http response after error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Software caused connection abort: socket write error

Answer 1

You need to use a untilSuccess on your HTTP/HTTPS connector

<until-successful maxRetries="5" synchronous="true" doc:name="Until Successful">
            <http:request config-ref="HTTP_Request_Configuration" path="/test" method="POST" doc:name="HTTP"/>
        </until-successful>