Failed to get destinations... timed-out and fallback disabled

Question

We use CAP to query data through ODATA V2 caller. But there is exception during my local env testing. I could successfully get the result in beginning. After I tested several times, it would raise exception every time. Please help me fix this issue.

Destination setup:

URL=https\://cxs-calmdevprovider-calmdev-sap-calm-imp-pjm-srv.cfapps.sap.hana.ondemand.com
Name=PJM-SRV-DEST
TrustAll=TRUE
ProxyType=Internet
Type=HTTP
Authentication=AppToAppSSO

Two applications bind to one XSUAA. And set the VCAP_SERVICE for local run.

        try {

            logger.debug("==> now execute query on Products");
            FilterExpression filter = new FilterExpression("project_guid", "eq",
                    ODataType.of(UUID.fromString("10c0b945-1ff5-4510-a160-24294bfe3b58")));

            CacheKey cKey = CacheKey.ofTenantAndUserIsolation();
            ODataQueryResult result = ODataQueryBuilder.withEntity("/odata/v2/CBLD_PROJECT_SRV", "CBLD_C_PROJECT_MS_TP")
                    .filter(filter).enableMetadataCache(cKey).build().execute("PJM-SRV-DEST");

            logger.debug("==> After calling backend OData V2 service: result: " + result);

            List<ProjectMilestone> projectMilestones = result.asList(ProjectMilestone.class);
            logger.info(projectMilestones.toString());

        } catch (Exception e) {
            logger.error("==> Exception calling backend OData V2 service for Query of Products: " + e.getMessage());
            ErrorResponse errorResponse =
                    ErrorResponse.getBuilder().setMessage("There is an error.  Check the logs for the details.")
                            .setStatusCode(500).setCause(e).response();
            return UpdateResponse.setError(errorResponse);
        }

Exception stack:

2019-08-20 14:05:03,446 DEBUG [http-nio-8081-exec-2] - [com.sap.calm.imp.tkm.srv.odata.handler.TaskServiceHandler] [tenant_id=05e65d26-f3e2-4937-9987-eb412f4cd732, component_id=7d44b23e-c4c1-42ba-9c60-0048c14a6937, component_name=cxs-calm-build-tkm-srv, organization_name=-, component_type=application, endpoint=PATCH: /odata/v2/CBLD_TASK_SRV/CBLD_C_TASK_TP(guid%27cba383f1-809f-40f1-8c76-cddd2b381c4c%27), space_name=CALMDev, component_instance=0, organization_id=-, correlation_id=825395c1-583d-4e5c-8ae4-6f9eaf19aae2, space_id=3bbd276d-7965-4625-aae9-da06139845ea, request_id=-, container_id=-] - ==> now execute query on Products
2019-08-20 14:05:12,322 WARN  [http-nio-8081-exec-2] - [com.netflix.config.sources.URLConfigurationSource] [tenant_id=05e65d26-f3e2-4937-9987-eb412f4cd732, component_id=7d44b23e-c4c1-42ba-9c60-0048c14a6937, component_name=cxs-calm-build-tkm-srv, organization_name=-, component_type=application, endpoint=PATCH: /odata/v2/CBLD_TASK_SRV/CBLD_C_TASK_TP(guid%27cba383f1-809f-40f1-8c76-cddd2b381c4c%27), space_name=CALMDev, component_instance=0, organization_id=-, correlation_id=825395c1-583d-4e5c-8ae4-6f9eaf19aae2, space_id=3bbd276d-7965-4625-aae9-da06139845ea, request_id=-, container_id=-] - No URLs will be polled as dynamic configuration sources.
2019-08-20 14:05:19,270 ERROR [http-nio-8081-exec-2] - [com.sap.cloud.sdk.odatav2.connectivity.ODataQuery] [tenant_id=05e65d26-f3e2-4937-9987-eb412f4cd732, component_id=7d44b23e-c4c1-42ba-9c60-0048c14a6937, component_name=cxs-calm-build-tkm-srv, organization_name=-, component_type=application, endpoint=PATCH: /odata/v2/CBLD_TASK_SRV/CBLD_C_TASK_TP(guid%27cba383f1-809f-40f1-8c76-cddd2b381c4c%27), space_name=CALMDev, component_instance=0, organization_id=-, correlation_id=825395c1-583d-4e5c-8ae4-6f9eaf19aae2, space_id=3bbd276d-7965-4625-aae9-da06139845ea, request_id=-, container_id=-] - Could not connect to destination service [No Access] :com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to get destinations: com.sap.cloud.sdk.cloudplatform.connectivity.DestinationServiceCommand#t=05e65d26-f3e2-4937-9987-eb412f4cd732#u= timed-out and fallback disabled. If your application is running on Cloud Foundry, make sure to have a binding to both the destination service and the authorization and trust management (xsuaa) service, AND that you either properly secured your application or have set the "ALLOW_MOCKED_AUTH_HEADER" environment variable to true. Please note that authentication types with user propagation, for example, principal propagation or the OAuth2 SAML Bearer flow, require that you secure your application and will not work when using the "ALLOW_MOCKED_AUTH_HEADER" environment variable. If your application is not running on Cloud Foundry, for example, when deploying to a local container, consider declaring the "destinations" environment variable to configure destinations.
2019-08-20 14:05:19,271 ERROR [http-nio-8081-exec-2] - [com.sap.cloud.sdk.odatav2.connectivity.ODataQuery] [tenant_id=05e65d26-f3e2-4937-9987-eb412f4cd732, component_id=7d44b23e-c4c1-42ba-9c60-0048c14a6937, component_name=cxs-calm-build-tkm-srv, organization_name=-, component_type=application, endpoint=PATCH: /odata/v2/CBLD_TASK_SRV/CBLD_C_TASK_TP(guid%27cba383f1-809f-40f1-8c76-cddd2b381c4c%27), space_name=CALMDev, component_instance=0, organization_id=-, correlation_id=825395c1-583d-4e5c-8ae4-6f9eaf19aae2, space_id=3bbd276d-7965-4625-aae9-da06139845ea, request_id=-, container_id=-] - Could not connect to destination service [No Access] : [Ljava.lang.StackTraceElement;@7beb95af
2019-08-20 14:05:19,280 ERROR [http-nio-8081-exec-2] - [com.sap.calm.imp.tkm.srv.odata.handler.TaskServiceHandler] [tenant_id=05e65d26-f3e2-4937-9987-eb412f4cd732, component_id=7d44b23e-c4c1-42ba-9c60-0048c14a6937, component_name=cxs-calm-build-tkm-srv, organization_name=-, component_type=application, endpoint=PATCH: /odata/v2/CBLD_TASK_SRV/CBLD_C_TASK_TP(guid%27cba383f1-809f-40f1-8c76-cddd2b381c4c%27), space_name=CALMDev, component_instance=0, organization_id=-, correlation_id=825395c1-583d-4e5c-8ae4-6f9eaf19aae2, space_id=3bbd276d-7965-4625-aae9-da06139845ea, request_id=-, container_id=-] - ==> Exception calling backend OData V2 service for Query of Products: Unable to execute the OData operation : Failed to execute OData request.

score 1 · Answer 1 · answered Aug 22 '19 at 12:44

It takes almost 9 seconds from your DEBUG message to the first WARN, complaining about the timeout. In fact the default timeout for SAP Cloud SDK internal DestinationServiceCommand is set to 6 seconds.

If querying all destinations really takes longer than 6 seconds in your case, then I would suggest a workaround until we find a proper solution:

1.) Create a new class CustomHystrixPropertiesStrategy

import com.netflix.hystrix.HystrixCommandKey;
import com.netflix.hystrix.HystrixCommandProperties;
import com.netflix.hystrix.strategy.HystrixPlugins;
import com.netflix.hystrix.strategy.properties.HystrixPropertiesCommandDefault;
import com.netflix.hystrix.strategy.properties.HystrixPropertiesStrategy;

import javax.annotation.Nonnull;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class CustomHystrixPropertiesStrategy extends HystrixPropertiesStrategy
{
    private final Map<HystrixCommandKey, HystrixCommandProperties> commandProperties = new ConcurrentHashMap<>();

    @Nonnull
    private HystrixPropertiesStrategy delegate;

    public static CustomHystrixPropertiesStrategy register()
    {
        final HystrixPropertiesStrategy delegate = HystrixPlugins.getInstance().getPropertiesStrategy();
        final CustomHystrixPropertiesStrategy strategy = new CustomHystrixPropertiesStrategy();

        HystrixPlugins.reset();
        HystrixPlugins.getInstance().registerPropertiesStrategy(strategy);
        strategy.delegate = delegate;

        return strategy;
    }

    public void addProperties(
        @Nonnull final HystrixCommandKey key,
        @Nonnull final HystrixCommandProperties.Setter customSetter )
    {
        final HystrixCommandProperties properties = new HystrixPropertiesCommandDefault(key, customSetter);
        commandProperties.put(key, properties);
    }

    @Override
    public
        HystrixCommandProperties
        getCommandProperties( final HystrixCommandKey key, final HystrixCommandProperties.Setter builder )
    {
        if( commandProperties.containsKey(key) ) {
            return commandProperties.get(key);
        }
        return delegate.getCommandProperties(key, builder);
    }
}

2.) Register the custom strategy instance

You need to run the following operation and keep the returned instance. This can happen at a central point of your application or statically in your service class, such that the instruction is only run once.

final CustomHystrixPropertiesStrategy strategy = CustomHystrixPropertiesStrategy.register();

Note: If you want to make sure this is run only once, you can put it into a static block:

static {
  final CustomHystrixPropertiesStrategy strategy = CustomHystrixPropertiesStrategy.register();
}

3.) Add the custom timeout property for a specific Hystrix command.

You can add the following code in front of the (currently) breaking call.

// reconstruct existing command key
final String tenantId = TenantAccessor.getCurrentTenantIfAvailable().map(Tenant::getTenantId).orElse(null);
final String commandKey = HystrixUtil.getCommandKey(DestinationServiceCommand.class, tenantId, null);
final HystrixCommandKey destinationServiceCommandKey = HystrixCommandKey.Factory.asKey(commandKey);

// construct custom command properties
final HystrixCommandProperties.Setter customSetter =
    HystrixCommandProperties
        .Setter()
        .withExecutionTimeoutInMilliseconds(60000) // 60 SECONDS INSTEAD OF 6 SECONDS
        .withCircuitBreakerEnabled(true)
        .withCircuitBreakerSleepWindowInMilliseconds(1000)
        .withFallbackEnabled(false);

// apply custom Command properties
strategy.addProperties(destinationServiceCommandKey, customSetter);

Make sure the reference to the custom strategy is in the scope.

CustomHystrixPropertiesStrategy strategy;

If it is not working as expected, please try to add a breakpoint or some log output to ensure the new code is actually reached at application runtime.

Failed to get destinations... timed-out and fallback disabled

1 Answers1

1.) Create a new class CustomHystrixPropertiesStrategy

2.) Register the custom strategy instance

3.) Add the custom timeout property for a specific Hystrix command.