0

I'm using GCP+R2DBC in a Spring Boot application. This is a simplified version of my code:

public class WidgetDetailsCustomRepositoryImpl implements WidgetDetailsCustomRepository {

    private final WidgetDetailsRepository widgetDetailsRepository;

    private final R2dbcEntityTemplate r2dbcEntityTemplate;

    private Mono<WidgetDetailsModel> saveWidgetDetails(WidgetDetailsModel widgetDetailsModel) {
        Mono<WidgetDetailsModel> result = widgetDetailsRepository.existsById(widgetDetailsModel.getWidgetTrackingNumber())
                .flatMap(widgetExists -> {
                    if (widgetExists) {
                        log.info("Updating WidgetDetails: {}", widgetDetailsModel.getWidgetTrackingNumber());
                        return r2dbcEntityTemplate.update(widgetDetailsModel);
                    } else {
                        log.info("Inserting WidgetDetails: {}", widgetDetailsModel.getWidgetTrackingNumber());
                        return r2dbcEntityTemplate.insert(widgetDetailsModel);
                    }
                });

    }
}

I get a DEADLINE_EXCEEDED error after about 19s on the existsById(). I get something similar for a findById(). To be clear, I'm not doing a complex query, rather fetch, save or update exactly 1 record in a spanner table. My CPU usage is about 1.5% and this is a dev based project. Also, a single record has about 10 columns and is less than about 255 characters.

In the same project, I can successfully fetch a message from a PubSub subscription. I mention this to emphasize that I'm not having any network issues as mentioned in the troubleshooting guide from GCP.

There was talk about changing the RPC retry values, but they fail to mention how to change these things for R2DBC. I realize there's some JSON files, but they don't say how to, for eg. specify your own values.

Also, there is a "Spanner health check failed" message. This is coming from somewhere in a GCP library, suggesting that I did something wrong.

GCP does say to specify retry off some blockingStub object. I'm not doing any blocking calls. The simplest of calls fails with a DEADLINE_EXCEEDED error.

In short, I verified network connectivity. I verified CPU usage. Is there some magic R2DBC+Spanner setting I need to do?

Woodsman
  • 901
  • 21
  • 61

1 Answers1

1

This is most likely caused by a timeout error while trying to authenticate or refresh the OAuth token. The 19 seconds (or close to 20 seconds) timeout corresponds with the default timeout for that.

There is no default timeout in the standard Cloud Spanner client that corresponds with approx 19 seconds.

What kind of authentication is your application using? Are you sure the credentials that you are using have access to the Cloud Spanner database?

Knut Olav Løite
  • 2,964
  • 7
  • 19
  • I have the GOOGLE_CLOUD_APPLICATION set and have credentials set for both pubsub and spanner in my application.yml file. Pubsub works spanner does not. – Woodsman Jun 16 '23 at 12:23
  • What do you mean with '... have credentials for both pubsub and spanner in my application.yml file'? If you have GOOGLE_APPLICATION_CREDENTIALS set, I would expect that to be the default credentials that are used for all Google Cloud services. If you have specific credentials set in your application.yml file for Spanner and Pubsub, that sounds like you are overriding the default set in GOOGLE_APPLICATION_CREDENTIALS. Are you using different credentials for Spanner and Pubsub? – Knut Olav Løite Jun 17 '23 at 04:27
  • If I were to guess anything based on the information here at the moment, I would say that this is where the DEADLINE_EXCEEDED error is coming from: https://github.com/googleapis/google-auth-library-java/blob/27db5f2979ff868e78ce7d6cb0dbefd817684723/oauth2_http/java/com/google/auth/oauth2/ServiceAccountCredentials.java#L508 That request has a default timeout of 20 seconds, which fits your error description perfectly: 1. It happens for the first (simple) query (i.e. it does a token refresh first) 2. The timeout value is 20 seconds. – Knut Olav Løite Jun 17 '23 at 04:38