1

We recently shifted our apis to make use of the openapi web contract with the combination of eventbus to send and receive api responses for our clients.

It was working fine until today where we noticed that most of our apis were timing out, the following errors were thrown.

API FAILURE: (TIMEOUT,-1) Timed out after waiting 30000(ms) for a reply. address: __vertx.reply.448, repliedAddress: /v1/analytics/post
at io.vertx.core.eventbus.impl.HandlerRegistration.sendAsyncResultFailure(HandlerRegistration.java:146)
at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$new$0(HandlerRegistration.java:78)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:907)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:866)
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:320)
at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

We were unable to debug as our consumers where not even receiving the messages to reply back. Our logs just had these error messages.

e.g openapi web contract and eventbus setup, all our apis are configured this way

routerFactory.addHandlerByOperationId("addViews", routingContext -> {
           RequestParameters requestParameters = routingContext.get("parsedParameters");

                vertx.eventBus()
                        .send(Endpoints.API_ANALYTICS_POST,
                                Json.encodeToBuffer(requestParameters.body().getJsonObject()),
                                result -> {
                                    if (result.succeeded()) {
                                        successResponse(routingContext, result.result());
                                    } else {
                                        errorResponse(routingContext, result.cause());
                                    }
                                });

        });
        registerFailureHandler(routerFactory,"addViews");

We use the shared event bus as of now not the cluster event bus. Are there any limits to how many messages the eventbus can handle?

MessageConsumer<Buffer> views = eventBus.consumer(Endpoints.API_ANALYTICS_POST);


views.handler(message->{
        addPostViews(message, message.body().toJsonObject());
    });



private void addPostViews(Message message, JsonObject request) {
    LOGGER.info("PostViewsEvent: {0}",  request.toString());
    postMetricsService.addPostViews(request).setHandler(res -> {
        JsonObject response = new JsonObject();
        response.put("captured", true);
        if (res.succeeded()) {
            message.reply(ApiResponse.getCustomSuccessResponse(response));
        } else {
            errorResponse(message,res.cause());
        }
    });
}

public Future<Boolean> addPostViews(JsonObject views) {
    Future<Boolean> future = Future.future();
    future.complete(true);
    // mongo db query to store views
    // redis query to store views
}
user160108
  • 930
  • 3
  • 8
  • 38
  • There are no limits. Can you show the code on the other side of `API_ANALYTICS_POST`? Are you sure there are no blocking calls on the message consumer of `API_ANALYTICS_POST`? And mostly important, are you sure the message consumer always replies? – Francesco Guardiani Apr 24 '19 at 06:12
  • @FrancescoGuardiani included the code for the other side of the api. So if the request was process and reached the other side the Log value for PostViewsEvent should have been visible in my logs but where not visible. Not sure if i am doing something wrong here. After restarting servers the api was working again. Also the app runs inside a docker container. – user160108 Apr 24 '19 at 07:37
  • @FrancescoGuardianiso my verticles are mostly configuring consumers to listen to events on the event bus for our apis. Do these verticles need to be workers? as they do not start any http servers. At the moment they are non worker verticles. – user160108 Apr 24 '19 at 07:43
  • The message consumer verticles should not be worker verticles if they use all non blocking methods https://vertx.io/docs/vertx-core/java/#worker_verticles. Verticles that exposes HttpServer and produce messages are non blocking by nature, so they should not be worker. What I understood (but maybe I'm wrong) is that you are blocking the thread somewhere. Are you sure `addPostViews` doesn't contain any blocking code? Are you sure in this verticle there is no other blocking code? If you block the thread, It should produce a warning in your log saying "thread blocked" – Francesco Guardiani Apr 24 '19 at 08:13
  • If you are running verticles in different jvm instances/containers, are you sure the docker configuration is correct? – Francesco Guardiani Apr 24 '19 at 08:15
  • @FrancescoGuardiani we had -Dvertx.threadChecks=false option set, have removed this to debug further. regarding the addPostViews it returns a Future immediately so as to not block the code. – user160108 Apr 24 '19 at 08:25

0 Answers0