Spring boot - Pubsub subscriber load balancing

Question

My Spring boot app subscribes to pub-sub around 20K messages per hour. the avg process time for each message is 2 minutes so Im handles each message with a async method (messageConsumerService.handleMessage(message);) and ack/nack at the end of the message process.

It look like each pod tries to subscribe many messages as its can and after a few minutes/hours all of my pods stop to work and I can see this WARN:

com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 59.999988070s. [closed=[], open=[[buffered_nanos=60047213107, waiting_for_connection]]]
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:94)
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)

Pubsub config code:

@Configuration
@EnableIntegration
@RequiredArgsConstructor
@Slf4j
public class SubscriberConfiguration {

    private final MessageConsumerService messageConsumerService;

    @Value("${pubsub.subscription.subscriptionId}")
    private String subscriptionId;

    @Bean
    public MessageChannel pubSubInputChannel() {
        return new PublishSubscribeChannel();
    }

    @Bean
    public MessageChannel errorChannel() {
        return new PublishSubscribeChannel();
    }

    @Bean
    public PubSubInboundChannelAdapter messageChannelAdapter(
            @Qualifier("pubSubInputChannel") MessageChannel inputChannel,
            @Qualifier("errorChannel") MessageChannel errorChannel,
            PubSubTemplate pubSubTemplate) {
        PubSubInboundChannelAdapter adapter =
                new PubSubInboundChannelAdapter(pubSubTemplate, subscriptionId);
        adapter.setOutputChannel(inputChannel);
        adapter.setErrorChannel(errorChannel);
        adapter.setAckMode(AckMode.MANUAL);

        return adapter;
    }

    @ServiceActivator(inputChannel = "pubSubInputChannel")
    public void messageReceiver(
            @Header(GcpPubSubHeaders.ORIGINAL_MESSAGE) BasicAcknowledgeablePubsubMessage message) {
        messageConsumerService.handleMessage(message); //async method
    }

    @ServiceActivator(inputChannel = "errorChannel")
    public void errorReceiver(Throwable error) {
        log.error("Error occurred while processing message: " + error.getMessage());
    }
}

how can i achieve load balancing between all of my pods and also be able to handle all 20K message before the next hour?

try to use a single thread and executor channel but it didn't handle the load or crash with the DeadlineExceededException.