Spring cloud config server - Vault(AWS_IAM authentication)

Question

I am setting up a spring boot cloud config server which is using VAULT to store secrets.

Vault Authentication mechanism is set as AWS_IAM. I also have aws-java-sdk in my class path. Aws cli is also configured.

I am able to get secrets from VAULT when i make request to config server except the very first request

First request to the config server fails as it tries to schedule task to renew vault token after expiration(from vault response)

The configuration in application.properties


       spring.profiles.active=vault
       spring.cloud.config.server.vault.kv-version=2
       spring.cloud.config.server.vault.authentication=aws_iam

Error logs are


    2020-02-07 18:47:17.089 DEBUG 19328 --- [nio-8088-exec-3] o.s.v.a.AwsIamAuthentication             : Login successful ************************************************************
    2020-02-07 18:47:17.091  INFO 19328 --- [nio-8088-exec-3] o.s.v.a.LifecycleAwareSessionManager     : Scheduling Token renewal
    2020-02-07 18:47:17.100 DEBUG 19328 --- [nio-8088-exec-3] o.s.web.servlet.DispatcherServlet        : Failed to complete request: java.lang.IllegalStateException: ThreadPoolTaskScheduler not initialized
    2020-02-07 18:47:17.107 ERROR 19328 --- [nio-8088-exec-3] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalStateException: ThreadPoolTaskScheduler not initialized] with root cause

    java.lang.IllegalStateException: ThreadPoolTaskScheduler not initialized
        at org.springframework.util.Assert.state(Assert.java:73) ~[spring-core-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.getScheduledExecutor(ThreadPoolTaskScheduler.java:155) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.schedule(ThreadPoolTaskScheduler.java:307) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.vault.authentication.LifecycleAwareSessionManager.lambda$scheduleRenewal$3(LifecycleAwareSessionManager.java:369) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at java.util.Optional.ifPresent(Optional.java:159) ~[na:1.8.0_201]
        at org.springframework.vault.authentication.LifecycleAwareSessionManager.scheduleRenewal(LifecycleAwareSessionManager.java:369) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.authentication.LifecycleAwareSessionManager.doGetSessionToken(LifecycleAwareSessionManager.java:318) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.authentication.LifecycleAwareSessionManager.getSessionToken(LifecycleAwareSessionManager.java:277) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultTemplate.lambda$getSessionInterceptor$1(VaultTemplate.java:276) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.vault.client.VaultClients.lambda$createRestTemplate$0(VaultClients.java:128) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:93) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:77) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:677) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:615) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.vault.core.VaultKeyValueAccessor.lambda$doRead$1(VaultKeyValueAccessor.java:133) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultKeyValueAccessor.lambda$doRead$2(VaultKeyValueAccessor.java:168) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:466) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:165) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:132) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:109) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultKeyValue2Template.get(VaultKeyValue2Template.java:53) ~[spring-vault-core-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cloud.config.server.environment.vault.SpringVaultEnvironmentRepository.read(SpringVaultEnvironmentRepository.java:51) ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cloud.config.server.environment.AbstractVaultEnvironmentRepository.findOne(AbstractVaultEnvironmentRepository.java:88) ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cloud.config.server.environment.EnvironmentRepository.findOne(EnvironmentRepository.java:31) ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cloud.config.server.environment.EnvironmentRepository$$FastClassBySpringCGLIB$$6ae93c63.invoke() ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:120) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.cloud.config.server.environment.vault.SpringVaultEnvironmentRepository$$EnhancerBySpringCGLIB$$4d24df30.findOne() ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:58) ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cloud.config.server.environment.EnvironmentEncryptorEnvironmentRepository.findOne(EnvironmentEncryptorEnvironmentRepository.java:61) ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cloud.config.server.environment.EnvironmentController.getEnvironment(EnvironmentController.java:144) ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.cloud.config.server.environment.EnvironmentController.defaultLabel(EnvironmentController.java:108) ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_201]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_201]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_201]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_201]
        at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282) ~[spring-core-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.cloud.context.scope.GenericScope$LockedScopedProxyFactoryBean.invoke(GenericScope.java:499) ~[spring-cloud-context-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.cloud.config.server.environment.EnvironmentController$$EnhancerBySpringCGLIB$$fb6d1754.defaultLabel() ~[spring-cloud-config-server-2.2.1.RELEASE.jar:2.2.1.RELEASE]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_201]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_201]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_201]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_201]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108) ~[spring-boot-actuator-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1598) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_201]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_201]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.30.jar:9.0.30]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

This is a bug, see https://github.com/spring-cloud/spring-cloud-config/issues/1556. — mp911de, Mar 18 '20 at 10:14

snowe · Answer 1 · 2020-03-19T15:56:19.687

This is a bug with spring-cloud-config. See this issue. It should be fixed in Ilford.M1. There is no fix besides updating versions. Per this comment:

That’s a bug: SpringVaultClientConfiguration is not a @Configuration class. Spring Vault still assumes CGlib proxies while the Spring Cloud Config Vault support isn’t annotated with @Configuration

my old answer for reference. It was wrong

We were encountering the same issue until I fixed it today. It appears that even if you are able to hit vault, spring defaults to a NativeEnvironmentRepository. Simply providing your own bean of type VaultEnvironmentRepository fixes the issue.

@Bean
public VaultEnvironmentRepository vaultEnvironmentRepository(
        ObjectProvider<HttpServletRequest> objectProvider, 
        EnvironmentWatch environmentWatch,
        RestTemplate restTemplate,
        VaultEnvironmentProperties vaultEnvironmentProperties
) {
    return new VaultEnvironmentRepository(objectProvider, environmentWatch, restTemplate, vaultEnvironmentProperties);
}

@Bean
public RestTemplate restTemplate() {
    return new RestTemplate();
}

Hmm. this forces the client to provide their own client token, which, if you don't want that functionality, is less than desirable. — snowe, Mar 16 '20 at 18:23

Spring cloud config server - Vault(AWS_IAM authentication)

1 Answers1