Persistent Failure of Token Renewal and Configuration Sync in Spring Cloud Vault After Network Disruption

[yidongnan]

Describe the bug
I have encountered an issue while using Spring Cloud Vault. When my application starts successfully and the Vault service is running, the client correctly renews the token and synchronizes the configuration every 10 seconds by default.

However, I noticed a problem:
if there is a temporary network disruption between the client and Vault, the KeyValueDelegate.getSecret method throws an exception. After this exception occurs, the periodic renewal and configuration sync processes stop entirely, even after the network connection is restored.

This behavior leads to a situation where the Vault client does not resume its renewal and synchronization tasks automatically after a brief network issue, which can cause significant problems in production environments.

Is there any way to ensure that the periodic renewal and synchronization tasks can be automatically resumed after a network disruption?

version
spring-cloud-starter-vault-config 4.1.3

exception stacks

2024-08-26T15:23:49.566+08:00  WARN 34535 --- [hello] [g-Cloud-Vault-2] LeaseEventPublisher$LoggingErrorListener : [RequestedSecret [path='mid/dev/hello', mode=ROTATE]] Lease [leaseId='null', leaseDuration=PT0S, renewable=false] I/O error on GET request for "http://localhost:8200/v1/mid/data/dev/hello": Connect to http://localhost:8200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused

org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8200/v1/mid/data/dev/hello": Connect to http://localhost:8200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
	at org.springframework.web.client.RestTemplate.createResourceAccessException(RestTemplate.java:915) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:895) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:790) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:404) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.vault.core.VaultTemplate.lambda$doRead$5(VaultTemplate.java:464) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:451) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.VaultTemplate.doRead(VaultTemplate.java:461) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.VaultTemplate.read(VaultTemplate.java:356) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.util.KeyValueDelegate.getSecret(KeyValueDelegate.java:80) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.lease.SecretLeaseContainer.doGetSecrets(SecretLeaseContainer.java:709) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.lease.SecretLeaseContainer.doStart(SecretLeaseContainer.java:391) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.lease.SecretLeaseContainer.onLeaseExpired(SecretLeaseContainer.java:810) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.lease.SecretLeaseContainer.doRenewLease(SecretLeaseContainer.java:742) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.lease.SecretLeaseContainer.renewAndSchedule(SecretLeaseContainer.java:644) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.lease.SecretLeaseContainer.lambda$scheduleLeaseRenewal$4(SecretLeaseContainer.java:636) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.vault.core.lease.SecretLeaseContainer$LeaseRenewalScheduler$1.run(SecretLeaseContainer.java:937) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.1.12.jar:6.1.12]
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96) ~[spring-context-6.1.12.jar:6.1.12]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Caused by: org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[na:na]
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602) ~[na:na]
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[na:na]
	at java.base/java.net.Socket.connect(Socket.java:633) ~[na:na]
	at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.lambda$connectSocket$0(PlainConnectionSocketFactory.java:91) ~[httpclient5-5.3.1.jar:5.3.1]
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) ~[na:na]
	at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:90) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.socket.ConnectionSocketFactory.connectSocket(ConnectionSocketFactory.java:123) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:189) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:450) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183) ~[httpclient5-5.3.1.jar:5.3.1]
	at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:99) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:112) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.vault.core.VaultTemplate.lambda$getSessionInterceptor$1(VaultTemplate.java:258) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:88) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.vault.client.RestTemplateBuilder.lambda$createTemplate$4(RestTemplateBuilder.java:239) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:88) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.vault.client.VaultClients.lambda$createRestTemplate$0(VaultClients.java:117) ~[spring-vault-core-3.1.1.jar:3.1.1]
	at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:88) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:72) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) ~[spring-web-6.1.12.jar:6.1.12]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:889) ~[spring-web-6.1.12.jar:6.1.12]
	... 23 common frames omitted