This error usually occurs if there is a mismatch between TLS protocol versions or ciphers used between the Azure DevOps server and Azure Key Vault.
Currently, more and more Microsoft products/services are deprecating TLS 1.0 and 1.1, you need to ensure your applications to be working well using TLS 1.2.
Try forcing TLS version 1.2 from your Azure DevOps Server and then try it again.
Following Azure DevOps TLS 1.2 transition readiness checker instructions to identify the issue and enable TLS1.2 on your target machines by referencing the outputted mitigation advice after running the “AzureDevOpsTls12Analysis.ps1
” script on your Azure DevOps Server.
Besides, please also make sure the affected service principal used in the service connection has been assigned the correct role to access your Key Vault.
If all above are working well, then you can try to run Azure CLI on your Azure DevOps Server machine to Sign in with the affected service principal and Retrieve a secret from Key Vault to see if that works. If it works, then it should be something wrong with your Azure DevOps Server settings. Please check if you are using proxy, ensure it's not blocked by the firewall or policy settings.