Databricks Jobs REST API call does not work with Powershell

Question

So, here is the link to Jobs API call in Databricks here

Everything works in Python using requests. E.g both the job creation and job listing works

import requests
proxy= "http://127.0.0.1:8888"
access_token="tokenabc"

proxies = {
"https": proxy,
}


header_read = {
'Authorization': "Bearer " + access_token,
'Accept': "application/scim+json"
}

#list jobs
init_get=requests.get('https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/list', headers=header_read, proxies=proxies)

#create job
init_post=requests.post('https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/create', headers=header_read, proxies=proxies,verify=False,json=job_payload)

However, strangely in Powershell, only the job creation works and Job listing fails.

#this works
Invoke-RestMethod -Uri 'https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/create' -Headers @{ 'Authorization' = "Bearer $bearertoken" } -Method Post -Body $content -ContentType  'application/json'

#this does not work
Invoke-WebRequest -Uri 'https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/list' -Headers @{ 'Authorization' = "Bearer $bearertoken" } -Method Get

#RestMethod also does not work
Invoke-RestMethod -Uri 'https://databricksworkspaceid.azuredatabricks.net/api/2.0/jobs/list' -Headers @{ 'Authorization' = "Bearer $bearertoken" } -Method Get

I have also tried setting the Content type on these but nothing helps. Also, explicitly setting the proxy (fiddler) does not help.

-proxy "http://127.0.0.1:8888"

But should not be the proxy also , as the post method works.

I just keep getting an error like

Invoke-WebRequest : The underlying connection was closed: An unexpected error occurred on a send.
At line:22 char:5
+     Invoke-WebRequest -Uri 'https://databricksworkspaceid.azuredatabricks.net/ap ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

or in case of RestMethod

Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send.
At line:22 char:5
+     Invoke-RestMethod -Uri 'https://databricksworkspaceid.azuredatabricks.net/ap ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

I could have understood if everything failed in Powershell, but the post method (job creation) works, so not sure, why the connection would be terminated in case of a Get request but not a post.

Going by some forum posts, I have also tried the following but to no avail-

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Does anyone know what am I doing wrong/missing? Baffling to see working in python but only a part in Powershell.

score 2 · Accepted Answer · answered Sep 20 '20 at 18:17

So, I finally found the answer. Basically 2 things:

Had to do this before hitting the list endpoint (strangely as I said- the create endpoint) worked

Basically allowing TLS, TLS1.1 and TLS1.2

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12

Doing only point 1 didn't work. I HAD to also use an "elevated" powershell session. Basically running the script with step 1 included, using a "Run as Administrator".

Databricks Jobs REST API call does not work with Powershell

1 Answers1