databricks-cli: JSONDecodeError when running job in bash script

Question

I am trying to run a Databricks job with notebook parameters within a bash script on a Linux server. I am following instructions from the docs and I have verified that the commands work in terminal.

Here is my script:

#!/bin/bash

### this commands works in terminal but not when running script
databricks jobs run-now --job-id 1 --notebook-params '{"param1":"value1"}'

### this command works fine in terminal and when running script
databricks runs list

The error I get is:

Error: JSONDecodeError: Expecting value: line 1 column 59 (char 58)

What is also interesting is that other databricks-cli commands work in that do not require a JSON string argument.

Try to run the command explicitly with an utf-8 locale: `LANG=en_US.UTF8 databricks jobs run-now --job-id 1--notebook-params '{"param1":"value1"}'`. Does that work? — hek2mgl, May 08 '19 at 14:04
I tried adding `export LANG=en_US.UTF8` to script and also tried just adding `LANG=en_US.UTF8` in front of the databricks command and still getting same error — Korean_Of_the_Mountain, May 08 '19 at 15:33
the script runs on the same host where you run it in terminal, right? (and not in a container or vm, right?) — hek2mgl, May 08 '19 at 15:35
Correct I'm running script in the terminal (`./run_script.sh`) and it's the same terminal where I was able to run the commands successfully. They just don't work when I try to execute the commands inside the script. — Korean_Of_the_Mountain, May 08 '19 at 16:42
do you mean another Databricks user? I used my Databricks user token and I am the one who created the Databricks job and notebook — Korean_Of_the_Mountain, May 08 '19 at 19:29
No, I meant system user. Could be that you run the script with sudo. Otherwise I have not idea, must be a databricks specific problem then. (I thought it could be system related) — hek2mgl, May 08 '19 at 19:39
@hek2mgl Sorry but I stupidly didn't include the fact that I was passing a variable to the JSON string and that was the issue all along. — Korean_Of_the_Mountain, May 09 '19 at 14:52

score 1 · Answer 1 · answered May 09 '19 at 14:51

The script I provided in the question was not actually representative of the script I was running.

The script I'm actually running is something like this:

#!/bin/bash

currentdate=`date +\%Y\%m\%d`
RUNDATE=$(date "--date=${currentdate} - ${stepsize} day" +%Y%m%d)

### this commands works in terminal but not when running script
databricks jobs run-now --job-id 1 --notebook-params '{"param1":"value1", "rundate":"$RUNDATE"}'

### this command works fine in terminal and when running script
databricks runs list

and the issue was passing a variable to the JSON.

The working version is this:

#!/bin/bash

stepsize=1
currentdate=`date +\%Y\%m\%d`
RUNDATE=$(date "--date=${currentdate} - ${stepsize} day" +%Y%m%d)

echo $rundate 

databricks jobs run-now --job-id 263 --notebook-params '{"param1":"value1", "rundate":"'"$RUNDATE"'"}'

databricks runs list

databricks-cli: JSONDecodeError when running job in bash script

1 Answers1