1

I know there are a lot of questions here about how to handle quotes in environment variables. This question has a different focus so please read on:

Before last week we had set our environment variables on our databricks cluster (7.3 LTS, includes Apache Spark 3.0.1, Scala 2.12) like this:

EXAMPLE_FOO="gaga"

For whatever reason (don't remember) we needed the quotes to get this result in python:

print(os.environ["EXAMPLE_FOO"])  => gaga

Since last week the behavior changed, now we get:

print(os.environ["EXAMPLE_FOO"])  => "gaga"

with the quotes. We have now clue why this suddenly changed. There was no software update or alike from our side on this production system. We would like to understand the root cause. Has some library on databricks changed or is there a setup flag in the databricks configuration where you can toggle this behavior?

Note: We know how to handle both cases in python so ne need to tell me how to handle the variables. We need to know what suddenly may have caused the issue.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
HeyMan
  • 1,529
  • 18
  • 32

1 Answers1

1

It looks like your workspace was already upgraded to incorporate this breaking change that was highlighted in the release notes. You also should have communication from the Databricks support about this change. Basically, you don't need to use escaping anymore, so you can remove the quotes.

But it's really better to raise a support ticket with Microsoft to understand impact of this issue, and define the next steps.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132