9

I am trying to accomplish the following use case on Apache Zeppelin: When I write an sql query, for example 

%sql SELECT * FROM table1 WHERE column1 = ${column1=1,1|2|3|4}

I get a combo box displayed with these values (1,2,3,4) as options. What I want to do is populate this list with all distinct values available for this column (or as a matter of fact any other set of values which I might want to take from another paragraph in form of a variable). So currently I am stuck at how to use some variables defined in one paragraph inside an sql statement in another paragraph ?

Diving into the code, I saw that inside the Zeppelin-interpreter, a file called Input.java checks for a pattern ${someColumn=someValues}, fills up the combo-box options and then creates a simple query, and hence I have dropped the idea of populating it by running a query in the same paragraph.

Venkatachalam
  • 16,288
  • 9
  • 49
  • 77
kunalc92
  • 91
  • 1
  • 5

2 Answers2

4

I'm using a Scala variable from one paragraph to Shell Script in another paragraph. Here's the answer.

In Scala Cell

%spark2
val myVal = "test-value-across-paragraphs"
z.put("objName", myVal)

In Shell Cell

%sh
echo {objName}

This requires object interpolation to be enabled which can be done by setting the value of the property zeppelin.shell.interpolation to true. Check Apache Zeppelin for further help.

Update May 19, 2019

The above procedure may not work in Zeppelin 2.2 but obviously works in Zeppelin 2.3. Also in 2.3, the value of interpolation can be changed from the sh.config cell. 

%sh.conf
zeppelin.shell.interpolation true
Abu Shoeb
  • 4,747
  • 2
  • 40
  • 45
  • 1
    How to set object in shell and use in spark? Consider your example in reverse? – Akshay Trivedi Jul 24 '20 at 06:13
  • Not aware of the exact method but you can try setting up an `ENVIRONMENT VARIABLE` in shell and then access it in spark. For example, `%sh export MYVAR=test` and then in spark `%spark System.getenv("MYVAR")` or `%spark sys.env("MYVAR")`. – Abu Shoeb Jul 24 '20 at 16:59
3

You can use ZeppelinContext to accomplish this, as it enables you to use put() and get() to set and retrieve objects between paragraphs.

Quoting the example from the linked page, note that the z object is the default instance of ZeppelinContext:

// Put object from scala
%spark
val myObject = ...
z.put("objName", myObject)

# Get object from python
%spark.pyspark
myObject = z.get("objName")
jmng
  • 2,479
  • 1
  • 25
  • 38