Is there any other port needed to use LIVY with Sparklyr except the LIVY port? (with default 8998).
I have two machines from which I try to use Spakrlyr with LIVY:
- my local Windows station in network zone A
- a remote linux server with R in network zone B
The cluster (Cloudera) is in zone B. When I run the following code on the linux server in the same zone as cluster all is fine:
library(sparklyr)
library(dplyr)
library(DBI)
sc <- spark_connect(master = "http://<my_cluster_and_domain>:8998", method = "livy")
I have asked to open in the Firewall between zones A and B the following ports: 8021 - 8090 and 8998, but I am not able to get a Spark Context on my Windows machine. The defulat LIVY port 8998 is set on the cluster. The code above basically hangs without any answer forever.
Before enabling LIVY I tried a "normal" connection with all possible variants/parameters I could imagine and the result is always the same - a hanging spark_connection call without any answer.
I am also able to open from that Windows machine the LIVY console ("http://my_cluster_and_domain:8998/ui") and send to LIVY a Spark job via requests in Python, so the connection on the default port 8998 is allowed to go thru FW.
On my Windows machine I have R 3.5.3 (64-bit) and Sparklyr 1.0.0.
Thank you for any hints.
