I understand livy session statement intakes code statements like the below example.
data = {
'code': textwrap.dedent("""
import random
NUM_SAMPLES = 100000
def sample(p):
x, y = random.random(), random.random()
return 1 if x*x + y*y < 1 else 0
count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample).reduce(lambda a, b: a + b)
print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
""")
}
r = requests.post(statements_url, data=json.dumps(data), headers=headers)
but is there a way in which I can provide pyspark files, maybe something like this:
data = {
'pySparkFile': file_name.py
}
I understand livy batch provides this functionality but I want an interactive session where users can pass multiple scripts one after another and we can also call variables of other scripts, just like in an interactive pySpark session.
