0

I have developed a web application where user can choose machine learning framework/ number of iterations/ some other tuning parameter. How can I invoke Spark job from user interface by passing all the inputs and display response to user. Depending on the framework (dl4j/ spark mllib/ H2o) user can either upload input csv or the data can be read from Cassandra.

  1. How can I call SPARK job from user interface?
  2. How can I display the result back to user?

Please help.

lalithark
  • 107
  • 9

1 Answers1

1

You can take a look at this github repository.

In this what is being done is as soon as a GET request is arrived, it takes out the data from the Cassandra and then Collect the data and throws it back as the response.

So in your case :

What you can do is , as soon as you recieve a POST request , you can get the parameters from the request and perform the operations accordingly using these parameters and the collect the Result on the master and then throw it back to the user as the Response.

P.S: Collecting on Master is a bit tricky and lot of data can cause OOM. What you can do is save the results on hadoop and send back the URL to the Results or something like that.

For more info look into this blog related to this github: https://blog.knoldus.com/2016/10/12/cassandra-with-spark/

Community
  • 1
  • 1
Shivansh
  • 3,454
  • 23
  • 46
  • I am new to SPRAK and AKKA. I have downloaded the project and successfully executed it in my local machine. Can I run the same application in Yarn? Can I use it in production? – lalithark Nov 24 '16 at 05:43
  • It is not a production level code ! You can change it according to your use case ! – Shivansh Nov 24 '16 at 06:02
  • If I want run the code in yarn mode, What are all the changes I should do? Can I not use AKKA rest in yarn mode? – lalithark Nov 24 '16 at 06:11
  • That I have to check ! Please go ahead and see for yourself and let me know here too ! If you can document it how to run it in cluster mode successfully and make a PR on this Repository . I would be glad to accept it ! – Shivansh Nov 24 '16 at 06:23