I am writing a ASP.Net Web API that at some point will communicate with an Apache Spark cluster.
The communication is established using Livy server on the spark cluster that exposes a REST API interface and an HTTP client i wrote.
In my business logic i need to define operations that will be submitted to spark as jobs. For now i create a Livy PySpark Session and submit PySpark code.
The PySpark Code is generated by the logic of my Web API. As an example of code generation, if i want to import a python module i do this :
string pythonImportTemplate = "from {0} import {1}";
statements.Add(string.format(pythonImportTemplate, param1, param2));
/*** Post Statements To Spark ***/
This seems very wrong and repetitive to me.
My question : is there any robust way (or design pattern) for generating such code strings minimizing the probability of interpreter (Livy) syntax errors ?
