2

I am running spark code using Nifi ExecuteSparkInteractive processor and I see it's outcome either success, failed and wait. I am able to manage and route the result perfectly fine hen it come to success and failed state but sometimes I see file is getting lost and I think it's going to wait state and no routing defined in my Nifi flow for "wait" state. Can anyone help me understand what exactly is the "wait" state and how can I route the control in this state without impacting end result.

Thank you in advance!

I have tried to route the nifi control to UpdateAttribute processor but I see no result from spark code. Looks like code is not gettting executed during "wait"state from ExecuteSparkINteractive.

There is no code as such for this problem as Nifi provide check boxes and connection controller between processors.

I have passed control from ExecuteSparkInteractive processor to RouteOnAttribute process for success and failure conditions and also for wait state, but spark code is not getting executed during wait state.

Spark code should execute during wait state and then it should route to success and failure.

Ajay Ahuja
  • 1,196
  • 11
  • 26

1 Answers1

0

I figured the purpose of “wait” state in ExecuteSparkInteractive processor.

Whenever control/file flows through Execute spark interactive processor it executed the spark code from this processor but it looks for available Livy Spark session running on yarn cluster first. If non of the Livy Spark session’s available (May be other processes are running on those sessions) then control goes to wait state.

Ideally, ExecuteSparkInteractive processors is waiting for Livy spark session to be available to execute the code. That’s the reason code doesn’t execute if “wait” state is not handled.

To solve this problem, I called ExecuteSparkInteractive to itself in “wait” state, basically loop the control back to same processor to run the spark code whenever Livy session is available for the processor. This approached solved my problem.

I attached screenshot of my template for reference. enter image description here

Ajay Ahuja
  • 1,196
  • 11
  • 26