I suspect that your program is exiting because you never use "End" to end your if statement. Unless the var D is less than zero, the program WILL EXIT on that If statement, because every line after that is considered part of the If statement and is therefore ignored. Use the "End" token to specify what lines are part of the If statement and what lines are not. I suspect you want:
:If D<0
:Then
:ClrHome
:Disp "Nonreal answer"
:"..."
:Pause
:ClrHome
:Menu("Menu", "Restart", D"
:End
:...
Or something similar to that. I am 99% certain that this is your problem.
The other things to note is that you declare Lbl A twice. In TI-BASIC, when you use Goto A, the calculator starts at the beginning of the program and looks for the first "Lbl A", which is in this case, the first line. Secondly, in your menu, by putting that argument "D" after your one and only option, once the user clicks on that, you're going to label D, which does not exist, therefore, the program will exit with the error Lbl.
This line:
Lbl prgmQUADSTE3
Is not a valid line. I'm assuming you want the program to either call itself or call a seperate program. If the program is calling itself, I suggest you instead use a Goto statement, as your method will cause something similar to a stack overflow/memory leak.