Run Python script line-by-line in shell vs atomically

Question

I'm integrating MicroPython into a microcontroller and I want to add a debug step-by-step execution mode to my product (via a connection to a PC).

Thankfully, MicroPython includes a REPL aka Python shell functionality: I can feed it one line at a time and execute.

I want to use this feature to single-step on the PC-side and send in the lines in the Python script one-by-one.

Is there ANY difference, besides possibly timing, between running a Python script one line at a time vs python my_script.py?

Let's say you've got a loop. Are you going to have your own interpreter manage flow control? Because otherwise you can't step one line at a time -- either the whole loop runs, or none of it does. — Charles Duffy, Sep 29 '17 at 19:42
What that means, in effect, is that you need something much more involved to be able to get the effect of single-stepping: Effectively, you need to inject a new function call at any point where a breakpoint may be set. It's *conceivable* to do that by instrumenting code (ideally, parsing and programmatically modifying the AST), but that's going to be an awful mess, and approaching the amount of effort you'd need to actually do things *right* at the VM layer. — Charles Duffy, Sep 29 '17 at 19:45

score 2 · Accepted Answer · answered Sep 29 '17 at 19:53

2

Passing one line of code at a time on stdin is a completely unacceptable alternative to a proper debugger.

Let's say that you want to debug the following:

def foo():                            # 1
    for i in range(10):               # 2
        if i == 5:                    # 3
            raise Exception("Argh!")  # 4
                                      # 5
foo()                                 # 6

...in a proper step-by-step debugger, the user could use it like so:

break 4
run

Now, how are you going to do that? If you enter the function in a REPL, the function is defined as one operation, and it runs as one operation. It doesn't stop at line 6. It doesn't let you proceed line-by-line. The same is true of the for loop: Entering the text of the for loop one line at a time doesn't let you step it before the exception is thrown.

If you eliminate the function, and eliminate the loop (generating the code _something = iter(range(10)); i=_something.next(), maybe?), then you need to emulate the effects of scoping. It means you have a hugely different language than the one you're purportedly "debugging".

answered Sep 29 '17 at 19:53

Charles Duffy

280,126
43
390
441

I don't understand this statement: `Entering the text of the for loop one line at a time doesn't let you step it before the exception is thrown.` – Bob Sep 29 '17 at 20:14
Nvm I get it: it's the whole flow-control issue. Right? – Bob Sep 29 '17 at 20:16
Right. Once you call `foo()`, *all* of `foo` runs, it doesn't run one line at a time. – Charles Duffy Sep 29 '17 at 20:27
What if I sent it blocks of code? i.e. if-blocks, for-blocks, etc – Bob Sep 29 '17 at 20:28
Then (1) that means you need to simulate all the effects of entering and leaving function scopes on code (and those aren't always obvious), and (2) the blocks still run all at once, not a line at a time. – Charles Duffy Sep 29 '17 at 20:29
So `for i in range(10):` and everything under it still runs all at once. – Charles Duffy Sep 29 '17 at 20:29
I understand (2) but not your (1) – Bob Sep 29 '17 at 20:30
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/155632/discussion-between-charles-duffy-and-adrian). – Charles Duffy Sep 29 '17 at 20:31

score 1 · Answer 2 · answered Sep 29 '17 at 19:52

I don't know whether MicroPython has compile() and exec() built-in. But when embeded Python has them and when MCU has enough RAM, then I do the following:

Send a line to embeded shell to start a creation of variable with multiline string.

'_code = """\'
Send the code I wish executed (line by line or however)
Close the multiline string with """
Send exec command to run the transfered code stored in the variable on MCU and pick up the output.

If your RAM is small and you cannot transfer whole code at once, you should transfer it in blocks that would be executed. Like functions, loops, etc.

If you can compile bytecode for MicroPython on a PC, then you should be able to transfer it and prepare it for execution. This would use a lot less of RAM. But whether you can inject the raw bytecode into shell and run it depends on how much MicroPython resembles CPython.

And yep, there are differences. As explained in another answer line by line execution can be tricky. So blocks of code is your best bet.

If I did blocks of code, that would get around the issues mentioned? — Bob, Sep 29 '17 at 20:23
It should. But if you are executing block by block, then you will have to keep correcting line numbers when an exception is raised. If you'd really like to execute everything command by command and track the whole progress I am afraid you do not have any other option but to take the code, make AST out of it, include a call to your added tracing function after each command into the AST, compile the AST and then send that bytecode to MCU and run it. It is not so hard to achieve as it sounds. — Dalen, Sep 29 '17 at 20:38

wim · Answer 3 · 2017-09-29T19:32:13.070

0

Is there ANY difference ...

Yes.

The code below, for example, works in .py file, but is a SyntaxError in the interactive interpreter:

x = 1
if x == 1:
    pass
x = 2

There are many other differences, but this alone should be enough to scare you away from the idea.

edited Sep 29 '17 at 19:32

answered Sep 29 '17 at 19:29

wim

338,267
99
616
750

I don’t have access to interpreter right now; why does your example fail in shell? – Bob Sep 29 '17 at 19:32
It's complicated. You don't really need to understand it. – wim Sep 29 '17 at 19:35
I’d like to so I can determine if it’s just a few corner cases – Bob Sep 29 '17 at 19:35
1

It's not just a few corner cases. You should find a better way to write the debugger, for example by using the [`system trace function`](https://docs.python.org/3/library/sys.html#sys.settrace). – wim Sep 29 '17 at 19:39
1

@wim, ...unfortunately, it looks like `sys.settrace()` isn't available in micropython, per https://github.com/micropython/micropython/issues/3009. Which does not *at all* mean I'm endorsing the OP's proposed hackery, as opposed to implementing the necessary VM support properly.. – Charles Duffy Sep 29 '17 at 19:41
@wim this micro controller doesn’t have that implemented: https://docs.micropython.org/en/latest/pyboard/library/sys.html – Bob Sep 29 '17 at 19:43
Hmmm. That's unfortunate. Then, I don't know how to implement a debugger in micropython. Good luck. – wim Sep 29 '17 at 19:43
the syntax error can be avoided by having a couple of extra new lines at the end of a block. – rajesh.kanakabandi Sep 29 '17 at 19:45
1

@rajesh.kanakabandi, granted, but that's just one case. It's by no means the only one. And if running code in the OP's toolchain means the line numbers reported in exceptions are wrong, that's not going to make users happy. – Charles Duffy Sep 29 '17 at 19:46
If I were you, the next thing I would be googling for is jupyter notebook support on upython. – wim Sep 29 '17 at 19:47
completely agree. I am just saying that this case is just not a show stopper. look at my answer to see how the shell behaves. – rajesh.kanakabandi Sep 29 '17 at 19:48
@wim how does jupyter help me? – Bob Sep 29 '17 at 20:05

Run Python script line-by-line in shell vs atomically

3 Answers3