Why is the provided Python script not printing all of the lines of the by it traced code?

Question

While using sligthly modified code provided in a 5-years old but only today by me answered question How print every line of a python script as its being executed (including the console)? I run into problems with understanding in detail how the code works and especially why the tracing is skipping to mention running the lines 5 and 8?

I am aware of the existence of another way to trace execution of a script with

python3 -m trace --trace --ignore-module sys,time --ignore-dir /sys:/usr tmp.py

but I would like to have the option of adapting the printed output content and format to my needs and to understand how the code works in detail in order to have an explanation why not all lines of the executed code appear in the trace.

Please feel encouraged by my question to provide another answer with better code (not being a dirty hack as this one that follows) as response to the other question my question here started with.

Below the code:

import subprocess
tracerCode = """
import sys
import inspect

class SetTrace(object):
    '''Although, this produces the desired result, it is 
    a very bad hack since it will most likely break depending on what 
        python_string_base   is'''

    def __init__(self, func):
        self.func = func

    def __enter__(self):
        sys.settrace(self.func)
        return self

    def __exit__(self, ext_type, exc_value, traceback):
        sys.settrace(None)

def monitor(frame, event, arg):
    if event == "line":
        file_dict = dict(enumerate("{}".split("|")))
        line_number = frame.f_lineno-30 # Amount of lines in THIS string
        if line_number > 0:
           print( "line " + str(line_number)+ ": " + file_dict[line_number] )
           sys.stdout.flush()
    return monitor

def run():
    {}

with SetTrace(monitor):
    run()

"""

codeToRun = """
    import time
    for _ in range(1): 
        print( 
'hello' 
             )
        time.sleep(
5
                  )
    print ( 'goodbye')
"""

tracerCode = tracerCode.format("|".join([i.strip() for i in codeToRun.split("\n")]), codeToRun)
#print(tracerCode)

proc = subprocess.Popen(['python3.11', '-'], stdin=subprocess.PIPE,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
proc.stdin.write(bytes(tracerCode,"utf-8"))
proc.stdin.close()
for line in proc.stdout:
    print( '{}'.format(line.strip()) )
proc.wait()

printing

b'line 1: import time'
b'line 2: for _ in range(1):'
b'line 3: print('
b"line 4: 'hello'"
b'line 3: print('
b'hello'
b'line 6: time.sleep('
b'line 7: 5'
b'line 6: time.sleep('
b'line 2: for _ in range(1):'
b"line 9: print ( 'goodbye')"
b'goodbye'

where lines 5 and 8 are not covered in the output.

Answer 1

The simplest explanation is that lines 5 and 8 are not executed as they are continuations of a previously executed line. The closing ')' is simply part of lines 3 or 6 respectively and has nothing to execute on it. Likewise, any extra lines in your code to execute don't get traced as empty lines.

As a demo, I've made some tweaks to your example to include the following:

1. For the initial print statement I extended the string to be printed over multiple lines using both implicit concatenation and the + operator. The implicit joins are not traced, while the line with the + operator is traced.
2. I also added an f-string callout to a function, which does get its own trace line.
3. I also added the sep and end arguments on their own lines
4. I changed the input parameter for the "goodbye" line to a multi-line string. Only the start of the multi-line string is traced
5. I added an additional print call at the end with multiple input args, and the end and sep arguments. It only gets traced once, not once per arg.

import subprocess
tracerCode = """
import sys
import inspect

class SetTrace(object):
    '''Although, this produces the desired result, it is 
    a very bad hack since it will most likely break depending on what 
        python_string_base   is'''

    def __init__(self, func):
        self.func = func

    def __enter__(self):
        sys.settrace(self.func)
        return self

    def __exit__(self, ext_type, exc_value, traceback):
        sys.settrace(None)

def monitor(frame, event, arg):
    if event == "line":
        file_dict = dict(enumerate("{}".split("|")))
        line_number = frame.f_lineno-30 # Amount of lines in THIS string
        if line_number > 0:
           print( "line " + str(line_number)+ ": " + file_dict[line_number] )
           sys.stdout.flush()
    return monitor

def run():
    {}

with SetTrace(monitor):
    run()

"""

codeToRun = """
    import time
    
    def word():
        return 'today'
    
    for _ in range(1): 
        print( 
          'hello '
          'world '
          'how ' + 'are ' +  'you '
          f'{word()}?',
          end=';\\n',
          sep='--' 
          )
        time.sleep(
5
                  )
    print ( '''
    goodbye
    cruel
    world
    '''
    )

    print('Testing1', 'testing2', 'testing3', sep=' - ', end='FIN')
"""

tracerCode = tracerCode.format("|".join([i.strip() for i in codeToRun.split("\n")]), codeToRun)
#print(tracerCode)

proc = subprocess.Popen(['python3.11', '-'], stdin=subprocess.PIPE,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
proc.stdin.write(bytes(tracerCode,"utf-8"))
proc.stdin.close()
for line in proc.stdout:
    print( '{}'.format(line.strip()) )
proc.wait()

Gives the output:

b'line 1: import time'
b'line 3: def word():'
b'line 6: for _ in range(1):'
b'line 7: print('
b"line 8: 'hello '"
b"line 10: 'how ' + 'are ' +  'you '"
b"line 11: f'{word()}?',"
b"line 4: return 'today'"
b"line 10: 'how ' + 'are ' +  'you '"
b"line 8: 'hello '"
b"line 12: end=';"
b"',"
b"line 13: sep='--'"
b'line 7: print('
b'hello world how are you today?;'
b'line 15: time.sleep('
b'line 16: 5'
b'line 15: time.sleep('
b'line 6: for _ in range(1):'
b"line 18: print ( '''"
b''
b'goodbye'
b'cruel'
b'world'
b''
b"line 25: print('Testing1', 'testing2', 'testing3', sep=' - ', end='FIN')"
b'Testing1 - testing2 - testing3FIN'