2

Sorry in advance, I can't post actual code because of security restrictions at my job, but I'll try to make a contrived example.

I am working with python 3.6.1 and running a module in an Azure Pipeline (ADS 2019). In the module we have output done using a dictionary with the following structure

#dummy data, assume files could be in any order in any category

{
    "compliant": ['file1.py', 'file2.py'], #list of files which pass
    "non-compliant":['file3.py'], #list of files which fail
    "incompatible":['file4.py'] #list of files which could not be tested due to exceptions
}

When a failure occurs one of our customers wants the script to output the command to call a script that can be run to correct the non-compliant files. The program is written similar to what follows

result = some_func() #returns the above dict
print('compliant:')

for file in result['compliant']:
    print(file)

print('non-compliant:')
for file in result['non-compliant']:
    print(file)

print('incompatible:')
for file in result['incompatible']:
    print(file)

# prints a string to sys.stderr simillar to python -m script arg1 arg2 ...
# script that is output is based on the arguments used to call
print_command_to_fix(sys.argv)

When run normally I would get the correct output like follows:

#correct output: occurs on bash and cmd

compliant:
file1.py
file2.py
non-compliant:
file3.py
incompatible:
file4.py

python -m script arg1 arg2 arg_to_fix

when I run on the Azure Pipeline though, the output gets interleaved like follows

#incorrect output: occurs only on azure pipeline runs

compliant:
python -m script arg1 arg2  arg_to_fix
file1.py
file2.py
non-compliant:
file3.py
incompatible:
file4.py

Whether I try to use print or sys.stderr.write it doesn't seem to resolve the interleave, and I'm assuming the print_command_to_fix() is being called asynchronously somehow. But my guess probably isn't accurate since I haven't been working with ADS or python for very long.

TL;DR: What am I doing wrong to get the above interleaved output on Pipelines only?

Edit: clarified certain points and fixed typos

jleibman
  • 157
  • 1
  • 9
  • How did you config your build definition? Did you use the latest build agent? Please try to run the code manually on your build agent machine to check the result. Also, set system.debug to True in build pipeline and share the entire log. – Cece Dong - MSFT May 08 '20 at 05:41

1 Answers1

0

Discovered the answer after a few hours of troubleshooting and solutions.

ADS tracks both output streams in the program but does it asynchronously. The error was cause by outputting to both stdout and stderr. This being the case, outputting all output to one stream resolved the issue. The approach I took ended up being something like follows

result = some_func() #returns the above dict
output = []
output.append('compliant:')
output.extend(result['compliant'])

output.append(file)
output.extend(result['non-compliant'])

output.append('incompatible:')
output.extendresult['incompatible'])

# returns a string to simillar to python -m script arg1 arg2 ...
# script that is output is based on the arguments used to call
output.append(format_command_to_fix(sys.argv))
print('\n'.join(output))

Alternatively, I imagine other techniques for outputting async information should resolve as well.

jleibman
  • 157
  • 1
  • 9
  • Great to see your issue has been solved. You could [Accept it as an Answer](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work), this can be beneficial to other community members reading this thread. – Cece Dong - MSFT May 11 '20 at 02:15
  • Thanks for notifying me, I thought it was already closed but I guess I forgot. – jleibman May 11 '20 at 02:27