os.chdir() caused unexpected behaviour with python multithreading

Question

I'm new to multi-threading in Python. In my code, I called a function which changes its working directory with chdir() as follows.

import threading
import os
import shutil

def sayHello(dirName,userName):
    if not os.path.exists(dirName):
        os.makedirs(dirName)
    else:
        shutil.rmtree(dirName)
        os.makedirs(dirName)

    os.chdir(dirName)
    f = open("hello.txt","w")
    f.write("Hello %s\n" %userName)
    f.close()

thread1 = threading.Thread(target=sayHello,args=('hiDir1','Andrew'))
thread2 = threading.Thread(target=sayHello,args=('hiDir2','Michael'))

thread1.start()
thread2.start()

thread1.join()
thread2.join()

Expected behavior is,

1. thread1 : Create "hiDir1" directory, create "hello.txt" inside "hiDir1" and print "Hello Andrew" in "hello.txt"
2. thread2 : Create "hiDir2" directory, create "hello.txt" inside "hiDir2" and print "Hello Michael" in "hello.txt"

When I ran the code for the first time, it ran without errors. All files were generated correctly. But "hiDir2" was inside "hiDir1".

Without deleting the generated files, I ran it for the second time. Both directories were there. But only "hiDir2" had the correct text file with correct message printed on file. "hiDir1" didn't have the text file. Following error was popped.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "threadingError.py", line 9, in sayHello
    shutil.rmtree(dirName)
  File "/usr/lib/python3.5/shutil.py", line 478, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 476, in rmtree
    os.rmdir(path)
FileNotFoundError: [Errno 2] No such file or directory: 'hiDir1'ode here

When I ran it for 3rd time without deleting the files, vice versa of the second time run occurred. Both directories were there. But only "hiDir1" had the text file with correct output. 'hiDir2' was empty. Following error message was there.

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "threadingError.py", line 12, in sayHello
    os.chdir(dirName)
FileNotFoundError: [Errno 2] No such file or directory: 'hiDir2'

When I ran this repeatedly, 2nd and 3rd occurrences occurred exactly one after other.(How can this happen? It should give the same output each time, isn't it?)

As I understood, the issue is with 'chdir()'. So I rearranged the code getting rid of 'chdir()' as follows.

import threading
import os
import shutil

def sayHello(dirName,userName):
    if not os.path.exists(dirName):
        os.makedirs(dirName)
    else:
        shutil.rmtree(dirName)
        os.makedirs(dirName)

    filePath1 = dirName+'/hello.txt'
    print("filePath1: ", filePath1)
    # os.chdir(dirName)
    f = open(dirName+'/hello.txt',"w")
    f.write("Hello %s\n" %userName)
    f.close()

thread1 = threading.Thread(target=sayHello,args=('hiDir1','Andrew'))
thread2 = threading.Thread(target=sayHello,args=('hiDir2','Michael'))

thread1.start()
thread2.start()

thread1.join()
thread2.join()

Then, there was no issue. Code ran as expected. Is there anything wrong with os.chdir() when used in python multi-threading? Is this a bug in python threading module?

Thanks.

Answer 1

How about this:

import threading

from pathlib import Path


def say_hello(dir_name, username):
    """
    Creates dir_name (and its parents dirs) if not dir_name does not exist,
    then it creates a hello.txt file with the legent: 'Hello <username>'

    Examples:

    >>> say_hello('say_hello/slackmart', 'SLACKMART')
    say_hello/slackmart not found. Creating say_hello/slackmart
    Writing to say_hello/slackmart/hello.txt
    """
    path = Path(dir_name)
    if not path.exists():
        print(f'{dir_name} not found. Creating {dir_name}')
        path.mkdir(parents=True)
    else:
        # I wouldn't remove the dir_name path here as it could be dangerous
        print(f'Found {dir_name}')

    file_path = path / Path('hello.txt')  # Yes, you can join paths by using /
    print('Writing to', file_path)
    file_path.write_text(f'Hello {username}\n')


if __name__ == '__main__':
    andrew = threading.Thread(target=say_hello, args=('hiAndrewDir', 'Andrew'))
    michael = threading.Thread(target=say_hello, args=('hiMichaelDir', 'Michael'))

    andrew.start()
    michael.start()

    andrew.join()
    michael.join()

Demo time:

$ python3 sayhello.py

https://docs.python.org/3/library/pathlib.html