Python fails to execute code line by line and causes folder not found error by reading, writing, copying and deleting folders and csv files

Question

I'm not sure how Python works internally to execute the code but I got a wired issue, which does not always happen but sometimes out of blue. I can even not locate the bug. In the frontend user clicks update button and the corresponding function is invoked. It takes request class as input, inside which there are important parameters like oldTemplateName and newTemplateName. The csv and folder update feature can be described as:

• If the template name is unchanged, the function just opens the existing csv file and updates the content from the request.
• If the template name is changed, the function firstly copies the whole folder where the old csv file locates to a new destination with new template name as folder name and the new csv file name, along with the subfolder and its content. Then the function updates the newly copied csv file and saves it. In the last, the old folder with old template name will be deleted.

The main library I use here is pathlib. I've also tried with os embedded in Python, but the error happens also. The corresponding code is:

        logger.debug("Begin to debug for csv manipulation")
        templateRootPath = current_app.config["TEMPLATE_PATH"]

        formatDict = format_template_data(request)
        
        oldTemplateFolderPath = os.path.join(templateRootPath,request.form["oldTemplate"])
        newTemplateFolderPath = os.path.join(templateRootPath,request.form["currentTemplate"].replace("-","_").replace(" ","_"))
        oldcsv=request.form["oldTemplate"]+".csv"
        newcsv=request.form["currentTemplate"].replace("-","_").replace(" ","_")+".csv"
        oldImgPath= os.path.join(oldTemplateFolderPath,"Img")             
        newImgPath= os.path.join(newTemplateFolderPath,"Img") 
        logger.debug("Old template folder path is {}".format(oldTemplateFolderPath))
        logger.debug("Old csv filename is {}".format(oldcsv))
        logger.debug("Old image path is {}".format(oldImgPath))
        logger.debug("New template folder path is {}".format(newTemplateFolderPath))
        logger.debug("New csv filename is {}".format(newcsv))
        logger.debug("New image path is {}".format(newImgPath))

        if oldTemplateFolderPath==newTemplateFolderPath:
            logger.debug("Template name is unchanged")
            csvName = os.path.join(newTemplateFolderPath,newcsv)            
            fieldname=["filename","caption","content","hyperlink","imgNr","position","chapter"]
            with open(csvName,'w',newline='') as f:
                thewriter = csv.DictWriter(f,fieldnames=fieldname)
                thewriter.writeheader()
                for item in formatDict.items():
                    for v in item[1]:
                        thewriter.writerow({
                            "filename":secure_filename(v["filename"].replace("-","_").replace(" ","")),
                            "caption":v["caption"],
                            "content":v["content"],
                            "hyperlink":v["hyperlink"],
                            "imgNr":v["imgNr"],
                            "position":v["position"],
                            "chapter":v["chapter"]
                        })
            save_img(templateRootPath,request.form.to_dict()["currentTemplate"].replace("-","_").replace(" ","_"))  
            logger.debug("Image save success")
        else:
            logger.debug("Template name is changed.")
            Path(newImgPath).mkdir(parents=True,exist_ok=True)
            logger.debug("New image folder {} is created.".format(newImgPath))
            for file_name in os.listdir(oldImgPath):
                logger.debug("Loop indside the old image folder")
                full_file_name = os.path.join(oldImgPath, file_name)
                full_file_name_new = os.path.join(newImgPath, file_name)
                if (not Path(full_file_name_new).exists()) and Path(full_file_name_new).parent.exists():
                    logger.debug("New image file {} non exists".format(full_file_name_new))
                    logger.debug("New image folder {} exists".format(str(Path(full_file_name_new).parent)))
                    shutil.copyfile(full_file_name,full_file_name_new)
                    logger.debug("Copy image into new image folder")
            for csv_name in os.listdir(oldTemplateFolderPath):
                logger.debug("Loop indside the old template folder")
                full_csv_name = os.path.join(oldTemplateFolderPath, csv_name)
                full_csv_name_new = os.path.join(newTemplateFolderPath, newcsv)
                if (not Path(full_csv_name_new).exists()) and Path(full_csv_name_new).parent.exists() and os.path.isfile(full_csv_name):
                    logger.debug("New csv file {} non exists".format(full_file_name_new))
                    logger.debug("New csv folder {} exists".format(str(Path(full_csv_name_new).parent)))
                    logger.debug("File {} is a file".format(full_csv_name))
                    shutil.copyfile(full_csv_name,full_csv_name_new)
                    logger.debug("Copy csv into new template folder")

            csvName = os.path.join(newTemplateFolderPath,newcsv)            
            fieldname=["filename","caption","content","hyperlink","imgNr","position","chapter"]
            with open(csvName,'w',newline='') as f:
                thewriter = csv.DictWriter(f,fieldnames=fieldname)
                thewriter.writeheader()
                for item in formatDict.items():
                    for v in item[1]:
                        thewriter.writerow({
                            "filename":secure_filename(v["filename"].replace("-","_").replace(" ","")),
                            "caption":v["caption"],
                            "content":v["content"],
                            "hyperlink":v["hyperlink"],
                            "imgNr":v["imgNr"],
                            "position":v["position"],
                            "chapter":v["chapter"]
                        })
                

            newfile = os.path.join(newTemplateFolderPath,newcsv)

            logger.debug("Check if old template folder still exists")
            if os.path.exists(oldTemplateFolderPath):
                # time.sleep(1)
                logger.debug("Delete the whole old template folder")
                try:
                    shutil.rmtree(oldTemplateFolderPath,ignore_errors=True) 
                except FileNotFoundError as e:
                    logger.error(e.msg)
                logger.debug("Delete success") 

            # if oldTemplateFolderPath!=newTemplateFolderPath:
            tobeUpdated = Template.query.filter_by(template_name=request.form["oldTemplate"]).first()
            tobeUpdated.template_name=request.form["currentTemplate"].replace("-","_").replace(" ","_")
            tobeUpdated.path=newfile
            db.session.commit()

def save_img(basePath,templatename):
    if not os.path.exists(os.path.join(basePath,templatename)):
        Path(os.path.join(basePath,templatename)).mkdir(parents=True,exist_ok=True)
        # os.mkdir(os.path.join(basePath,templatename))
    imgPath = os.path.join(templatename,"img")
    imgFullname = os.path.join(basePath,imgPath)
    if not os.path.exists(imgFullname):                       
        os.mkdir(imgFullname)
    for item in request.files.to_dict().items():
        file = item[1]
        if file and allowed_file(file.filename):
            filename = secure_filename(file.filename)                        
            file.save(os.path.join(imgFullname,filename))

The function formate_template_date() is to format needed data from the request object into a dictionary which will be use to update the csv file. It has nothing to do with folder or file manipulation.

Then the issue comes, sometimes, saying the old folder path does not exist.But!! the new folder has been created and everything is updated,even the old folder is deleted. It can appear for the first several time and when I refresh the page or wait for some time, the issue disappears even I redo the the same opration. What's stranger is, the issue may not happen when I rollback (meaning delete the new folder and revocer the old folder).

I try to debug it but I find that the Python does not execute the code line by line. Especially in the folder delete part, after the first execution, the compiler goes back and reexecutes this line again, which is very frustrating because I have no idea! Of cause in this senario it catches exception that older folder does not exist because you've just delete it!

The code is a part of backend server, which is based on Flask framework. The database is MySql container on the docker.

I've struggled with the issue for weeks and I haven't found solution online. Hope I explain the issue clear.Could some one give me a hint or adivce.

Update:

I've update the code with logging module. The output shows that even in the successfule cases, some lines and loops are executed twice. I have no idea. The logging output is:

INFO 2021-04-26 14:12:35,559 113 - 127.0.0.1 - - [26/Apr/2021 14:12:35] "[37mPOST /creation/templates HTTP/1.1[0m" 200 -
INFO 2021-04-26 14:12:35,564 113 - 127.0.0.1 - - [26/Apr/2021 14:12:35] "[37mPOST /creation/templates HTTP/1.1[0m" 200 -
INFO 2021-04-26 14:17:06,734 113 - 127.0.0.1 - - [26/Apr/2021 14:17:06] "[37mGET /creation/newtemplates/Test_Template_bro HTTP/1.1[0m" 200 -
INFO 2021-04-26 14:17:06,768 113 - 127.0.0.1 - - [26/Apr/2021 14:17:06] "[37mGET /creation/newtemplates/Test_Template_bro/OIP.jpg HTTP/1.1[0m" 200 -
INFO 2021-04-26 14:17:06,768 113 - 127.0.0.1 - - [26/Apr/2021 14:17:06] "[37mGET /creation/newtemplates/Test_Template_bro/OIP2.jpg HTTP/1.1[0m" 200 -
DEBUG 2021-04-26 14:17:12,573 100 - Begin to debug for csv manipulation
DEBUG 2021-04-26 14:17:12,583 100 - Begin to debug for csv manipulation
DEBUG 2021-04-26 14:17:13,606 111 - Old template folder path is C:\Users\eng\Documents\NEMOWizard\NEMOWOWizardEE\WOTemplates\Test_Template_bro
DEBUG 2021-04-26 14:17:13,612 112 - Old csv filename is Test_Template_bro.csv
DEBUG 2021-04-26 14:17:13,612 113 - Old image path is C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro\Img
DEBUG 2021-04-26 14:17:13,612 114 - New template folder path is C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed
DEBUG 2021-04-26 14:17:13,612 115 - New csv filename is Test_Template_bro_changed.csv
DEBUG 2021-04-26 14:17:13,612 116 - New image path is C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img
DEBUG 2021-04-26 14:17:13,612 142 - Template name is changed.
DEBUG 2021-04-26 14:17:13,617 111 - Old template folder path is C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro
DEBUG 2021-04-26 14:17:13,617 112 - Old csv filename is Test_Template_bro.csv
DEBUG 2021-04-26 14:17:13,617 113 - Old image path is C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro\Img
DEBUG 2021-04-26 14:17:13,617 114 - New template folder path is C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed
DEBUG 2021-04-26 14:17:13,617 115 - New csv filename is Test_Template_bro_changed.csv
DEBUG 2021-04-26 14:17:13,617 116 - New image path is C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img
DEBUG 2021-04-26 14:17:13,618 142 - Template name is changed.
DEBUG 2021-04-26 14:17:13,618 144 - New image folder C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img is created.
DEBUG 2021-04-26 14:17:13,618 144 - New image folder C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img is created.
DEBUG 2021-04-26 14:17:13,618 146 - Loop indside the old image folder
DEBUG 2021-04-26 14:17:13,619 150 - New image file C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img\OIP.jpg non exists
DEBUG 2021-04-26 14:17:13,619 151 - New image folder C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img exists
DEBUG 2021-04-26 14:17:13,619 146 - Loop indside the old image folder
DEBUG 2021-04-26 14:17:13,619 150 - New image file C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img\OIP.jpg non exists
DEBUG 2021-04-26 14:17:13,620 151 - New image folder C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img exists
DEBUG 2021-04-26 14:17:13,621 153 - Copy image into new image folder
DEBUG 2021-04-26 14:17:13,621 146 - Loop indside the old image folder
DEBUG 2021-04-26 14:17:13,622 150 - New image file C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img\OIP2.jpg non exists
DEBUG 2021-04-26 14:17:13,622 151 - New image folder C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img exists
DEBUG 2021-04-26 14:17:13,622 153 - Copy image into new image folder
DEBUG 2021-04-26 14:17:13,622 146 - Loop indside the old image folder
DEBUG 2021-04-26 14:17:13,623 150 - New image file C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img\OIP2.jpg non exists
DEBUG 2021-04-26 14:17:13,623 151 - New image folder C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img exists
DEBUG 2021-04-26 14:17:13,624 153 - Copy image into new image folder
DEBUG 2021-04-26 14:17:13,624 155 - Loop indside the old template folder
DEBUG 2021-04-26 14:17:13,624 155 - Loop indside the old template folder
DEBUG 2021-04-26 14:17:13,625 153 - Copy image into new image folder
DEBUG 2021-04-26 14:17:13,625 159 - New csv file C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img\OIP2.jpg non exists
DEBUG 2021-04-26 14:17:13,625 160 - New csv folder C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed exists
DEBUG 2021-04-26 14:17:13,625 161 - File C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro\Test_Template_bro.csv is a file
DEBUG 2021-04-26 14:17:13,625 155 - Loop indside the old template folder
DEBUG 2021-04-26 14:17:13,626 155 - Loop indside the old template folder
DEBUG 2021-04-26 14:17:13,626 159 - New csv file C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed\Img\OIP2.jpg non exists
DEBUG 2021-04-26 14:17:13,626 160 - New csv folder C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro_changed exists
DEBUG 2021-04-26 14:17:13,626 161 - File C:\Users\eng\Documents\NE\WOTemplates\Test_Template_bro\Test_Template_bro.csv is a file
DEBUG 2021-04-26 14:17:13,627 163 - Copy csv into new template folder
DEBUG 2021-04-26 14:17:13,628 163 - Copy csv into new template folder
DEBUG 2021-04-26 14:17:13,628 188 - Check if olde template folder still exist
DEBUG 2021-04-26 14:17:13,629 188 - Check if olde template folder still exist
DEBUG 2021-04-26 14:17:13,629 191 - Delete the whole old template folder
DEBUG 2021-04-26 14:17:13,629 191 - Delete the whole old template folder
DEBUG 2021-04-26 14:17:13,631 196 - Delete success
DEBUG 2021-04-26 14:17:13,633 196 - Delete success
INFO 2021-04-26 14:17:13,669 113 - 127.0.0.1 - - [26/Apr/2021 14:17:13] "[37mPOST /creation/templates HTTP/1.1[0m" 200 -
INFO 2021-04-26 14:17:13,670 113 - 127.0.0.1 - - [26/Apr/2021 14:17:13] "[37mPOST /creation/templates HTTP/1.1[0m" 200 -