0

I am trying to run my test cases which are nearly 40k with below scripts.

Just showing some part of script -

#!/bin/bash
# RUN script



echo "please run with: nice nohup ./run_script"

# working directory where script is stored
WORKING_DIR=$(pwd)

# temp directory to build and run the cmake ctest
BUILD_DIR=${BUILD_DIR:-/localtemp/build}

# clean and make build directory
rm -rf $BUILD_DIR
mkdir -p $BUILD_DIR
mkdir -p $BUILD_DIR/../result

cmake -G Ninja

ninja test

Note: I am using parallel threading with 6 cores to run my test cases .

In first attempt all are passing which is true as I fixed all the bugs in my test cases.

But some time If I want to re-run same script freshly, then I am getting the error in running some test cases out of 40k. But If I run that failing test cases separately(one by one) then they are passing perfectly.

So I assumed that rm -rf is taking some time to delete that old binary and symbols of all cases (40 GB files). so I need to wait for complete deletion and then run my script once again . So should I add some delay after rm -rf command in my script .

I read somewhere that rm -rf will return the status once it finishes the work .Then only next command executes. But my scnerio looks like showing that rm -rf is running in background.
Means I should not start new run immediately when I stopped the earlier run . I need to give some time to delete the old output from earlier run using rm -rf command in script (delay introduction) and run my below ninja command after that. Is that true?

fresskoma
  • 25,481
  • 10
  • 85
  • 128
jailaxmi k
  • 89
  • 3
  • 11
  • 1
    How exactly are you running parallel threads? If six instances of this shell script are running concurrently, of course they will trample each others' state if they share the same build directory. – tripleee Jan 06 '20 at 11:51
  • I am uinsg the Below ctest macro : CTEST_PARALLEL_LEVEL . This is a CMake Environment Variable. – jailaxmi k Jan 06 '20 at 11:54
  • https://cmake.org/cmake/help/latest/envvar/CTEST_PARALLEL_LEVEL.html – jailaxmi k Jan 06 '20 at 11:55

2 Answers2

1

Edit: It appears that, while the general question is easy to answer (i.e. rm does not run in the background) the concrete problem you are having is more complex and requires some potentially OS-level debugging. Nobody on StackOverflow will be able to do this for you :D

For your particular problem, I'd suggest using a different strategy than deleting the files to potentially avoid this hastle. In order to make sure that your build directory is pristine, you could use a new temporary directory in case you want to do a clean build.

Initial answer:

The rm command will not run "in the background" or in any way concurrent to other commands in your script unless you explicitly tell it to (e.g. using & at the end of the command). Once it has finished (i.e. the next command in your bash script executes) the files should have been deleted. Thus your problem probably lies somewhere else.

That being said, there may be differences in behaviour if, for example, you are using network shares, FUSE filesystems or any other mechanism that alters how or when your filesystem reacts to deletion request by the operating system.

fresskoma
  • 25,481
  • 10
  • 85
  • 128
  • True . I read the same. But i am getting error if I did not wait to delete the complete 40 GB old binaries .and If I wait till complete folder get empty,I am succeeding in running my test cases.It shows that rm -rf is finishing before complete deletion of all old binaries ?? – jailaxmi k Jan 06 '20 at 11:42
  • What is your setup like? Are the files on a local disk? What is `rm`s exit code? Could there be other factors that influence your test run that also happen to disappear when waiting? e.g. system load, some temporary files that get deleted,...? – fresskoma Jan 06 '20 at 11:44
  • before running the script one more time i am trying to run rm -rf manually .so I can see the old folders and binaries get deleted completely . Then I am not getting benefit of my script . – jailaxmi k Jan 06 '20 at 11:45
  • ,I am running it on my severer . How showed I check the rm -rf return value in my script .Can you guide me to put the if statement in my script to check the return status of rm -rf .If it is proper then only I can go for the next commands to run – jailaxmi k Jan 06 '20 at 11:48
  • I've updated my answer. I'd encourage you to do the debugging part yourself if you're so inclined, since this will be difficult for someone here to do for you. I've proposed a potential solution that you might be able to pull off with little effort. – fresskoma Jan 06 '20 at 12:15
  • Thanks.I will do the same . – jailaxmi k Jan 06 '20 at 12:59
1

It's a little bit more complicated than that. rm will not run in background. What is sure is that the deletion is done at least logically (thus from user viewpoint things are deleted); this means that on some OSes it is possible that the low-level deletion is done in the kernel even if the command has terminated (the kernel then really clean some internal structures behind your back). But that should not normally interfer with your script. Your problem is surely elsewhere.

It could be that some processes hold some files even if they seems deleted... Thus the disk could be not freed as you expected. You'll then have to waut completion of those processes so that kernel really cleans the files...

Jean-Baptiste Yunès
  • 34,548
  • 4
  • 48
  • 69