Windows batch files and multiple processors

Question

I have a Windows batch file that performs the same action multiple times, except on multiple files. Is there a way I can tell the computer to use different processor cores to speed up the process? Otherwise, it takes longer since the same processor is being used all the time.

Should be asked on SU, but likely be closed there as dup of something like http://superuser.com/questions/427550/start-a-background-task-from-windows-7-cmd. — Alexei Levenkov, Sep 05 '15 at 06:05
`DOS` doesn't support multiple processors. If you're talking about Windows command line batch files then use `start /affinity`, there are lots of examples. Anyway it won't speedup anything because Windows is automatically using the additional processors. — wOxxOm, Sep 05 '15 at 06:05
possible duplicate of [How to set processor affinity from Batch File for Windows XP?](http://stackoverflow.com/questions/827754/how-to-set-processor-affinity-from-batch-file-for-windows-xp) — wOxxOm, Sep 05 '15 at 06:05
You could pretend your computer was running a proper OS and use GNU Parallel... http://blogs.msdn.com/b/hpctrekker/archive/2013/03/30/preparing-and-uploading-datasets-for-hdinsight.aspx — Mark Setchell, Sep 05 '15 at 07:25
super hacky way to do it in a pinch, split your files and run multiple instances of your bat on them. I'm sure I'll hear about how bad this is in the next comment. — cliffclof, Apr 24 '17 at 08:44

score 3 · Answer 1 · answered Sep 05 '15 at 06:54

3

In windows batch files you can use multithreading with pipes or start /b.

Something like this should work

@echo off
setlocal EnableDelayedExpansion
set "file1=%~1"

:startExtraThread
if NOT "%~2"=="" (
  start "Thread for %~2" /b "cmd /c %~f0 %~2"
  shift
  goto :startExtraThread
)

do my job with file1

answered Sep 05 '15 at 06:54

jeb

78,592
17
171
225

Let's say I have a program myboxer.exe and a file block_base.dat. Let's also say that the program converts the file from one format to another. How would the code look in this case? – posfan12 Sep 06 '15 at 10:57

score 3 · Answer 2 · answered Sep 05 '15 at 13:58

Although a Batch file can start several parallel threads, there are multiple problems related to the synchronization of all of they. The usual synchro methods use the presence of a file like a semaphore, but the code that check for the semaphore-file waste CPU time. Although a delay can be inserted in this code, the result is usually cumbersome and tricky.

I developed an efficient multi-thread scheme for Batch files that use a pipe to get a synchro signal; this point allows the waiting code to not waste CPU time, so all CPU cores can be used for data processing. The program below is an example that process several files selected by a wild-card, but this code can easily be adapted to process any other set of data; this method is specifically designed for the case when just one thread ends its processing at same time and may fail if the processing of two or more files ends in a very short time interval.

@echo off
setlocal EnableDelayedExpansion

rem Multi-thread scheme in Batch files
rem Antonio Perez Ayala aka Aacini

set "myID=%~2"
if "%~1" neq "" goto %1

set "signal=X"
for /L %%i in (1,1,10) do set "signal=!signal!!signal!"

rem Create the list of start commands for the concurrent working threads
set "numThreads=%NUMBER_OF_PROCESSORS%"
set "threads="
del availableThread* 2> NUL
for /L %%i in (1,1,%numThreads%) do (
   set "threads=!threads! & start "" /B cmd /C "%~NX0" Thread %%i"
   > availableThread%%i echo %%i
)

echo Start of application
del requestForThread* 2> NUL
( %threads:~3% ) | "%~NX0" Main
echo End of application
goto :EOF


:Main

echo Main - started

rem Controlling code: process all files and assign each one to available threads
for /F "delims=" %%f in ('dir /A-D /B *.txt') do (

   rem Get the number of next available thread; waits for anyone, if necessary
   if not exist availableThread* (
      echo X > mainIsWaiting
      echo Main - waiting for an available thread
      set /P availableSignal=
      del mainIsWaiting
   )

   set "nextThread="
   for %%t in (availableThread*) do if not defined nextThread (
      set /P "nextThread=" < %%t
      del %%t
   )

   if not defined nextThread (
      echo SYNCHRO FAILED^^!
      goto waitThreadsEnds
   )

   rem Assign the next file to the available thread
   echo %%f> requestForThread!nextThread!
   echo Main - file "%%f" assigned to thread #!nextThread!

)

echo Main - all files sent to processing

rem Wait until all threads ends their tasks
:waitThreadsEnds
   set availableThreads=0
   for %%t in (availableThread*) do set /A availableThreads+=1
if %availableThreads% lss %numThreads% goto :waitThreadsEnds
del availableThread*

rem Send "exit" signals to all threads
for /L %%i in (1,1,%numThreads%) do echo exit> requestForThread%%i
echo Main - end
goto :EOF


:Thread

rem Wait until a request for this thread appear
if not exist requestForThread%myID% goto Thread
set "request="
set /P "request=" < requestForThread%myID%
if not defined request goto Thread
del requestForThread%myID%
if "%request%" equ "exit" goto :EOF

rem Process the file
echo %time% - Thread #%myID% start: "%request%" > CON
set /A seconds=6+%random:~-1%
ping localhost -n %seconds% > NUL
echo %time% - Thread #%myID% end:   "%request%" > CON

rem Inform that this thread is available for other task
> availableThread%myID% echo %myID%
if exist mainIsWaiting echo %signal:~0,1021%

goto Thread

Of course, in order for this method to work as expected, your Windows computer must process just this Batch file alone with no other concurrent process.

For a further explanation of the method used in this solution, see this post.

Windows batch files and multiple processors

2 Answers2