Building a directory listing via batch script. Need to exclude some directories. How?

Question

I work with hundreds of thousands of folders/files, most of which have certain log files. It takes a lot of time to manually navigate through the folders to grab the logs, so Ive been using a script to do it.

Its a 2 parter. First part runs overnight to parse all of the directories (they change daily and new logs are added very often, so I index everything nightly to keep the index current).

This first part is automatic and I never touch it, it just runs nightly. Output goes to a txt file, which is used when I execute the second script.

Second script is where I put in keywords of what log I am looking for. This script relies on the txt output from the first script to quickly locate the file and pull it down.

(The reason I do it this way is because its a 5 second task to parse 300,000 line text file looking for a file, and its a 15-30 minute job to search (index all of the network shares). This way I can get my logs in 5 seconds and not 30+ minutes).

Prior to last week the log directories consisted of about 5000 files and folders. This past weekend IT switched over to a new system, and it is now consisting of over 500,000 files and folders. They dumped archives, and a bunch of other files in there that I do not need.

First script used to index it in about 10-15 minutes (its a network share, actually 4 network shares), no problem overnight. While now this is a 4+ hour feat. I need to be able to exclude certain directories during indexing thereby reducing the 500,000 file/folder count back down to under 5,000.

Ive been using this command to index:

dir /b /-d /-p /s /A:-D > C:\output.txt

I need have the indexing skip any directory containing the words "Common" or "Old" in them, and potentially others as well. The goal is to save time by not going inside of these directories at all.

I tried looking into PowerShell to do this, but I know nothing about it.

I need the output to be in the following format:

C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\dbInstaller.exe
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\DisplayDriver.nvi
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\DisplayDriverExt.dll
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\license.txt
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\mcu.ex_
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\nvae.inf
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\nvak.inf
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\nvapi.dl_
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\nvapi64.dl_
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\nvcompiler.dl_
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\nvcompiler32.dl_
C:\NVIDIA\DisplayDriver\GeForce332.21Driver\Display.Driver\nvcplsetupeng.exe

Thanks!

Answer 1

@ECHO OFF
SETLOCAL
SET "sourcedir=."
(
FOR /f "delims=" %%a IN (
  'dir /b /s /ad "%sourcedir%\*" ^|findstr /v /i /l /g:q22903564.txt'
  ) DO ( FOR /f "delims=" %%b IN ('dir /b /a-d "%%a" 2^>nul') DO ECHO(%%a\%%b
)
)>newfile.txt

GOTO :EOF

I used a file named q22903564.txt containing exclusion words for my testing. Produces newfile.txt.

Naturally, choice of sourcedir, q22903564.txt and newfile.txt are all in your court...

@ECHO OFF
SETLOCAL
:: make a tempfile
:maketemp
SET "tempfile=%temp%\%random%"
IF EXIST "%tempfile%*" (GOTO maketemp) ELSE (ECHO.>"%tempfile%a")

SET "sourcedir=."

:: get a dir listing from the root
(
FOR /f "delims=" %%a IN (
 'dir /b /ad "%sourcedir%\*" ^|findstr /v /i /l /g:q22903564.txt'
 ) DO (ECHO("%%~fa")
)>"%tempfile%b"

:again

(
FOR /f "usebackqdelims=" %%a IN ("%tempfile%b") DO (
 FOR /f "delims=" %%b IN (
 'dir /b /ad "%%~a\*" ^|findstr /v /i /l /g:q22903564.txt'
 ) DO FOR /f "delims=" %%c IN ('ECHO("%%~a\%%b"^|findstr /x /v /i /l /g:"%tempfile%b"') DO ECHO "%%~c"
)
)>"%tempfile%c"

FOR %%a IN ("%tempfile%c") DO SET /a sizec=%%~za
IF %sizec% gtr 0 TYPE "%tempfile%c">>"%tempfile%b"&GOTO again

(
FOR /f "usebackqdelims=" %%a IN ("%tempfile%b") DO (
 FOR /f "delims=" %%b IN (
 'dir /b /a-d "%%~a\*" 2^>nul'
 ) DO ECHO(%%~a\%%b
)
)>newfile.txt

DEL "%tempfile%*"

GOTO :eof

Well - need to charge you extra for doing what, according to some, is impossible.

---

@ECHO OFF
SETLOCAL
:: make a tempfile
:maketemp
SET "tempfile=%temp%\%random%"
IF EXIST "%tempfile%*" (GOTO maketemp) ELSE (ECHO.>"%tempfile%a")

SET "sourcedir=."

:: get a dir listing from the root
(
FOR /f "delims=" %%a IN (
 'dir /b /ad "%sourcedir%\*" ^|findstr /v /i /l /g:q22903564.txt'
 ) DO (ECHO("%%~fa")
)>"%tempfile%b"

SET /a sizec=0

:again
SET "skipcnt=usebackq"
IF %sizec% gtr 0 (
 FOR /f %%a IN ('type "%tempfile%b" ^|find /c /v ""') DO SET "skipcnt=usebackqskip=%%a"
 TYPE "%tempfile%c">>"%tempfile%b"
)

(
FOR /f "%skipcnt%delims=" %%a IN ("%tempfile%b") DO (
 FOR /f "delims=" %%b IN (
 'dir /b /ad "%%~a\*" ^|findstr /v /i /l /g:q22903564.txt'
 ) DO FOR /f "delims=" %%c IN ('ECHO("%%~a\%%b"^|findstr /x /v /i /l /g:"%tempfile%b"') DO ECHO "%%~c"
)
)>"%tempfile%c"

FOR %%a IN ("%tempfile%c") DO SET /a sizec=%%~za
IF %sizec% gtr 0 GOTO again

(
FOR /f "usebackqdelims=" %%a IN ("%tempfile%b") DO (
 FOR /f "delims=" %%b IN (
 'dir /b /a-d "%%~a\*" 2^>nul'
 ) DO ECHO(%%~a\%%b
)
)>newfile.txt

DEL "%tempfile%*"
GOTO :eof

Third time's a charm. This version works out how many lines of directory it's already processed and skips those lines when re-processing the concatenated lists b and c. IOW, it only processes the additions.

And in typing this, I've realised there's another little simplification... have to wait for morning tea, though...

---

@ECHO OFF
SETLOCAL
:: make a tempfile
:maketemp
SET "tempfile=%temp%\%random%"
IF EXIST "%tempfile%*" (GOTO maketemp) ELSE (ECHO.>"%tempfile%a")

SET "sourcedir=."

:: get a dir listing from the root
(
FOR /f "delims=" %%a IN (
 'dir /b /ad "%sourcedir%\*" ^|findstr /v /i /l /g:q22903564.txt'
 ) DO ECHO(%%~fa
)>"%tempfile%c"

SET /a sizec=0

:again
SET "skipcnt=usebackq"
IF %sizec% neq 0 FOR /f %%a IN ('type "%tempfile%b" ^|find /c /v ""') DO SET "skipcnt=usebackqskip=%%a"
TYPE "%tempfile%c">>"%tempfile%b"

(
FOR /f "%skipcnt%delims=" %%a IN ("%tempfile%b") DO (
 FOR /f "delims=" %%b IN (
 'dir /b /ad "%%a\*" ^|findstr /v /i /l /g:q22903564.txt'
 ) DO ECHO(%%a\%%b
)
)>"%tempfile%c"

FOR %%a IN ("%tempfile%c") DO SET /a sizec=%%~za
IF %sizec% gtr 0 GOTO again

(
FOR /f "usebackqdelims=" %%a IN ("%tempfile%b") DO (
 FOR /f "delims=" %%b IN (
 'dir /b /a-d "%%a\*" 2^>nul'
 ) DO ECHO(%%a\%%b
)
)>newfile.txt

DEL "%tempfile%*"
GOTO :EOF

This time for sure! (no lion's roar thought)

I realised that the re-filtering against the original tempfileb content was redundant - and removing that meant that the quoting regime to overcome awkward characters could be scaled back. Then in reality the original file was the original addition (tempfilec) to an empty tempfileb, so - a little reorganisation and hand-waving (because 'skip=0' is invalid) and the result is that the only directories that are scanned later are thos that have beed added in the last iteration, optimising the directory-scan methodology.

Now the consequences are that the resultant list will be \a\, \b\, \c\, \a\a1\, \b\b1\, and so on - but a simple sort of the result should take care of that, if required.

Answer 2

This may help: it excludes the terms in line 3 where you see "Common Old"

It creates a list of the folders, filters out the excluded terms, and uses the remaining folders to create the list.

If you only want *.log *.txt files etc then it could be made more efficient again - otherwise remove the *.log *.txt terms in the batch file in line 7

@echo off
dir /b /ad /s /-p > "C:\output.txt"
findstr /v /i "Common Old" "C:\output.txt" >"C:\output2.txt"
del "C:\output.txt"

for /f "usebackq delims=" %%a in ("C:\output2.txt") do (
   pushd "%%a"
      for /f "delims=" %%b in ('dir *.log *.txt /b /-p /a-d') do >>"C:\output.txt" echo(%%a\%%b
   popd
)