0

I'm copying a large amount of data using AzCopy and I need to have a way to retrieve the files that failed to copy.

AzCopy creates a nice log from each job and can do something like this:

Select-String -Path C:\Users\XXX\.azcopy\Projects\304c22cc-d37d-d743-7597-a160ac0ebad2.log -Pattern 'UPLOADFAILED'

But the output looks like that:

.azcopy\Projects\304c22cc-d37d-d743-7597-a160ac0ebad2.log:25528:2021/01/04 16:45:19 ERR: [P#0-T#2357] UPLOADFAILED: %5C%5C\UNC\fileserver.contoso.network\PROJ$\AAAA\BBBB\CCCC\Eigenerkl+�rungen.pdf_DOC001719.pdf : 000 : Could not check destination file existence. -> github.com/Azure/azure-storage-file-go/azfile.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-file-go@v0.8.0/azfile/zc_storage_error.go:42

I need to carve out only the file path and name from this output. In the exampe below, I need to carve out:

fileserver.contoso.network\PROJ$\AAAA\BBBB\CCCC\Eigenerkl+�rungen.pdf_DOC001719.pdf

Does anyone have any idea how? I can't search based on the file name because I have over 2000 files that have failed and I need to carve out all of them.

Kind regards, Wojciech

wrozansk
  • 1

2 Answers2

0
$file = 'C:\Users\XXX\.azcopy\Projects\304c22cc-d37d-d743-7597-a160ac0ebad2.log'

# Define the pattern as a regex that captures only the part of interest.
$pattern = '(?<=UPLOADFAILED:%5C%5C\\UNC\\)[^_]+'

(Select-String -Pattern $pattern -LiteralPath $file).Matches.Value

The assumptions are:

  • %5C%5C\UNC\ is a fixed string preceding the path of interest (note how the \ are escaped as \\ in order to treat them verbatim in the regex).

  • A _ character marks the end of the path.

Also note that Select-String matches case-insensitively by default; use -CaseSensitive as needed.

Finally, the presence of � (REPLACEMENT CHARACTER, U+FFFD) in your sample data suggests that the file's character encoding is being misinterpreted, which you may be able to fix via the -Encoding parameter. Then again, these characters may point to a prior problem that caused these paths to be listed as failed to begin with.

mklement0
  • 382,024
  • 64
  • 607
  • 775
0

Thank you! You pointed me in the right direction. Damn I need to learn regex. I made a small adjustment in the pattern though. This worked for me:

$pattern2 = '(?<=UPLOADFAILED:\s%5C%5C\?\\UNC\\).*(?=\s\:\s000)'

Thanks again, Wojtek

wrozansk
  • 1