recursive checking/copying of files not reaching deeper directory level to check file attributes first

Question

I'm trying to copy files for a backup script. I know that the some files aren't going to copy without an error message because of attributes. I want to log those files and not try to copy them, since it takes forever to fail in the copy. Plus we want to go back and fix/log or remove any files that don't copy once we find them.

Unfortunately, what I'm trying is finding a deeper subdir, and trying to copy that dir, and it's copying all the files in the dir, when some have an attribute I wouldn't want to copy. Ex:

\drive\folder\Forms\C Forms\ and it says it's a good attribute "16"

but there's a file in that dir that the copy is trying to copy over to my dest dir and I'm seeing it has this for attributes:

file.pdf Archive, SparseFile, ReparsePoint, Offline.

I found that by cd to the dir, and using this command:

dir | select Name,Attributes

I looked at recurse not hitting all levels, but I think it's not that recurse isn't hitting all levels, it's that it's copying higher level directories and the files are coming along too, without checking attributes first.

I was looking how check if path at is dir or file for ideas, but when I tried adding the check around the copy, it was skipping adding the dir above the file in it:

if(!((Get-Item $_) -is [System.IO.DirectoryInfo])) #true gives dir, false gives file, only handle if file

This is my code right now for the copy function (I tried an exclude but it didn't work):

    function CopyFileToFolderUNC($SourcePath, $DestinationPath, $exclude){
       if(-Not (Test-Path $SourcePath )) #-PathType Container
       {
          $global:ErrorStrings.Add("Exception: No such path, $SourcePath;;  ")
          write-output  "++ Error: An error occured during copy operation. No such path, $SourcePath ++"
       }
       #$tempFileName = [System.IO.Path]::GetFileName($SourcePath)
       #$tempDestFileNamePath = "$DestinationPath\$tempFileName"
       Get-ChildItem -Path $SourcePath  -Recurse  -Depth 2| ForEach {$_} { #added file removed force -File , $SourcePath\*, $SourcePath\*\* -Depth 3
          #test if maybe we are dealing with an off-line file here
          #or use the enum name 'Offline'
          # or use the numeric value: 4096
          #$oldAttribs = $null
          $attr = $_.Attributes.value__
          write-output  "++ $_ $attr ++"
          if (($_.Attributes -eq [System.IO.FileAttributes]::Offline) -or ($_.Attributes.value__ -eq "4096")) {
             $_.Attributes=[System.IO.FileAttributes]::Normal
             #$oldAttribs = $_.Attributes
             #make it a 'normal' file with only the Archive bit set
             #$_.Attributes = [System.IO.FileAttributes]::Archive
             #log that the file was an issue and copy the other ones
             $global:ErrorStrings.Add("Found offline file in backup dir, $_. Logging info and not copying this file. Offline. Please investigate.;;  ")
             write-output  "++ Error: An error occured during copy operation. No such path or file, Offline $_ ++"
          } #if
          elseif(($_.Attributes -eq [System.IO.Fileattributes]::Archive) -or ($_.Attributes.value__ -eq "32")) {
             $global:ErrorStrings.Add("Found archive file in backup dir, $_. Logging info and not copying this file. Archive. Please investigate.;;  ")
             write-output  "++ Error: An error occured during copy operation. No such path or file, Archive $_ ++"
          } #elseif
          elseif(($_.Attributes -eq [System.IO.Fileattributes]::SparseFile) -or ($_.Attributes.value__ -eq "512")) {
             $global:ErrorStrings.Add("Found sparse file in backup dir, $_. Logging info and not copying this file. SparseFile. Please investigate.;;  ")
             write-output  "++ Error: An error occured during copy operation. No such path or file, SparseFile $_ ++"
          } #elseif
          elseif(($_.Attributes -eq [System.IO.Fileattributes]::ReparsePoint) -or ($_.Attributes.value__ -eq "1024")) {
             $global:ErrorStrings.Add("Found reparse point file in backup dir, $_. Logging info and not copying this file. ReparsePoint. Please investigate.;;  ")
             write-output  "++ Error: An error occured during copy operation. No such path or file, ReparsePoint $_ ++"
          } #elseif
          elseif( ($_.Attributes.value__ -eq "5664")) { #($_.Attributes -eq [System.IO.Fileattributes]::ReparsePoint) -or
             $global:ErrorStrings.Add("Found reparse point file in backup dir, $_. Logging info and not copying this file. ReparsePoint. Please investigate.;;  ")
             write-output  "++ Error: An error occured during copy operation. No such path or file, ReparsePoint $_ ++"
          } #elseif
          else {

             #the file is not or no longer off-line, so proceed with the copy
             #only if it's a file and not a dir
             #if(!((Get-Item $_) -is [System.IO.DirectoryInfo])) #true gives dir, false gives file.  We want to copy the files.
             #{
                write-output "Copying $_"
                $_ | Copy-Item -Destination $DestinationPath -Force -Recurse -ErrorVariable errors
                foreach($error in $errors)
                {
                  if ($error.Exception -ne $null)
                  {
                      $global:ErrorStrings.Add("Exception: $($error.Exception);;  ")
                      write-output  "++ Error: An error occured during copy operation. Exception: $($error.Exception) ++"
                  }
                  write-output  "Error: An error occured during copy operation. Exception: $($error.Exception)"
                }
             #}
             #else
             #{
             #   write-output "this is a dir"
             #}
          } #else
    } #Get-ChildItem
 } #function

This is where it's called from:

$tempSource = @("\\drive\Svcs\Cr\C FORMS\",...) 
$ToLocation = @("E:\DisasterBackup\toLoc\All Forms \",...)   
$excludeNonSharepoint = @("\\drive\Svcs\Cred\op\zOLD_NOT USED_keep for now"...)

#non-sharepoint ones first
for($i=0; $i -lt ($tempSource.Length); $i++) {
   CopyFileToFolderUNC $tempSource[$i] $ToLocation[$i] $excludeNonSharepoint
}

As requested, Minimal, Complete, and Verifiable example:

So for this dir/files/attributes, the copy will complete as follows after checking:

source From Location:
\\drive\folder\Forms attrib = normal/16 
\\drive\folder\Forms\a.txt attrib = normal/16 will copy
\\drive\folder\Forms\b.txt attrib = not normal/32/512/1024/etc will not copy  but log
\\drive\folder\Forms\C Forms = normal/16 
\\drive\folder\Forms\C Forms\c.txt  attrib = normal/16 will copy
\\drive\folder\Forms\C Forms\d.txt attrib = not normal/32/512/1024/etc will not copy but log/write-output
\\drive\folder\Forms\C Forms\e.txt attrib = not normal/32/512/1024/etc will not copy but log/write-output
\\drive\folder\Forms\C Forms\f.txt attrib = not normal/32/512/1024/etc will not copy but log/write-output
\\drive\folder\Forms\C Forms\subdir attribute = normal/16 
\\drive\folder\Forms\C Forms\subdir\g.txt  attribute = normal/16 
\\drive\folder\Forms\C Forms\subdir\h.txt attribute = not normal/32/512/1024/etc will not copy but log/write-output

final To Location:
E:\DisasterBackup\toLoc\All Forms\a.txt
E:\DisasterBackup\toLoc\All Forms\C Forms
E:\DisasterBackup\toLoc\All Forms\C Forms
E:\DisasterBackup\toLoc\All Forms\C Forms\c.txt
E:\DisasterBackup\toLoc\All Forms\C Forms\subdir\
E:\DisasterBackup\toLoc\All Forms\C Forms\subdir\g.txt  

output string at the end of the script will be something like:
"Could not copy \\drive\folder\Forms\C Forms\d.txt attrib archive
Could not copy \\drive\folder\Forms\C Forms\e.txt attrib sparse
Could not copy \\drive\folder\Forms\C Forms\f.txt attrib reparse
Could not copy \\drive\folder\Forms\C Forms\subdir\h.txt attrib offline"

It's unclear to me if you don't want to copy files with special attributes - or want to copy those files but without the attributes. What about using Robocopy? And please read what a [mcve] is. — , Jan 04 '19 at 19:38
@lotpings - The ones with the bad attributes, I don't want to copy, just log them. I'll add the Minimal/Complete/Verifiable ex to the question. — Michele, Jan 04 '19 at 19:40
@trebleCode - The problem with the file attributes is that someone outside our dept. archived the files/etc and we lost a bunch, leaving the mess. We know the mess is there, but want to find new ones once we clean it up. This is what I heard happened before I started. — Michele, Jan 04 '19 at 20:16
@Michele are you stuck with cleaning up the SharePoint files, or do you have a SharePoint admin that can help? Or is that who left? — trebleCode, Jan 04 '19 at 20:19
@trebleCode - We're copying non-sharepoint files in this question. We aren't sharepoint admins, so have to copy them to a source dir to copy them. We don't have sharepoint powershell plugin. Keep it to non-sharepoint in this question. Sharepoint files are fine (not corrupt). Thanks! — Michele, Jan 04 '19 at 20:26
It may be useful to mention you [earlier question](https://stackoverflow.com/q/53946910/9898643) about try his — Theo, Jan 04 '19 at 21:43
Can't you simply use [`robocopy`](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy) and check its log afterwards? — Ansgar Wiechers, Jan 06 '19 at 23:37
@ansgar wiechers - I have a ton of time into the powershell scripts. It seems like I'm close to finishing and I'd like to figure out my current question. — Michele, Jan 07 '19 at 13:30
@theo, none of the answers worked in the other question. I thought that since the question was a couple or more weeks old, I should try to re-ask a little differently. Thanks for your help!! — Michele, Jan 07 '19 at 13:31

recursive checking/copying of files not reaching deeper directory level to check file attributes first

0 Answers0