Copy new files from external HD to Desktop with Powershell

Question

Thank you for reading my post and your advice :) The scenario is, that I have an HD built in my desktop where I collected files for many years. So one day I did a backup to an external HD, which I then took travelling and kept on collecting photos from my phone etc.

Since then I changed the folder structure on my desktop a lot, so I can't compare the folders/files 1on1.

The goal is obviously, that all new files that are on my external HD get copied to my internal HD, all put in a folder named like '2SORT'.

I found a faster version of compare-object (see comments) but yet the results are not correct, it will copy a lot of files that already exist.

Here's what I got in Powershell:

cls

$path_desktop = 'C:\Files\Media\Bilder'
$path_external = 'E:\Bilder'
$path_destination = 'C:\Files\Media\Bilder\2SORT'

$ref = Get-ChildItem -path $path_desktop -Recurse -File
$com = Get-ChildItem -path $path_external -Recurse -File

$file_Diffs = Compare-Object -ReferenceObject $ref -DifferenceObject $com | Where { $_.SideIndicator -eq '=>' }

$file_Diffs | 
foreach {

 $copyParams = @{}
 $copyParams.Path = $_.InputObject.FullName
 $copyParams.Destination = "$path_destination\" + $_.InputObject.Name

 copy-Item @copyParams -force -PassThru | Write-Host
     
}

I found an improved version of Compare-Object using hash-tables, a lot faster but the results make no sense. https://learn.microsoft.com/de-de/archive/blogs/ashleymcglone/use-hash-tables-to-go-faster-than-powershell-compare-object — siboney, Jan 21 '20 at 01:14
Have you considered `robocopy`? Copying all of the original files to `2SORT`, then `robocopy /mir E:\Bilder C:\Files\Media\Bilder\2SORT` should only copy the new files to the `2SORT` directory. — Glenn, Jan 21 '20 at 02:35
Thanks Glenn, this could be an option, but sadly the amount of files (100gb+) we're talking about is way to big! Can't be so hard to compare some filenames is it? :) — siboney, Jan 21 '20 at 13:05

Brandon McClure · Answer 1 · 2020-01-22T14:49:11.463

If this is something you would do regularly, I would recommend refactoring to use robocopy. After the first run any subsequent runs will be much faster because robocopy was designed to perform these tasks. The first run is going to be slow no matter what tool you use.

Compare-Object is going to compare property values. By passing a bunch of values in it will have more work to do. Also, you are not actually moving any files while this function is running, so if this operation is canceled, your script has not moved anything. If you just want to compare a list of file names in each and copy if they exist in $ref but not $com, You could limit the properties you are getting, and iterate/move over each in $ref manually like I do below.

If you want to keep using Compare-Object take a look at it's -Property parameter to limit the number of properties it is comparing.

$path_desktop = 'C:\Files\Media\Bilder'
$path_external = 'E:\Bilder'
$path_destination = 'C:\Files\Media\Bilder\2SORT'

$ref = Get-ChildItem -path $path_desktop -Recurse -File | select name, fullname
$com = Get-ChildItem -path $path_external -Recurse -File | select name, fullname

$ref | 
foreach {
    if ($_.name -in $com | select -ExpandProperty name)
        continue;

    --Copy file
}

You could also get other file properties as needed, then you could use those properties when you are doing the comparison (either "manually" or using compare-object and iterating over the diff)

Get-ChildItem -path $path_desktop -Recurse -File | select Name,fullname, LastWriteTime

to see what properties Get-ChildItem returns, pass it into `Get-Member -Type Properties

Get-ChildItem -path $path_desktop -Recurse -File | Get-Member -Type Properties

Thanks for your response! the problem is when I expandproperty, I lose the filepath to the file that was "found" to be new/not existent.. The -in / -notin operator sadly can't just "find" the filename in the object, because it will compare all properties — siboney, Jan 22 '20 at 14:14
@silboney, I updated the answer to do the iteration/logic using both properties. I also added a bit about using `Compare-Object` to do this by limiting the properties it is comparing — Brandon McClure, Jan 22 '20 at 14:50

siboney · Answer 2 · 2020-01-22T19:13:29.743

In the end I had to compare the files by their MD5-Hash because I got many duplicates with just different 'lastwritetime':

cls

    $path_desktop = 'C:\Files\Media\Bilder\'
    $path_external = 'C:\Files\Media\ext'
    $path_destination = 'C:\Files\Media\Bilder\2SORT'
    $path_intcsv = 'C:\Files\Media\Bilder\inthash2.csv'
    $path_extcsv = 'C:\Files\Media\Bilder\exthash2.csv'

    $count=0
    $res = @()

    Get-ChildItem -path $path_desktop -Recurse -File | % {
     
     Write-Host ($count++)
     $hash = Get-FileHash $_.FullName -Algorithm MD5
     $res += New-Object PSObject -Property @{Hash=$hash.Hash; FullName=$_.FullName}
    }

    $res | epcsv -path $path_intcsv -NoTypeInformation

    $count=0
    $res = @()

    Get-ChildItem -path $path_external -Recurse -File | % {
     Write-Host ($count++)
     $hash = Get-FileHash $_.FullName -Algorithm MD5
     $res += New-Object PSObject -Property @{Hash=$hash.Hash; FullName=$_.FullName}
    }

    $res | epcsv -path $path_extcsv -NoTypeInformation
     

    $int = Import-Csv $path_intcsv 
    $ext = Import-Csv $path_extcsv
      
    $IntHash = @{}          
    $ExtHash = @{}          
    $DifHash = @{}          
     
    $int | ForEach-Object {
     $newKey=$_.Hash
     if (!$IntHash.ContainsKey($newKey)) {
      $IntHash.Add($newKey, $null)
     }
    }               

    $ext | ForEach-Object {
     $newKey=$_.Hash
     if (!$ExtHash.ContainsKey($newKey)) {
      $ExtHash.Add($newKey, $_.FullName)
     }
    }  

    ForEach ($key in $ExtHash.Keys) {            
     If (!$IntHash.ContainsKey($key)) {            
      $DifHash.Add($key, $ExtHash[$key])
     }    
    }     

    #check folder exists
    if (-not (test-path $path_destination)) { 
      $make = new-item -ItemType directory -Path $path_destination
    }      

    #copy files
    ForEach ($key in $DifHash.Keys) {
     $fullname = $DifHash[$key]
     Copy-Item $fullname -Destination $path_destination -Force
    }

Copy new files from external HD to Desktop with Powershell

2 Answers2