Removing duplicate values from a PowerShell array

Question

How can I remove duplicates from a PowerShell array?

$a = @(1,2,3,4,5,5,6,7,8,9,0,0)

score 251 · Accepted Answer · edited Jun 22 '19 at 18:51

251

Use Select-Object (whose alias is select) with the -Unique switch; e.g.:

$a = @(1,2,3,4,5,5,6,7,8,9,0,0)
$a = $a | select -Unique

edited Jun 22 '19 at 18:51

mklement0

382,024
64
607
775

answered Sep 08 '09 at 03:46

Keith Hill

194,368
42
353
369

9

That was too easy :-(. In PowerShell 2 you can also use `Get-Unique` (or `gu`) if your array is already sorted. – Joey Sep 08 '09 at 05:30
2

Johannes, Get-Unique is available in v1 :) – Shay Levy Sep 08 '09 at 06:58
5

cool, this works fine, just a Note, if you want to be concise, it could be shortened even to `-u`. `select -u` I would use this on a command line, but written in a code, it's suggested to use the full PS wording: `Select-Object -Unique` – papo May 27 '18 at 05:12
This only works with single digit? For a large list of array it seems not to works. – EagleDev Jul 30 '18 at 17:55
This isn't working correctly for me, it's only keeping the one value that has a duplicate, almost like it's creating a list of duplicates. – Mike Q Apr 02 '21 at 21:51
1

Just to note, select -unique doesn't account for case sensitivity so this won't work in a lot of cases. sort -unique is the correct method, (although you'll end up with a sorted array) – az1d Aug 25 '21 at 11:50

score 100 · Answer 2 · edited Jun 22 '19 at 18:53

100

Another option is to use Sort-Object (whose alias is sort, but only on Windows) with the
-Unique switch, which combines sorting with removal of duplicates:

$a | sort -unique

edited Jun 22 '19 at 18:53

mklement0

382,024
64
607
775

answered Sep 08 '09 at 06:54

Shay Levy

121,444
32
184
206

2

This also solves my next problem which is how do I sort it. Thanks! – Registered User May 18 '12 at 16:38
3

This one also looks a tiny bit faster. – Gneo Pompeo May 22 '14 at 18:10
Note that this does not preserve order of the elements, e.g. `@("B", "a", "A") | sort -unique` produces `A B`. – roxton Feb 11 '21 at 19:56
2

this is the better answer. select -unique doesn't account for case sensitivity – az1d Aug 25 '21 at 11:51

DarkLite1 · Answer 3 · 2022-12-01T07:13:24.757

26

In case you want to be fully bombproof, this is what I would advise:

@('Apples', 'Apples ', 'APPLES', 'Banana') | 
    Sort-Object -Property @{Expression={$_.Trim()}} -Unique

Output:

Apples
Banana

This uses the Property parameter to first Trim() the strings, so extra spaces are removed and then selects only the -Unique values.

More info on Sort-Object:

Get-Help Sort-Object -ShowWindow

edited Dec 01 '22 at 07:13

answered Jun 01 '17 at 10:58

DarkLite1

13,637
40
117
214

1

Awesome answer. Solved my issue, `Get-Unique` didn't work because spaces – Kellen Stuart Nov 01 '17 at 20:29

score 10 · Answer 4 · edited Jun 22 '19 at 18:55

10

$a | sort -unique

This works with case-insensitive, therefore removing duplicates strings with differing cases. Solved my problem.

$ServerList = @(
    "FS3",
    "HQ2",
    "hq2"
) | sort -Unique

$ServerList

The above outputs:

FS3
HQ2

edited Jun 22 '19 at 18:55

mklement0

382,024
64
607
775

answered Apr 24 '17 at 18:50

Briscomo

101
1
2

this also works well for arrays with multiple members – JoeRod Feb 07 '19 at 13:55

Omzig · Answer 5 · 2021-01-27T17:06:53.930

This is how you get unique from an array with two or more properties. The sort is vital and the key to getting it to work correctly. Otherwise you just get one item returned.

PowerShell Script:

$objects = @(
    [PSCustomObject] @{ Message = "1"; MachineName = "1" }
    [PSCustomObject] @{ Message = "2"; MachineName = "1" }
    [PSCustomObject] @{ Message = "3"; MachineName = "1" }
    [PSCustomObject] @{ Message = "4"; MachineName = "1" }
    [PSCustomObject] @{ Message = "5"; MachineName = "1" }
    [PSCustomObject] @{ Message = "1"; MachineName = "2" }
    [PSCustomObject] @{ Message = "2"; MachineName = "2" }
    [PSCustomObject] @{ Message = "3"; MachineName = "2" }
    [PSCustomObject] @{ Message = "4"; MachineName = "2" }
    [PSCustomObject] @{ Message = "5"; MachineName = "2" }
    [PSCustomObject] @{ Message = "1"; MachineName = "1" }
    [PSCustomObject] @{ Message = "2"; MachineName = "1" }
    [PSCustomObject] @{ Message = "3"; MachineName = "1" }
    [PSCustomObject] @{ Message = "4"; MachineName = "1" }
    [PSCustomObject] @{ Message = "5"; MachineName = "1" }
    [PSCustomObject] @{ Message = "1"; MachineName = "2" }
    [PSCustomObject] @{ Message = "2"; MachineName = "2" }
    [PSCustomObject] @{ Message = "3"; MachineName = "2" }
    [PSCustomObject] @{ Message = "4"; MachineName = "2" }
    [PSCustomObject] @{ Message = "5"; MachineName = "2" }
)

Write-Host "Sorted on both properties with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property Message,MachineName -Unique | Out-Host

Write-Host "Sorted on just Message with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property Message -Unique | Out-Host

Write-Host "Sorted on just MachineName with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property MachineName -Unique | Out-Host

Output:

Sorted on both properties with -Unique

Message MachineName
------- -----------
1       1          
1       2          
2       1          
2       2          
3       1          
3       2          
4       1          
4       2          
5       1          
5       2          


Sorted on just Message with -Unique

Message MachineName
------- -----------
1       1          
2       1          
3       1          
4       1          
5       2          


Sorted on just MachineName with -Unique

Message MachineName
------- -----------
1       1          
3       2

Source: https://powershell.org/forums/topic/need-to-unique-based-on-multiple-properties/

For loops that you might not know the header for (ie. parsing automatically some csv data) - use Sort-Object * -unique — Sebastian Wiszowaty, Jun 15 '21 at 08:25

score 4 · Answer 6 · edited Jan 09 '17 at 17:20

4

If the list is sorted, you can use the Get-Unique cmdlet:

 $a | Get-Unique

edited Jan 09 '17 at 17:20

jpaugh

6,634
4
38
90

answered May 26 '16 at 18:32

Martin Brandl

56,134
13
133
172

2

It requires the list to be sorted on beforehand – Johny Skovdal Aug 30 '16 at 20:21

score 3 · Answer 7 · answered Jul 24 '18 at 21:56

3

With my method you can completely remove duplicate values, leaving you with values from the array that only had a count of 1. It was not clear if this is what the OP actually wanted however I was unable to find an example of this solution online so here it is.

$array=@'
Bananna
Apple
Carrot
Pear
Apricot
Pear
Bananna
'@ -split '\r\n'

($array | Group-Object -NoElement | ?{$_.count -eq 1}).Name

answered Jul 24 '18 at 21:56

Will

131
1
1
9

I really like this answer, but there's a big flaw with it. if there are multiple elements with the same name, all those elements are lost, it should instead keep one of them. This fixed it: `($properties | Group-Object -NoElement).Name | Get-Unique` – az1d Aug 25 '21 at 11:57

score 1 · Answer 8 · edited Aug 16 '19 at 06:21

1

Whether you're using SORT -UNIQUE, SELECT -UNIQUE or GET-UNIQUE from Powershell 2.0 to 5.1, all the examples given are on single Column arrays. I have yet to get this to function across Arrays with multiple Columns to REMOVE Duplicate Rows to leave single occurrences of a Row across said Columns, or develop an alternative script solution. Instead these cmdlets have only returned Rows in an Array that occurred ONCE with singular occurrence and dumped everything that had a duplicate. Typically I have to Remove Duplicates manually from the final CSV output in Excel to finish the report, but sometimes I would like to continue working with said data within Powershell after removing the duplicates.

edited Aug 16 '19 at 06:21

techguy1029

743
10
29

answered Nov 08 '17 at 01:30

Christopher M Ramos

29
2

this is where the Where-Object comes in handy. You first narrow the list down by using the right where clause, then pipe it to a sort-object, select one or more columsn to sort, and give the sort-object a -unique. – LPChip Feb 08 '19 at 12:46
1

@Christopher, please see my post to fix your problem. Consider removing this answer to clean up the solutions. – Omzig Jan 27 '21 at 17:00

roxton · Answer 9 · 2022-07-24T08:00:17.767

To get unique items from an array and preserve their order, you can use .NET HashSet:

$Array = @(1, 3, 1, 2)
$Set = New-Object -TypeName 'System.Collections.Generic.HashSet[int]' -ArgumentList (,[int[]]$Array)

# PS> $Set
# 1
# 3
# 2

Works best with string arrays that contain both uppercase and lowercase items where you need to preserve first occurrence of each item in case-insensitive manner:

$Array = @("B", "b", "a", "A")
$Set = New-Object -TypeName 'System.Collections.Generic.HashSet[string]' -ArgumentList ([string[]]$Array, [StringComparer]::OrdinalIgnoreCase)

# PS> $Set
# B
# a

Works as expected with other types.

Shortened syntax, compatible with PowerShell 5.1 and newer:

$Array = @("B", "b", "a", "A")
$Set = [Collections.Generic.HashSet[string]]::new([string[]]$Array, [StringComparer]::OrdinalIgnoreCase)

$Array = @(1, 3, 1, 2)
$Set = [Collections.Generic.HashSet[int]]::new([int[]]$Array)

score 1 · Answer 10 · answered Aug 25 '21 at 12:01

A lot of the provided answers give buggy results. select -Unqiue is not case sensitive. sort -Unique gives you sorted results, which you might want in the original order.

Will gave a great answer, but it's flawed as it discards all duplicate results but forgets to keep one of them.

This is a version I created that seems to work perfectly. It gives unique results back and retains the original sort order.

($properties | Group-Object -NoElement).Name | Get-Unique

Pulia Zlaya · Answer 11 · 2022-12-15T02:21:15.563

0

$a = @('D:/', 'two', 'three')
$s = "D:/"
$null -ne ($a | ? { $s -match $_ })  # Returns $true

another solution can be used here , u're welcome

edited Dec 15 '22 at 02:21

answered Dec 15 '22 at 00:42

Pulia Zlaya

51
4

1

Remember that Stack Overflow isn't just intended to solve the immediate problem, but also to help future readers find solutions to similar problems, which requires understanding the underlying code. This is especially important for members of our community who are beginners, and not familiar with the syntax. Given that, **can you [edit] your answer to include an explanation of what you're doing** and why you believe it is the best approach? – Jeremy Caney Dec 15 '22 at 00:57

Removing duplicate values from a PowerShell array

11 Answers11

Linked

Related