How to remove duplicates in an interface array

Question

Lets say the input can contain string or integer values

names = ["rahul", "rohit","srujan", "rahul"] --> output = ["rahul", "rohit","srujan"] 
 age=[12,18,12,21] --> output = [12,18,21]

we can make use of this function to filter duplicates

package main
  
import (
"fmt"
)
  
func unique(intSlice []int) []int {
    keys := make(map[int]bool)
    list := []int{} 
    for _, entry := range intSlice {
        if _, value := keys[entry]; !value {
            keys[entry] = true
            list = append(list, entry)
        }
    }    
    return list
}
  
func main() {
    intSlice := []int{1,5,3,6,9,9,4,2,3,1,5}
    fmt.Println(intSlice) 
    uniqueSlice := unique(intSlice)
    fmt.Println(uniqueSlice)
}

This works only if the input is either string or integer but not both How to make sure this function works for array interface

There is no array interface. And arrays of interface like `[]interface{}` likely don't work how you're thinking here. An `[]int` is not assignable to `[]interface{}`, nor is `[]string`. — Hymns For Disco, Dec 07 '20 at 23:21
@HymnsForDisco I have a use case something similar to this with a small twist https://stackoverflow.com/questions/64234626/how-to-merge-a-json-array-of-json-objects-to-a-single-json-object/64235350#64235350 This returns the merge array but we need to make sure the merge array is filtering duplicates. Is there a way that can be handled? — Rahul, Dec 07 '20 at 23:45
"How to make sure this function works for array interface[?]" You don't. In Go you write several functions, only the ones needed. Note that seemingly valid code using reflection might break if stuff like floatingpoint NaNs have to to be processed or types with no equality. — Volker, Dec 08 '20 at 06:39
on top of other answers, you might want to use a third party library https://github.com/thoas/go-funk#funkuniq — , Dec 08 '20 at 08:58

score 2 · Answer 1 · answered Dec 08 '20 at 01:09

Use the reflect package to write a function that works with any slice type:

func unique(src interface{}) interface{} {
    srcv := reflect.ValueOf(src)
    dstv := reflect.MakeSlice(srcv.Type(), 0, 0)
    visited := make(map[interface{}]struct{})
    for i := 0; i < srcv.Len(); i++ {
        elemv := srcv.Index(i)
        if _, ok := visited[elemv.Interface()]; ok {
            continue
        }
        visited[elemv.Interface()] = struct{}{}
        dstv = reflect.Append(dstv, elemv)
    }
    return dstv.Interface()
}

Use it like this:

uniqueIntSlice := unique(intSlice).([]int)

Run the code on the Go Playground.

score 1 · Answer 2 · answered Dec 08 '20 at 08:47

How to make sure this function works for a (unsorted) slice of empty interface{}

Considered that empty interface{} are comparable (https://stackoverflow.com/a/54003329/4466350)

Thus, to answer your question, it is very simple to rewrite your original code

package main

import (
    "fmt"
)

func main() {
    intSlice := []interface{}{1, 5, 3, 6, 9, 9, 4, 2, 3, 1, 5}
    fmt.Println(unique(intSlice))
}

func unique(src []interface{}) []interface{} {
    keys := make(map[interface{}]bool)
    list := []interface{}{}
    for _, entry := range src {
        if _, value := keys[entry]; !value {
            keys[entry] = true
            list = append(list, entry)
        }
    }
    return list
}

https://play.golang.org/p/vW7vgwz9yc1

If your question become, how to remove duplicates of any slice type, please check this other answer https://stackoverflow.com/a/65191679/4466350

score -1 · Answer 3 · edited Dec 08 '20 at 01:00

Nothing elegant and very prone to errors, but you can us a function that receives two interface{} arguments, the first one is the slice to filter and the second is a pointer to the filtered slice, obviously if the first parameter is a slice of int, the second one MUST be s pointer to slice of int.

Inside the function you can check for the types of the parameters and treat them separately.

package main

import (
    "fmt"
)

func unique(slice interface{}, filtered interface{}) error {

    // Check for slice of string
    if sliceOfString, ok := slice.([]string); ok {

        // If slice is slice of string filtered MUST also be slice of string
        filteredAsSliceOfString, ok := filtered.(*[]string)
        if  !ok {
            return fmt.Errorf("filtered should be of type %T, got %T instead", &[]string{}, filtered)
        }

        keys := make(map[string]bool)

        for _, entry := range sliceOfString {
            if _, value := keys[entry]; !value {
                keys[entry] = true
                *filteredAsSliceOfString = append(*filteredAsSliceOfString, entry)
            }
        }

    }else if sliceOfInt, ok := slice.([]int); ok {

        // If slice is slice of int filtered MUST also be slice of int
        filteredAsInt, ok := filtered.(*[]int)
        if !ok {
            return fmt.Errorf("filtered should be of type %T, got %T instead", &[]string{}, filtered)
        }

        keys := make(map[int]bool)

        for _, entry := range sliceOfInt {
            if _, value := keys[entry]; !value {
                keys[entry] = true
                *filteredAsInt = append(*filteredAsInt, entry)
            }
        }

    } else {
        return fmt.Errorf("only slice of in or slice of string is supported")
    }

    return nil

}

func main() {
    intSlice := []int{1,5,3,6,9,9,4,2,3,1,5}
    intSliceFiltered := make([]int, 0)
    stringSlice := []string{"a", "b", "b", "c", "c", "c", "d"}
    stringSliceFiltered := make([]string, 0)

    fmt.Println(intSlice)
    err := unique(intSlice, &intSliceFiltered) // Very important to send pointer in second parameter
    if err != nil {
        fmt.Printf("error filtering int slice: %v\n", err)
    }
    fmt.Println(intSliceFiltered)

    fmt.Println(stringSlice)
    err = unique(stringSlice, &stringSliceFiltered) // Very important to send pointer in second parameter
    if err != nil {
        fmt.Printf("error filtering string slice: %v\n", err)
    }
    fmt.Println(stringSliceFiltered)
}

As I said, it's not elegant. I haven't check this for errors.

Here it is running.

Use a [type switch](https://golang.org/ref/spec#Type_switches) to simplify the code a little. https://play.golang.org/p/Mb9_FE6TbAH — , Dec 08 '20 at 01:26

How to remove duplicates in an interface array

3 Answers3

How to make sure this function works for a (unsorted) slice of empty interface{}

Related