30

Given a list such as

(list "foo" "bar" nil "moo" "bar" "moo" nil "affe")

how would I build a new list with the duplicate strings removed, as well as the nils stripped, i.e.

(list "foo" "bar" "moo" "affe")

The order of the elements needs to be preserved - the first occurence of a string may not be removed.

The lists I'm dealing with here are short, so there's no need to use anything like a hash table for the uniqueness check, although doing so certainly wouldn't hurt either. However, using cl functionality is not a viable option.

rafl
  • 11,980
  • 2
  • 55
  • 77

5 Answers5

46

Try "Sets and Lists" in the "Lists" section of the Emacs Lisp Reference Manual:

(delq nil (delete-dups (list "foo" "bar" nil "moo" "bar" "moo" nil "affe")))
aculich
  • 14,545
  • 9
  • 64
  • 71
scottfrazer
  • 17,079
  • 4
  • 51
  • 49
  • 2
    Much more elegant and straight-forward. Somtimes i really wish there was something like namespaces grouping related functions together, or maybe a more consistent naming scheme that'd allow looking for something in particular by guessing it's name :-/ – rafl Sep 28 '10 at 23:25
  • 1
    Grouping (via naming conventions or namespaces) is hard, because there are always many ways to group. Emacs doesn't try very hard to do it, and instead provides things like `apropos`. But that's not ideal either. We should probably try harder, tho it's a bit late to fix most offenders. – Stefan Sep 09 '13 at 00:23
20

The Common Lisp package contains many list manipulation functions, in particular remove-duplicates.

(require 'cl)
(remove-duplicates (list "foo" "bar" nil "moo" "bar" "moo" nil "affe")
                   :test (lambda (x y) (or (null y) (equal x y)))
                   :from-end t)

Yes, I realize you said you didn't want to use cl. But I'm still mentioning this as the right way to do it for other people who might read this thread.

(Why is cl not viable for you anyway? It's been shipped with Emacs for about 20 years now, not counting less featured past incarnations.)

Gilles 'SO- stop being evil'
  • 104,111
  • 38
  • 209
  • 254
  • 1
    I've been doing a couple of tiny patches to Gnus, and in its commit log I constantly see changes replacing something from cl with a non-cl equivalent. I'm not quite sure what the reasons are exactly, but it might very well be something between supporting weird Emacs flavours or versions, or trying to avoid loading the `cl` package unless really necessary. – rafl Sep 28 '10 at 21:50
  • 3
    @rafl I believe the restriction against `cl` started from RMS' desire to keep emacs-lisp small. Check out a recent thread discussing this point: http://lists.gnu.org/archive/html/emacs-devel/2010-09/msg01278.html – Trey Jackson Sep 28 '10 at 23:16
  • 8
    @Trey: oh, I see. In order to keep the core of Emacs small, require every package to reimplement their own basic data structures functions. Well, if you're the one who decides that the core of Emacs will contain exactly the features you use, it works. Everyone else gets duplicated effort and bloat... – Gilles 'SO- stop being evil' Sep 29 '10 at 00:32
6

If you use dash.el library, that's all you need:

(-distinct (-non-nil '(1 1 nil 2 2 nil 3)) ; => (1 2 3)

dash.el is written by Magnar Sveen and it's a great list manipulation library with many functions for all kinds of tasks. I recommend to install it if you write lots of Elisp code. Function -distinct removes duplicate elements in a list, -non-nil removes nil elements. While the above code is sufficient, below I describe an alternative approache, so feel free to ignore the rest of the post.

-non-nil was added in version 2.9, so if for some reason you have to use earlier versions, another way to achieve the same is to use -keep with built-in identity function, which just returns whatever it is given: (identity 1) ; => 1. The idea is that -keep keeps only elements, for which the predicate returns true (“non-nil” in Lisp jargon). identity obviously returns non-nil only for whatever values that are not nil:

(-distinct (-keep 'identity '(1 1 nil 2 2 nil 3)) ; => (1 2 3)
Mirzhan Irkegulov
  • 17,660
  • 12
  • 105
  • 166
2

This is a short example:

(delete-duplicates '("~/.emacs.d" "~/.emacs.d") :test #'string-equal) ;; '("~/emacs.d")

Basically you use the :test keyword to select the function string-equal to test if the elements are duplicated.

Else the default function test doesn't check string equality.

kcurtet
  • 46
  • 3
0

Here ya go:

(defun strip-duplicates (list)
  (let ((new-list nil))
    (while list
      (when (and (car list) (not (member (car list) new-list)))
        (setq new-list (cons (car list) new-list)))
      (setq list (cdr list)))
    (nreverse new-list)))
Sean
  • 29,130
  • 4
  • 80
  • 105
  • This does the trick quite nicely. I've learned about `member` now. Is there also a similar function that allows specifying a comparison function? – rafl Sep 28 '10 at 18:31
  • No, not that I know of. It would be pretty trivial to write, though. – Sean Sep 28 '10 at 18:51
  • There's also `memq` and `memql`. – phils Sep 28 '10 at 21:39
  • 9
    A [better answer](http://stackoverflow.com/a/3817714/462302) comes from reading the documentation before reinventing common patterns that already have an implementation (in this case as `delete-dups`). – aculich Sep 08 '13 at 18:37
  • 1
    And if you're curious you can compare your implementation: [delete-dups in subr.el](http://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/subr.el?h=emacs-24#n376) – aculich Sep 08 '13 at 18:43