Idiomatic efficient Haskell append?

Question

List and the cons operator (:) are very common in Haskell. Cons is our friend. But sometimes I want to add to the end of a list instead.

xs `append` x = xs ++ [x]

This, sadly, is not an efficient way to implement it.

I wrote up Pascal's triangle in Haskell, but I had to use the ++ [x] anti-idiom:

ptri = [1] : mkptri ptri
mkptri (row:rows) = newRow : mkptri rows
    where newRow = zipWith (+) row (0:row) ++ [1]

imho, this is a lovely readable Pascal's triangle and all, but the anti-idiom irks me. Can someone explain to me (and, ideally, point me to a good tutorial) on what the idiomatic data structure is for cases where you want to append to the end efficiently? I'm hoping for near-list-like beauty in this data structure and its methods. Or, alternately, explain to me why this anti-idiom is actually not that bad for this case (if you believe such to be the case).

---

[edit] The answer I like the best is Data.Sequence, which does indeed have "near-list-like beauty." Not sure how I feel about the required strictness of operations. Further suggestions and different ideas are always welcome.

import Data.Sequence ((|>), (<|), zipWith, singleton)
import Prelude hiding (zipWith)

ptri = singleton 1 : mkptri ptri

mkptri (seq:seqs) = newRow : mkptri seqs
    where newRow = zipWith (+) seq (0 <| seq) |> 1

Now we just need List to be a class, so that other structures can use its methods like zipWith without hiding it from Prelude, or qualifying it. :P

Answer 1

Keep in mind that what looks poor asymptotics might actually not be, because you are working in a lazy language. In a strict language, appending to the end of a linked list in this way would always be O(n). In a lazy language, it's O(n) only if you actually traverse to the end of the list,in which case you would have spent O(n) effort anyway. So in many cases, laziness saves you.

This isn't a guarantee... for example, k appends followed by a traversal will still run in O(nk) where it could have been O(n+k). But it does change the picture somewhat. Thinking about performance of single operations in terms of their asymptotic complexity when the result is immediately forced doesn't always give you the right answer in the end.

Answer 2

Standard Sequence has O(1) for addition from 'both ends' and O(log(min(n1,n2))) for general concatenation:

http://hackage.haskell.org/packages/archive/containers/latest/doc/html/Data-Sequence.html

The difference from lists though is that Sequence is strict

Answer 3

Something like this explicit recursion avoids your append "anti-idiom". Although, I don't think it is as clear as your example.

ptri = []:mkptri ptri
mkptri (xs:ys) = pZip xs (0:xs) : mkptri ys
    where pZip (x:xs) (y:ys) = x+y : pZip xs ys
          pZip [] _ = [1]

Answer 4

In your code for Pascal's Triangle, ++ [x] is not actually a problem. Since you have to produce a new list on the left hand side of ++ anyway, your algorithm is inherently quadratic; you cannot make it asymptotically faster merely by avoiding ++.

Also, in this particular case, when you compile -O2, GHC's list fusion rules (should) eliminate the copy of the list that ++ would normally create. This is because zipWith is a good producer and ++ is a good consumer in it's first argument. You can read about these optimizations in GHC User's Guide.

Answer 5

Depending on your use case, the ShowS method (appending via function composition) might be useful.

Answer 6

If you just want cheap append (concat) and snoc (cons at the right) a Hughes list, also called DList on Hackage, is the simplest to implement. If you want to know how they work, look at Andy Gill and Graham Hutton's first Worker Wrapper paper, John Hughes's original paper doesn't seem to be online. As others have said above ShowS is a String specialized Hughes list / DList.

A JoinList is a bit more work to implement. This is a binary tree but with a list API - concat and snoc are cheap and you can reasonably fmap it: the DList on Hackage has a functor instance but I contend it shouldn't have - the functor instance has to metamorph in and out of a regular list. If you want a JoinList then you'll need to roll your own - the one on Hackage is mine and it's not efficient, nor well written.

Data.Sequence has efficient cons and snoc, and is good for other operations - takes, drops etc. that a JoinList is slow for. Because the internal finger tree implementation of Data.Sequence has to balance the tree, append is more work than its JoinList equivalent. In practice because Data.Sequence is better written, I'd expect it still out-performs my JoinList for append.

Answer 7

another way would to avoid concatenation at all by just using infinite lists:

ptri = zipWith take [0,1..] ptri'
  where ptri' = iterate stepRow $ repeat 0
        stepRow row = 1 : zipWith (+) row (tail row)

Answer 8

I wouldn't necessarily call your code "anti-idomatic". Oftentimes, clearer is better, even if that means to sacrifice a few clock cycles.

And in your particular case, the append at the end doesn't actually change the big-O time complexity! Evaluating the expression

zipWith (+) xs (0:xs) ++ [1]

will take time proportional length xs and no fancy sequence data structure is going to change that. If anything, only the constant factor will be affected.

Answer 9

Chris Okasaki has a design for a queue that addresses this issue. See page 15 of his thesis http://www.cs.cmu.edu/~rwh/theses/okasaki.pdf

You may need to adapt code slightly, but some use of reverse and keeping two pieces of the list lets you work more efficiently on average.

Also, someone put up some list code in the monad reader with efficient operations. I admit, I didn't really follow it, but I thought I could figure it out if I concentrated. It turns out it was Douglas M. Auclair in Monad Reader issue 17 http://themonadreader.files.wordpress.com/2011/01/issue17.pdf

---

I realized the above answer does not directly address the question. So, for giggles, here is my recursive answer. Feel free to tear it apart -- it is not pretty.

import Data.List 

ptri = [1] : mkptri ptri

mkptri :: [[Int]] -> [[Int]]
mkptri (xs:ys) =  mkptri' xs : mkptri ys

mkptri' :: [Int] -> [Int]
mkptri' xs = 1 : mkptri'' xs

mkptri'' :: [Int] -> [Int]
mkptri'' [x]        = [x]
mkptri'' (x:y:rest) = (x + y):mkptri'' (y:rest)

Answer 10

I wrote an example of @geekosaur's ShowS approach. You can see many examples of ShowS in the prelude.

ptri = []:mkptri ptri
mkptri (xs:ys) = (newRow xs []) : mkptri ys

newRow :: [Int] -> [Int] -> [Int]
newRow xs = listS (zipWith (+) xs (0:xs)) . (1:)

listS :: [a] -> [a] -> [a]
listS [] = id
listS (x:xs) = (x:) . listS xs

---

[edit] As @Dan's idea, I rewrote newRow with zipWithS.

newRow :: [Int] -> [Int] -> [Int]
newRow xs = zipWithS (+) xs (0:xs) . (1:)

zipWithS :: (a -> b -> c) -> [a] -> [b] -> [c] -> [c]
zipWithS z (a:as) (b:bs) xs =  z a b : zipWithS z as bs xs
zipWithS _ _ _ xs = xs

Answer 11

If you're looking for a general purpose solution, then how about this:

mapOnto :: [b] -> (a -> b) -> [a] -> [b]
mapOnto bs f = foldr ((:).f) bs

This gives a simple alternate definition for map:

map = mapOnto []

We can a similar definition for other foldr-based functions, like zipWith:

zipOntoWith :: [c] -> (a -> b -> c) -> [a] -> [b] -> [c]
zipOntoWith cs f = foldr step (const cs)
  where step x g [] = cs
        step x g (y:ys) = f x y : g ys

Again deriving zipWith and zip fairly easily:

zipWith = zipOntoWith []
zip = zipWith (\a b -> (a,b))

Now if we use these general purpose functions, your implementation becomes pretty easy:

ptri :: (Num a) => [[a]]
ptri = [] : map mkptri ptri
  where mkptri xs = zipOntoWith [1] (+) xs (0:xs)

Answer 12

You can represent a list as a function to build a list from []

list1, list2 :: [Integer] -> [Integer]
list1 = \xs -> 1 : 2 : 3 : xs
list2 = \xs -> 4 : 5 : 6 : xs

Then you can easily append lists and add to either end.

list1 . list2 $ [] -> [1,2,3,4,5,6]
list2 . list1 $ [] -> [4,5,6,1,2,3]
(7:) . list1 . (8:) . list2 $ [9] -> [7,1,2,3,8,4,5,6,9]

You can rewrite zipWith to return these partial lists:

zipWith' _ [] _ = id
zipWith' _ _ [] = id
zipWith' f (x:xs) (y:ys) = (f x y :) . zipWith' f xs ys

And now you can write ptri as:

ptri = [] : mkptri ptri
mkptri (xs:yss) = newRow : mkptri yss
    where newRow = zipWith' (+) xs (0:xs) [1]

Taking it further, here's a one-liner that's more symmetrical:

ptri = ([] : ) . map ($ []) . iterate (\x -> zipWith' (+) (x [0]) (0 : x [])) $ (1:)

Or this is simpler yet:

ptri = [] : iterate (\x -> 1 : zipWith' (+) (tail x) x [1]) [1]

Or without zipWith' (mapAccumR is in Data.List):

ptri = [] : iterate (uncurry (:) . mapAccumR (\x x' -> (x', x+x')) 0) [1]