How to observe progress while consuming a lazy sequence?

Question

I'd like to know if the following way of "observing" a sequence while consuming it is correct. I've read the following SO answer(s) but I'm a bit surprised in that I've read several times that the "correct" way to do so was to use laziness (and hence explicitely make your sequence lazy even if it wasn't), yet the word "lazy" isn't even mentioned once there:

How to implement the Observer Design Pattern in a pure functional way?

So basically I have a lazy sequence and I want to keep that lazy sequence 100% pure and I don't want the observer to leak anything into that lazy sequence. In my example I'm simply using (range 100000) but any lazy sequence would do.

Then I'm using some mutability (in Clojure using an atom) doing something like this (the code is runnable, you can copy/paste as is in a REPL):

(let [a (atom (range 100000))]
  (loop [i 0]
    (if (= 0 (mod i 10)) (println "i: " i)) ; the observer
    (if (= 100 i)
      (take 1 @a)
      (do (reset! a (rest @a)) (recur (inc i))))))

The point is not that I'm using a mutable atom here but that the implementation of the lazy sequence doesn't know at all that it is being observed. The observer can obviously be fancier: like actually notifying observers instead of using side-effects to print i (once again: printing i here is just an example).

Is this a "correct" way of observing while consuming a lazy sequence?

If this is not correct, how would you observe while consuming a lazy sequence in Clojure?

Alternatively, how would you do it in Haskell?

Given that clojure sequences are immutable, and the observer pattern is intended to observe change (wich won't happen to the sequence), what exactly is it that you want to observe? — Joost Diepenmaat, Jul 06 '14 at 16:05
@JoostDiepenmaat: as I wrote in the question: how "far" I'm in while consuming the sequence. For example say I want to sum the first the first one billion prime numbers and display a progress bar saying how far I'm in during the computation (and updating the progress bar, for example, every 1 million prime). But I don't want to modify my lazy-sequence generating prime numbers because the concept of "observing" is orthogonal to the generation of prime numbers. I just made that example up but it's something that I need to do once in a while to "see" what's going on. I hope it makes sense. — Cedric Martin, Jul 06 '14 at 16:20
The lazy sequence doesn't even need to be finite: even for infinite sequence it may make sense to update the UI as to what's happening. Say you're looking for a prime number with some very specific properties by analyzing an (infinite) lazy-sequence of prime numbers until you find the one matching your criteria: it may be convenient to update, once every *x* primes tested, the UI, by displaying: *"Tried xxx primes so far"*. — Cedric Martin, Jul 06 '14 at 16:24

A. Webb · Answer 1 · 2014-07-06T19:30:53.167

If you merely want to intersperse a side effect during the consumption, then yes, the sensible thing to do in Clojure would be to wrap in another lazy-sequence.

(defn lazy-report
  "Wrap s in a lazy sequence that will call f, presumably for side effects, 
   on every nth element of s during consumption."
  [s f n] 
  (let [g (cons (comp first (juxt identity f)) (repeat (dec n) identity))]
    (map #(% %2) (rest (cycle g)) s)))

(println "sum:" (reduce + (lazy-report (range 1 1000) #(println "at:" %) 100)))
;=> at: 100
;   at: 200
; ...
;   at: 900
;   sum: 499500

cdk · Answer 2 · 2014-07-06T20:38:14.750

Here's how I would do this in Haskell. What you're asking for is a lazy data structure that occasionally produces some value. This can be modeled as a stream with skipping.

data Stream a
    = Done 
    | Skip    (Stream a)
    | Yield a (Stream a)

A Stream can either Yield a value and the rest of the stream, Skip and return the rest of the stream, or it is Done which means the stream has been fully consumed.

We can evaluate a Stream that produces side-effects by simply recursing through it, executing the side-effects in Yield, until we reach the end of the Stream.

eval :: Monad m => Stream (m a) -> m ()
eval Done         = return ()
eval (Yield ma s) = ma >> eval s
eval (Skip     s) = eval s

We can build a Stream from a lazy sequence (a list in this case) given a "decision" function that chooses whether to produce a side-effect or skip for each element in the original sequence.

observe :: Monad m => (a -> Maybe (m b)) -> [a] -> m ()
observe f = eval . go
    where go []     = Done
          go (x:xs) = case f x of
              Nothing -> Skip     (go xs)
              Just mb -> Yield mb (go xs)

If we wanted to observe other types of sequences (including lists), we can generalize observe to work for any Foldable instance with this nicely succinct definition:

observe :: (Foldable f, Monad m) => (a -> Maybe (m b)) -> f a -> m ()
observe f = eval . foldr (maybe Skip Yield . f) Done

The final product,

f :: Int -> Maybe (IO ())
f x | x `rem` 10 == 0 = Just (print x)
    | otherwise       = Nothing

main = observe f [0..100]

This works for infinite sequences as well.

How to observe progress while consuming a lazy sequence?

2 Answers2