How lazy sequence run in this code

Question

Code is here:

(def fib-seq (lazy-cat [0 1]  (map + (rest fib-seq) fib-seq )))

As I could understand that the fib-seq is a lazy sequence generator that generate a serial of fibonacci number.
By take a look at (take 5 fib-seq) I will get fibonacci number as below:
(0 1 1 2 3)

But I can not figure out how the lazy sequence is generated when need, so I add some side effect on it.

(def fib-seq (lazy-cat [0 1] (map + 
    (do (println "R") (rest fib-seq)) 
    (do (println "B") fib-seq))))

By adding the println I expect it to print out R and B whenever lazy sequence try to generate new entry when it needed, but unfortunately this turn out to be like this.

user=> (take 5 fib-seq) ; this is the first time I take 5 elements
(0 R
B
1 1 2 3)

The output above look already weird as it do not print R and B element by element, but let's take a look at next step.

After take elements for the first time:

user=> (take 20 fib-seq)
(0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181)

I will never receive the R and B anymore, which makes me puzzle as it conflicted with my comprehension to lazy sequence's generate.

Could any explain it step by step to me?
BTW, is there any possibility to have a debug utility to debug it step by step just like Java and C?

Answer 1

Ok, step by step:

1. Here's the source code for lazy-cat (link):

   (defmacro lazy-cat
     "Expands to code which yields a lazy sequence of the concatenation
     of the supplied colls.  Each coll expr is not evaluated until it is
     needed. 
   
     (lazy-cat xs ys zs) === (concat (lazy-seq xs) (lazy-seq ys) (lazy-seq zs))"
     {:added "1.0"}
     [& colls]
     `(concat ~@(map #(list `lazy-seq %) colls)))
   

   So your code:

   (def fib-seq (lazy-cat [0 1] (map + 
       (do (println "R") (rest fib-seq)) 
       (do (println "B") fib-seq))))
   

   Expands to this:

   (def fib-seq (concat (lazy-seq [0 1])
                        (lazy-seq (map +
                                       (do (println "R") (rest fib-seq)) 
                                       (do (println "B") fib-seq)))))
   

   concat itself returns a lazy sequence which means that the body of the concat form is not evaluated until you traverse fib-seq.

2. When you traverse fib-seq for the first time (when you take the first 5 elements), the body of the concat form is first evaluated:

   (concat (lazy-seq [0 1])
           (lazy-seq (map +
                          (do (println "R") (rest fib-seq))
                          (do (println "B") fib-seq))))
   

   The first two elements of the lazy sequence returned by concat are taken from (lazy-seq [0 1]), which in turn takes them from [0 1] ; after this point, [0 1] is exhausted so so is (lazy-seq [0 1]) and the next elements of the concat sequence are taken from the (lazy-seq (map ...)) subsequence.

   Here the do special forms get evaluated, both of them, and you see R and B printed. The semantics of do is to evaluate all the forms within it, then to return the result of the last one.

   So (do (println "R") (rest fib-seq) prints R, then returns the result of (rest fib-seq), and (do (println "B") fib-seq)) prints B, then returns fib-seq.

   (map ...) returns a lazy sequence ; when the traversal reaches fib-seq's 3rd element, the first element of the map sequence is evaluated ; it is the sum of the first element of fib-seq (i.e. 0) and of the first of (rest fib-seq) i.e. the second of fib-seq (i.e. 1). Both are already evaluated at this point, so we don't end up with an infinite recursion.

   For the next elements, the laziness of map prevents an infinite recursion to occur, and the magic occurs.

3. On your second traversal of fib-seq (i.e. (take 20 fib-seq)), the first few elements of it have already been evaluated, so the do special forms are not re-evaluated and the traversal continues without side effects.

---

To have R and B printed whenever new elements are pulled from (rest fib-seq) and fib-seq, you'd have to do this:

(def fib-seq
  (lazy-cat [0 1]
            (map + 
                 (map #(do (println "R") %) (rest fib-seq)) 
                 (map #(do (println "B") %) fib-seq)))))

Answer 2

Thanks @omiel provide many useful information, but still did not touch the most sensitive point, after thinking a while, I figure out what happened in the generation of lazy sequence.
May clojure master in SO please correct me if I am actually wrong.

What I mean step by step actually foci on the generation of lazy-sequence item one by one, I already know some of the logic of clojure language.

As we know the fib-seq is defined as a lazy-seq and its first two items is 0 and 1, the rest of its items still left unevaluated, which is the most interesting feature of clojure.
While it is rather easy to understand that accessing the first two item just means to touch those two things, and they are in memory or cached, thus they could be directly return and print out.

As fib-seq do not have third item for now, it need to generate it when thread need to accessing the 3rd item, here is where my assumption start:

Since (map + (rest fib-seq) fib-seq ) is a lazy-seq itself, it contain no item in it currently and waiting for calling more command on it.
Here calling the 3rd item of fib-seq means calling the first item of lazy sequence (map...), hence it need to generate and real execute the code.
By simply replace variable name with list, the code of map seems like this:

(map + (rest [0 1 ..]) [0 1 ..] ); the '..' means it is a lazy sequence

then this code is become below after rest executed:

(map + [1 ..]  [0 1 ..] )
        ^       ^
        | ----- |
            |
            +
            1

As map generate lazy sequence, it is instructed to generate the first item of it, so by map these two lists, we got an item 1=(+ 1 0) which is the result of both the first item of these two lists add together.

Then the map stop generate item as it have no instruction to do so. Now after generate the new item 1 and concatenate it with [0 1], our fib-seq now look like this:

[0 1 1 ..]

Pretty good. Now let's touch the 4th item of fib-seq by (nth fib-seq 4).
fib-seq find it contain no item with index 4, but it found the third is cached so it will generate the 4th item from 3rd one.

Now thread move to (map ...) function and instruct map to hand out the second item of it.
map found it did not have No.2 item so it have to generate it. and replace fib-seq with real lazy seq:

(map + (rest [0 1 1..]) [0 1 1..] )

Then of course rest get the rest of seq:

(map + [1 1..] [0 1 1..] )

Here the most tricky thing happened I think.
Map add both the second rather than the first item of these lists:

(map + [1 1..] [0 1 1..] )
          ^       ^
          | ----- |
              |
              +
              2

So the map could return 2 as its 2nd item so as to complete the instruction.

The lazy-seq follow the same strategy in the follow item while instructed, and cache every generated item in memory for faster accessibility.

For this Fibonacci number generator, it just shift two list and add them one by one and recursively to generate required Fibonacci number like below:

0 1 1 2 3 5 ..   
1 1 2 3 5 ..

Which of course is a very deft way to generate Fibo.

summary

To sum up, from human's view, lazy seq will generate item always from its last status/position rather than starting from its initial state.

Please correct me if I am wrong, I am a newbie in clojure and I am eagerly to learn it well.

BTW

I want to write a netbeans language plugin for clojure integrate with leiningen as I think lighttable is useless, anyone have any suggestion?