Create Lazy Sequency by Concatenating Collections

Question

Create a lazy sequence by concatenating collections.

Consider the following function:

(defn req []
  (Thread/sleep 1000)
  (repeat 4 (rand-int 10)))

The sleep is added since the function will finally be a http request, thus it should emulate a delay.

Sample outputs:

(req)
;; (8 8 8 8)

(req)
;; (4 4 4 4)

I'm thinking of a function now, that creates a lazy sequence build by the concatenation of subsequent req results.

(take 10 (f req))
;; (3 3 3 3 2 2 2 2 9 9)

Here is one implementation:

(defn f [g]
  (lazy-seq (concat (g) (f g))))

Is this the way to go? I'm somehow guessing that there might be already an abstraction for this available.. I tried lazy-cat, but this macro seems to work only for a fixed number of given sequences.

It turns out that this is a working abstraction:

(take 10 (apply concat (repeatedly req)))

However it looks like chunking of lazy sequences causes req to be called more often than needed here, which would not be acceptable if it's an http request.

This is not working. It returns a list of lists instead of a list. It's omitting the concat part of it. — Anton Harald, Jan 12 '17 at 15:44
Sorry--small correction: (take 10 (mapcat (fn [_] (req)) (range))) — DarrylG, Jan 12 '17 at 16:18
@DarrylG this also leads to redundant realization of elements of lazy sequence — OlegTheCat, Jan 12 '17 at 16:19
@OlegTheCat: example output this provides is: (5 5 5 5 9 9 9 9 3 3) Isn't this the same as the problem description? — DarrylG, Jan 12 '17 at 16:30

OlegTheCat · Accepted Answer · 2017-01-12T21:24:47.733

The "unneeded" realizations of elements of lazy sequence is happening because apply needs to know the number of arguments that passed function is applied to.

Having a quick look at Clojure core lib, it seems, that it doesn't provide a function that concatenates a sequence of sequences and, at the same time, handles laziness in a way you want (doesn't redundantly realize the elements of passed lazy sequence), so, you'll need to implement it yourself.

Here's possible solution:

(defn apply-concat [xs]
  (lazy-seq
   (when-let [s (seq xs)]
     (concat (first s) (apply-concat (rest s))))))

And then:

user> (defn req []
        (println "--> making request")
        [1 2 3 4])
#'user/req
user> (dorun (take 4 (apply-concat (repeatedly req))))
--> making request
nil
user> (dorun (take 5 (apply-concat (repeatedly req))))
--> making request
--> making request
nil
user> (dorun (take 8 (apply-concat (repeatedly req))))
--> making request
--> making request
nil
user> (dorun (take 9 (apply-concat (repeatedly req))))
--> making request
--> making request
--> making request
nil

~~The only concern with this approach is danger of blowing the stack, since apply-concat is potentially infinitely recursive.~~

Update:

To be precise apply realizes (arity of passed function + 1) elements of passed lazy sequence:

user> (dorun (take 1 (apply (fn [& xs] xs) (repeatedly req))))
--> making request
--> making request
nil
user> (dorun (take 1 (apply (fn [x & xs] xs) (repeatedly req))))
--> making request
--> making request
--> making request
nil
user> (dorun (take 1 (apply (fn [x y & xs] xs) (repeatedly req))))
--> making request
--> making request
--> making request
--> making request
nil

There are no stack concerns here. The stack-trampolining effect of lazy seqs takes care of all the problems. You can see for yourself with, eg, `(last (apply-concat (repeat 1e6 [1])))`. — amalloy, Jan 12 '17 at 18:05
@OlegTheCat I'm still struggling with the question, if it's due to apply that the first 4 items are evaluated. To me it looks like `(apply concat (repeatedly req))` does take the first fn definition of apply (apply ([f args] ..)). Would you agree with that? — Anton Harald, Jan 12 '17 at 21:07
So I have doubts that apply causes this. Look at this: `(take 1 (apply (fn [& xs] xs) (repeatedly req)))` - it evaluates req only 2 times. — Anton Harald, Jan 12 '17 at 21:13
@AntonHarald The number of evaluated elements depends on arity function passed to apply (as I mentioned in my answer this is due to apply realizes as many elements as passed function accepts plus one). So `(take 1 (apply (fn [& xs] xs) (repeatedly req)))` realized `req` two times, `(take 1 (apply (fn [x & xs] (cons x xs)) (repeatedly req)))` - three times, `(take 1 (apply (fn [x y & xs] (cons x (cons y xs))) (repeatedly req)))` - four times, and so on. — OlegTheCat, Jan 12 '17 at 21:20
Alright, I see now. So it's concats [x y & zs] plus one that causes the 4 initial evals. Too bad actually, I'd say without this, there would be quite some situations more, where the power of lazy sequences could be used. — Anton Harald, Jan 12 '17 at 21:32

birdspider · Answer 2 · 2017-01-12T17:31:15.693

2

how about

(take 14 
  (mapcat identity (repeatedly req)))

explanation:

(defn req [] 
  (print ".") 
  (repeat 4 (rand-int 10)))

(def x 
  (take 80 (mapcat identity (repeatedly req))))
; prints .... = 4x ; this is probably some repl eagerness

; to take 80 items, 20 realizatons (of 4 items) are requrend 
(def y 
  (doall
    (take 80 (mapcat identity (repeatedly req))))) 
 ; prints ..................... = 21x

EDIT: about those 4 early realizations:

I think this is due apply, which us used by mapcat. It realizes up to 4 args [^clojure.lang.IFn f a b c d & args] given multiple ones.

edited Jan 12 '17 at 17:31

answered Jan 12 '17 at 16:37

birdspider

3,034
1
16
25

as I said, its either a repl-eagerness or a clojure seq chunking - (clojure used to realize by 1 and this behaviour got changed to realize the next 32 items in clojure.1.1 I think), I'm investigating the exact cause right now. – birdspider Jan 12 '17 at 16:55
@OlegTheCat, I'm pretty sure this is due `apply`, which us used by `mapcat`. It realizes up to 4 args `[^clojure.lang.IFn f a b c d & args]` given multiple ones - but I might be wrong. So yeah `take 0-16` will always at least realize 4 chunks up front, whereas `take 17` will realize 4 times up front but only after 16 items again – birdspider Jan 12 '17 at 17:27

Create Lazy Sequency by Concatenating Collections

2 Answers2