4

I have a database server and I fetch data from it. Sometimes data have millions rows and more so I use laziness for downloading. I use Server Side Cursors from clojure.jdbc library https://funcool.github.io/clojure.jdbc/latest/#cursor-queries to fetch data lazily.

Now I have a problem. I need produce initial 500 elements from a lazy-sequence, then the program must wait for 10 minutes to get a signal which report to the program produce next 500 elements and so on until i receive all data from server. But if a report didn`t arrive for 10 minutes, the program must close a connection.

I wrote sample:

(def lazyseq_maps (atom {:seq_1 {:next_500 false :data nil} :seq_2 {:next_500 false :data nil}})) ; here is a collection of all unfinished lazy sequences waiting for signal to continue produce elements

(jdbc/atomic conn
 (with-open [cursor (jdbc/fetch-lazy conn sql]
   (let [lazyseq (jdbc/cursor->lazyseq cursor)]
     (swap! lazyseq_maps assoc seq_id {:next_500 true :data nil})
     (loop [lazyseq_rest lazyseq
            count 1]
            (if (:next_500 (seq_id @lazyseq_maps))
              (do
                (swap! lazyseq_maps update-in [seq_id :data] conj (first lazyseq_rest))
                (when (= 0 (mod count 500))
                  (swap! lazyseq_maps assoc-in [seq_id :next_500] false))
                (recur (rest lazyseq) (inc count)))
              ;
              (func-for-waiting-signal)))) ; here I don`t know how to create function waiting signal to continue fetching data
   (seq_id @lazyseq_maps)))

Can you help what clojure tools I should use to solve my problems? I assume I should use core.async to create channels for loops. Am I right? And how should I create function which stop executing loop for 10 minutes or continue executing if I receive appropriate signal?

Verbery
  • 157
  • 7
  • Ten minutes is really long to leave a connection open and unused, you might want to consider using something like keyset pagination instead. – Mark Rotteveel Nov 01 '17 at 08:09

1 Answers1

1

Indeed you should use core.async for this, they have timeout channel for the timeout purpose and with alt! you can "wait" for a value to pop out from any number of channels. The complete code will be a bit complex though. Output rows would be "pushed" to an output channel.

To my knowledge with-open and lazy sequences don't compose nicely, as the cursor would be closed prematurely. But I am not familiar with clojure.jdbc library.

NikoNyrh
  • 3,578
  • 2
  • 18
  • 32
  • Can you tell what you meant "with-open and lazy sequences don't compose nicely" I has opened question https://stackoverflow.com/questions/47323454/clojure-core-async-how-to-download-lazily-with-go-block-inside-with-open Is It about what you said? – Verbery Nov 16 '17 at 07:57