Best way to manage a pool of resources in Clojure

Question

I have a a web service endpoint that uses a mutable resource from a Java library. That web service endpoint can receive multiple queries at the same time. (the endpoint is implemented using Ring/Compojure). Creating these resources is costly, so re-creating them for every web service call is really inefficient.

What I want to do is to create a pool of that resource that I populate when the web service starts. Then each time the endpoint is called, it takes a resource from the pool, use it for its processing, and then push it back into the pool and wait for the next call to happen.

I am wondering what would be the best way to do that in Clojure? Is there a "pool" Clojure library that could help me with that?

I naively tried to implement that using an vector in an atom where each item of the vector is that resource. However, it quickly learned that it could not really work that way.

If you do not mind some heavy-duty Java interop, the [Apache Commons Pool](https://commons.apache.org/proper/commons-pool/) library is always there. — ez121sl, Jan 28 '16 at 02:12

score 3 · Accepted Answer · edited May 23 '17 at 12:33

This is based on Timothy Pratley's idea of using refs:

(def pool (ref ['a 'b 'c]))

(defn take' [pool]
  (dosync
    (let [[h & t] @pool]
      (ref-set pool (vec t))
      h)))

(defn put [pool x]
  (dosync
    (alter pool conj x)
    nil))

(take' pool)   ;; => 'a
(put pool 'a)  ;; => nil
(take' pool)   ;; => 'a
(take' pool)   ;; => 'b
(take' pool)   ;; => 'c

Maybe not the best way to attack this. But I like the simplicity of it.

score 2 · Answer 2 · answered Jan 28 '16 at 01:33

2

To implement a pool you need to address 2 concerns:

Concurrency. Use locking https://clojuredocs.org/clojure.core/locking or a ref instead of an atom. Requests can be simultaneous, so you need to be careful that it is impossible for two consumers to receive the same resource.
Releasing resources. Consider using a pattern like (with-open ...) i.e. a macro that expands to a try-finally where the resource is released back to the open pool when you leave the block scope.

You might want to designate an 'error' status as well as 'available' or 'in use', where the resource may need to be released and recreated.

answered Jan 28 '16 at 01:33

Timothy Pratley

10,586
3
34
63

Is it possible to use a ref and not include the actual work of the resource within the transaction, though? Because retrying on that would be undesirable. An example would be awesome. – muhuk Jan 28 '16 at 10:18
There can be no io (or side effects) done within a transaction. I was not recommending doing work in the transaction, but rather using the ref or locking to force consistency when accessing the list of resources, taking them, and releasing them. You should avoid locking anything while the resource is in use as that can potentially block for a long time. Sure I can add an example soon. – Timothy Pratley Jan 28 '16 at 15:04

score 2 · Answer 3 · answered Jan 28 '16 at 07:55

2

Have a look to this kul/pool. It uses Apache Commons Pool. I hope it's useful.

answered Jan 28 '16 at 07:55

icamts

186
1
8

Will keep that in mind, but I think I prefer the idiomatic way in clojure, thanks! – Neoasimov Jan 28 '16 at 14:38

score 0 · Answer 4 · answered Jan 12 '17 at 11:37

You can also use an atom:

(def pool (atom ['c 'b 'a]))

(defn take'
  [pool]
  (loop []
    (let [p @pool]
      (if (compare-and-set! pool p (pop p))
        (peek p)
        (recur)))))

(defn put
  [pool x]
  (swap! pool conj x)
  nil)

(take' pool)   ;; => 'a
(put pool 'a)  ;; => nil
(take' pool)   ;; => 'a
(take' pool)   ;; => 'b
(take' pool)   ;; => 'c

Best way to manage a pool of resources in Clojure

4 Answers4