Equivalent to find_each in DataMapper

Question

Is there a way to do the equivalent of ActiveRecord#find_each in DataMapper ?

(find_each will iterate over the result of a query by fetching things in memory by batch of 1000 rather than loading everything in memory)

Answer 1

I checked dm-chunked_query as @MichaelKohl suggested, but I couldn't make it work as I'd expect, it gets the whole collection (I'd expect it to use OFFSET+LIMIT). So I wrote my own extension, it's pretty simple, hope it helps:

class DataMapper::Collection
  def batch(n)
    Enumerator.new do |y|
      offset = 0
      loop do 
        records = slice(offset, n)
        break if records.empty?
        records.each { |record| y.yield(record) }
        offset += records.size
      end
    end
  end
end

# Example
Model.all(:order => :id.asc).batch(1000).each { |obj| p obj }

Answer 2

I don't DM much, but it would not be that hard to write your own, assuming DM lets you apply your own 'limit'and 'offset' manually to queries.

Check out the implementation of find_each/find_in_batches in AR, only a couple dozen lines.

https://github.com/rails/rails/blob/master/activerecord/lib/active_record/relation/batches.rb#L19

https://github.com/rails/rails/blob/master/activerecord/lib/active_record/relation/batches.rb#L48