0

I'm writing a multi-threaded Ruby application that is generating and loading data via dm-sweatshop into a legacy MySQL database for use in a load test. Everything is working fine with one exception, I'm loading several million records, but I can't seem to increase the DB connection pool size to get things done faster. It seems to be stuck at 10 connections. I've tried increasing the thread pool size of the application itself, but watching MySQL, I can't seem to get more than 10 connections established.

I originally was just using URL for the DataMapper setup:

DataMapper.setup(:default, 'mysql://user:password@db-server/testing')

But I've moved to this to try to set a DB pool size:

DataMapper.setup(:default, {
  :adapter  => 'mysql',
  :pool     => 20,
  :host     => 'db-server',
  :database => 'testing',
  :username => 'user',
  :password => 'password'
})

How do I increase the pool size? I feel like I've poured over every RDoc and piece of documentation I can find on DataMapper, but I can't find how to do it. I would have switched to ActiveRecord to make this work at this point, but it doesn't support composite primary keys, which I need for this legacy DB.

For reference, I'm utilizing the 'thread/pool' gem for multi-threading and the app runs in Ruby 1.9.3. Here is how I'm using DataMapper in multiple threads:

pool = Thread::Pool.new(@config[:thread_pool_size])
...
10000.times { pool.process {Customer.gen} }
...
pool.shutdown
sleep 5
puts "DONE!"
jgnagy
  • 171
  • 2
  • Are you sure it's not the MySQL which is restricting connections? – morbusg Feb 04 '13 at 20:46
  • Anything is possible, but this is a fresh MySQL installation on my Mac, and I just checked the `max_connections` variable, which is still the default of 151. I have also verified that running multiple instances of my Ruby script creates multiples of 10 connections to the same MySQL server. I think it is safe to say that it is DataMapper or the MySQL adapter. – jgnagy Feb 05 '13 at 22:43
  • Ever figure this out? – Nick Aug 23 '13 at 16:45
  • Nope... moved on. Just accepted that I needed to run more instances of my tool (and had to adjust my data generation to adjust appropriately to avoid collisions). – jgnagy Oct 22 '13 at 19:25

1 Answers1

1

Considering that this questions is very old and that DataMapper is no longer maintained it is kind of obsolete, but for the sake of completeness: DataMapper is using a hard-coded pool-size of 8 and is ignoring your configuration.

See also: Datamapper & connection pool logic

tsauerwein
  • 5,841
  • 3
  • 36
  • 49