0

I have a database with a few million entities needing a friendly_id slug. Is there a way to speed up the process of saving the entities? find_each(&:save) is very slow. At 6-10 per second I'm looking at over a week of running this 24/7.

I'm just wondering if there is a method within friendly_id or parallel processing trick that can speed this process up drastically.

Currently I'm running about 10 consoles, and within each console starting the value +100k:

Model.where(slug: nil).find_each(start: value) do |e|
  puts e.id
  e.save
end

EDIT

Well one of the biggest things that was causing the updates to go so insanely slow is the initial find query of the entity, and not the actual saving of the record. I put the site live the other day, and looking at server database requests continually hitting 2000ms and the culprit was @entity = Entity.find(params[:id]) causing the most problems with 5+ million records. I didn't realize there was no index on the slug column and active record is doing its SELECT statements on the slug column. After indexing properly, I get 20ms response times and running the above query went from 1-2 entities per second to 1k per second. Doing multiple of them got the job down quick enough for the one time operation.

James Chevalier
  • 10,604
  • 5
  • 48
  • 74
viotech
  • 105
  • 1
  • 10
  • 1
    Why do you need store slug in DB? You could find item by id and generate slug dynamically. – Alexander Shlenchack Sep 20 '14 at 23:08
  • Minor improvement: Do sent every id to IO, perhaps only every 1000th. Because IO to the console is slow. Possible huge improvement: Generate the slug in pure SQL, depends on your slug format and your data whether this is possible. Furthermore: Do you have an index in the `slug` column? – spickermann Sep 20 '14 at 23:23
  • I wish I would have come back and read your answer here a few days ago added an index to slug column yesterday and it made everything blazing quick =) Not sure how that one slipped by me. Thanks much! – viotech Sep 25 '14 at 19:08

1 Answers1

2

I think the fastest way to do this would be to go straight to the database, rather than using Active Record. If you have a GUI like Sequel Pro, connect to your database (the details are in your database.yml) and execute a query there. If you're comfortable on the command line you can run it straight in the database console window. Ruby and Active Record will just slow you down for something like this.

To update all the slugs of a hypothetical table called "users" where the slug will be a concatenation of their first name and last name you could do something like this in MySQL:

UPDATE users SET slug = CONCAT(first_name, "-", last_name) WHERE slug IS NULL
Danielspector
  • 58
  • 5
  • Your answer is valid to my question of a much faster way of updating slugs. I managed to find what caused such a massive slowdown in saving slugs yesterday that helped me finish my task. Gonna mark your answer as the right one since it can help others who wish to bypass active record and friendly_id callbacks to update their slugs. – viotech Sep 25 '14 at 19:02