0

I am using Paperclip and DelayedJob, to process uploaded styles in the background. When I send an image into the queue to be processed, DelayedJob finishes the job, but it gets sent into an infinite loop creating more of the exact same job.

Versions:

delayed_job_active_record version 4.0.2.

paperclip version 4.2.0.

Here is some code plus the worker log to show what is going on...

has_attached_file :data,
  url: '/updates/:id/:style/:basename.:extension',
  styles: { large: '750x750>', medium: '300x300>', thumb: '50x50#' },
  convert_options: { thumb: '-quality 75 -strip' }

validates_attachment_size :data, less_than: 20.megabytes
validates_attachment_content_type :data, content_type: /\Aimage\/.*\Z/

before_data_post_process do |update|
  if !update.processing && update.data_changed?
    update.processing = true
    false # stop processing
  end
end

after_save do |update|
  if update.processing
    update.delay.regenerate_styles!
  end
end

def regenerate_styles!
  data.reprocess!
  update_attribute(:processing, false)
  save
end

def data_changed?
  data_file_size_changed? ||
  data_file_name_changed? ||
  data_content_type_changed? ||
  data_updated_at_changed?
end

Worker response

[Worker(host:Justins-MacBook-Pro.local pid:98244)] Starting job worker
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=52) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=52) COMPLETED after 2.5247
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=53) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=53) COMPLETED after 0.1654
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=54) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=54) COMPLETED after 0.0895
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=55) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=55) COMPLETED after 0.0993
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=56) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=56) COMPLETED after 0.0990
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=57) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=57) COMPLETED after 0.0947
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=58) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=58) COMPLETED after 0.0949
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=59) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=59) COMPLETED after 0.0855
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=60) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=60) COMPLETED after 0.1004
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=61) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=61) COMPLETED after 0.0941
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=62) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=62) COMPLETED after 0.0860
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=63) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=63) COMPLETED after 0.0980
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=64) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=64) COMPLETED after 0.0928
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=65) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=65) COMPLETED after 0.1053
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=66) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=66) COMPLETED after 0.7533
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=67) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=67) COMPLETED after 0.0950
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=68) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=68) COMPLETED after 0.0952
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=69) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=69) COMPLETED after 0.0867
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=70) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=70) COMPLETED after 0.0944
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=71) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=71) COMPLETED after 0.0913
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=72) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=72) COMPLETED after 0.0918
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=73) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=73) COMPLETED after 0.0982
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=74) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=74) COMPLETED after 0.0976
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=75) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=75) COMPLETED after 0.0925
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=76) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=76) COMPLETED after 0.0914
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=77) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=77) COMPLETED after 0.0970
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=78) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=78) COMPLETED after 0.0922
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=79) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=79) COMPLETED after 1.4655
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=80) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=80) COMPLETED after 0.0892
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=81) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=81) COMPLETED after 0.0839
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=82) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=82) COMPLETED after 0.0848
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=83) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=83) COMPLETED after 0.0965
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=84) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=84) COMPLETED after 0.0963
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=85) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=85) COMPLETED after 0.0917
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=86) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=86) COMPLETED after 0.0919
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=87) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=87) COMPLETED after 0.0909
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=88) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=88) COMPLETED after 0.1006
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=89) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=89) COMPLETED after 0.0936
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=90) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=90) COMPLETED after 0.2311
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=91) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=91) COMPLETED after 0.1069
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=92) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=92) COMPLETED after 0.6727
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=93) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=93) COMPLETED after 0.0904
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=94) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=94) COMPLETED after 0.0993
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=95) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=95) COMPLETED after 0.0903
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=96) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=96) COMPLETED after 0.0976
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=97) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=97) COMPLETED after 0.0916
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=98) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=98) COMPLETED after 0.1023
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=99) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=99) COMPLETED after 0.0991
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=100) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=100) COMPLETED after 0.0997
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=101) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=101) COMPLETED after 0.0953
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=102) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=102) COMPLETED after 0.0933
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=103) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=103) COMPLETED after 0.0918
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=104) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=104) COMPLETED after 0.6735
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=105) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=105) COMPLETED after 0.0884
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=106) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=106) COMPLETED after 0.0916
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=107) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=107) COMPLETED after 0.0859
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=108) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=108) COMPLETED after 0.0886
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=109) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=109) COMPLETED after 0.0871
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=110) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=110) COMPLETED after 0.0853
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=111) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=111) COMPLETED after 0.0991
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=112) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=112) COMPLETED after 0.0889
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=113) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=113) COMPLETED after 0.0969
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=114) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=114) COMPLETED after 0.0956
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=115) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=115) COMPLETED after 0.0993
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=116) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=116) COMPLETED after 0.0987
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=117) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=117) COMPLETED after 0.6886
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=118) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=118) COMPLETED after 0.0927
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=119) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=119) COMPLETED after 0.0962
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=120) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=120) COMPLETED after 0.0991
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=121) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=121) COMPLETED after 0.0924
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=122) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=122) COMPLETED after 0.0938
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=123) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=123) COMPLETED after 0.0965
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=124) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=124) COMPLETED after 0.0949
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=125) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=125) COMPLETED after 0.0933
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=126) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=126) COMPLETED after 0.0950
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=127) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=127) COMPLETED after 0.0877
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=128) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=128) COMPLETED after 0.0954
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=129) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=129) COMPLETED after 0.8408
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=130) RUNNING

It probably takes 10 seconds for DJ to run through all those jobs as well. Has anyone run into this issue or can anyone spot an error with my configuration? Let me know if any more code is needed. Thanks in advance.

Justin
  • 4,922
  • 2
  • 27
  • 69

1 Answers1

0

You call data.reprocess! with a bang, this probably saves the model (before you set processing to false) which then with the after_save generates another job.

Your data changed method also checks updated_at (like you commented below) calling any kind of save (also the "processing, false" save) will update the updated_at. (causing processing again to be turned on)

Check the second answer here: How to update a single attribute without touch updated_at attribute?

and use update_column to skip the updating of the updated_at column.

thus:

def regenerate_styles!
  update_column(:processing, false)
  data.reprocess!
end
Community
  • 1
  • 1
knutsel
  • 1,281
  • 9
  • 22
  • This still causes the loop actually, but I may have found the cause on [Paperclip's GitHub issues](https://github.com/thoughtbot/paperclip/issues/866). If you read the fourth comment, it says that when you check for the `updated_at` column for your attribute with an after_save callback, it results in the reprocessing of the attachment. – Justin Sep 12 '14 at 11:44
  • I have updated my answer, i did not see the updated_at loop in your original message, must have misread – knutsel Sep 12 '14 at 12:12