Why is capistrano freezing in "deploy:cleanup"?

Question

I have no idea why, but capistrano (v2.15.5) is freezing sometimes when I try to deploy. There's no specific cause, and it's not always. But when it freezes, it freezes in the same task: "deploy:cleanup".

Check my "cap deploy" output in a frozen situation and my deploy.rb file: https://gist.github.com/anonymous/7632695 and shown below

deploy.rb

require "bundler/capistrano"

set :application, "my_app"
set :repository,  "git@bitbucket.org:fschuindt/weedmap.git"
set :user, "deploy"
set :use_sudo, "false"
default_run_options[:pty] = true

set :scm, :git
set :deploy_to, "/home/#{user}/#{application}"

role :web, "my_app.nl"
role :app, "my_app.nl"
role :db,  "my_app.nl", :primary => true

set :default_environment, {
  'PATH' => "/home/deploy/.rvm/gems/ruby-2.0.0-p247/bin:/home/deploy/.rvm/gems/ruby-2.0.0-p247@global/bin:/home/deploy/.rvm/rubies/ruby-2.0.0-p247/bin:/home/deploy/.rvm/gems/ruby-2.0.0-p247/bin:/home/deploy/.rvm/gems/ruby-2.0.0-p247@global/bin:/home/deploy/.rvm/rubies/ruby-2.0.0-p247/bin:/home/deploy/.rvm/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games",
  'RUBY_VERSION' => 'ruby-2.0.0-p247',
  'GEM_HOME' => '/home/deploy/.rvm/gems/ruby-2.0.0-p247',
  'GEM_PATH' => '/home/deploy/.rvm/gems/ruby-2.0.0-p247:/home/deploy/.rvm/gems/ruby-2.0.0-p247@global'
}

set :keep_releases, 3
after "deploy:update", "deploy:cleanup"

namespace :deploy do
  task :restart, :roles => :web do
    sudo "service nginx restart"
  end
end

load  'deploy/assets'

Output

wizard:weedmap fschuindt$ cap deploy
  * 2013-11-24 18:10:31 executing `deploy'
  * 2013-11-24 18:10:31 executing `deploy:update'
 ** transaction: start
  * 2013-11-24 18:10:31 executing `deploy:update_code'
    executing locally: "git ls-remote git@bitbucket.org:fschuindt/weedmap.git HEAD"
    command finished in 2319ms
  * executing "git clone -q git@bitbucket.org:fschuindt/weedmap.git /home/deploy/my_app/releases/20131124211034 && cd /home/deploy/my_app/releases/20131124211034 && git checkout -q -b deploy 25c78a55b60fd96c7039267b1ed862455c076b7f && (echo 25c78a55b60fd96c7039267b1ed862455c076b7f > /home/deploy/my_app/releases/20131124211034/REVISION)"
    servers: ["my_app.nl"]
Password: 
    [my_app.nl] executing command
    command finished in 8090ms
  * 2013-11-24 18:10:51 executing `deploy:finalize_update'
    triggering before callbacks for `deploy:finalize_update'
  * 2013-11-24 18:10:51 executing `bundle:install'
  * executing "cd /home/deploy/my_app/releases/20131124211034 && bundle install --gemfile /home/deploy/my_app/releases/20131124211034/Gemfile --path /home/deploy/my_app/shared/bundle --deployment --quiet --without development test"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 1738ms
  * 2013-11-24 18:10:53 executing `deploy:assets:symlink'
  * executing "rm -rf /home/deploy/my_app/releases/20131124211034/public/assets && mkdir -p /home/deploy/my_app/releases/20131124211034/public && mkdir -p /home/deploy/my_app/shared/assets && ln -s /home/deploy/my_app/shared/assets /home/deploy/my_app/releases/20131124211034/public/assets"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 920ms
  * executing "chmod -R -- g+w /home/deploy/my_app/releases/20131124211034 && rm -rf -- /home/deploy/my_app/releases/20131124211034/public/system && mkdir -p -- /home/deploy/my_app/releases/20131124211034/public/ && ln -s -- /home/deploy/my_app/shared/system /home/deploy/my_app/releases/20131124211034/public/system && rm -rf -- /home/deploy/my_app/releases/20131124211034/log && ln -s -- /home/deploy/my_app/shared/log /home/deploy/my_app/releases/20131124211034/log && rm -rf -- /home/deploy/my_app/releases/20131124211034/tmp/pids && mkdir -p -- /home/deploy/my_app/releases/20131124211034/tmp/ && ln -s -- /home/deploy/my_app/shared/pids /home/deploy/my_app/releases/20131124211034/tmp/pids"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 921ms
    triggering after callbacks for `deploy:update_code'
  * 2013-11-24 18:10:55 executing `deploy:assets:precompile'
    triggering before callbacks for `deploy:assets:precompile'
  * 2013-11-24 18:10:55 executing `deploy:assets:update_asset_mtimes'
  * executing "[ -e /home/deploy/my_app/shared/assets/manifest* ] && cat /home/deploy/my_app/shared/assets/manifest* || echo"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 1231ms
 ** Updating mtimes for ~288 assets...
    servers: ["my_app.nl"]
 ** scp upload #<StringIO:0x007f895b599c28> -> /home/deploy/my_app/TOUCH_ASSETS
[my_app.nl] /home/deploy/my_app/TOUCH_ASSETS
  * scp upload complete
  * executing "cat /home/deploy/my_app/TOUCH_ASSETS | while read asset; do touch -c -- \"$asset\"; done && rm -f -- /home/deploy/my_app/TOUCH_ASSETS"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 1330ms
  * executing "cd -- /home/deploy/my_app/releases/20131124211034 && RAILS_ENV=production RAILS_GROUPS=assets bundle exec rake assets:precompile"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 13721ms
  * executing "ls -1 /home/deploy/my_app/shared/assets/manifest* | wc -l"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 921ms
  * executing "ls /home/deploy/my_app/shared/assets/manifest*"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 920ms
  * executing "sudo -p 'sudo password: ' ls -x /home/deploy/my_app/releases"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 1535ms
  * executing "cp -- /home/deploy/my_app/shared/assets/manifest.yml /home/deploy/my_app/releases/20131124211034/assets_manifest.yml"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
    command finished in 920ms
  * 2013-11-24 18:11:19 executing `deploy:create_symlink'
  * executing "sudo -p 'sudo password: ' rm -f /home/deploy/my_app/current && sudo -p 'sudo password: ' ln -s /home/deploy/my_app/releases/20131124211034 /home/deploy/my_app/current"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
 ** [out :: my_app.nl] 
    command finished in 1308ms
 ** transaction: commit
    triggering after callbacks for `deploy:update'
  * 2013-11-24 18:11:20 executing `deploy:cleanup'
  * executing "sudo -p 'sudo password: ' ls -1dt /home/deploy/my_app/releases/* | tail -n +4 | sudo -p 'sudo password: ' xargs rm -rf"
    servers: ["my_app.nl"]
    [my_app.nl] executing command
 ** [out :: my_app.nl]

Once frozen, nothing happens. But if I edit anything in the project, like just a comment, and commit/push it, the deploy may succeed. No logical explanation is required.

@troelskn Yes, check it: `wizard:weedmap fschuindt$ cap deploy --verbose ** transaction: start Password: ** Updating mtimes for ~312 assets... ** scp upload # -> /home/deploy/weedsquare/TOUCH_ASSETS ** [out :: weedsquare.nl] ** transaction: commit ** [out :: weedsquare.nl] ` And it freezes... — fschuindt, Nov 26 '13 at 06:51
Possible duplicate http://stackoverflow.com/a/31275142/667127 — moeabdol, Jul 07 '15 at 17:13

score 0 · Answer 1 · answered Mar 07 '15 at 04:18

I encountered exactly the same thing. Deploy would completely hang on uploading the TOUCH_ASSETS file. In our case, it was working on ~815 assets.

I looked at the TOUCH_ASSETS file on the server and it looks like it never finished being written. The last line was truncated. I tailed the file a few times and it never changed. Also, when pressing ctrl-c to cancel the deploy, the rollback step failed as well. It is almost like the SSH connection was dropped.

Got my deploy to succeed by moving all the assets from shared/assets to a temporary folder, then deployed again. The deploy succeeded and all the assets were compiled.

This obviously won't work for a deploy to an online system. Thankfully I was deploying to a test environment, which can be wiped clean without issue. Hopefully this won't happen when trying to deploy to production.

Why is capistrano freezing in "deploy:cleanup"?

1 Answers1