How I learned Capistrano . . . 3

With my current web app my goal over the past few weeks has been to get it to a stable state so I can just deal with scaling it out.

To do that, I needed to reduce the amount of time I was spending in deploying updates and I needed a repeatable set of steps that I couldn't mess up. That meant I needed to use an automated deployment tool. Since this is a Ruby on Rails (RoR) project, Capistrano seemed the obvoius answer.

I hadn't used it in several years but I thought it wouldn't be too hard to get back up to speed. It turned out that I felt like I was having to relearn it all over again.

So I thought I would write this post to document how I went about getting up to speed again. It turned out to not be that bad.

To start with I ensured I had the prerequisites in this tutorial completed.

Since I already had a running application, steps 1 through 4 (which consist of installing Nginx, PostgreSQL, RVM, Ruby, Rails and Bundler) were already completed.

All I really had to complete was Step 5: Setting up SSH Keys so the server could download from the git repository.

Next I searched the net for tutorials on installing and configuring Capistrano. I soon realized the tutorials were a mashup of versions 2 and 3. It wasn't always clear that the post or tutorial I was looking at was about version 3.

My take away at this point was this wasn't going to be a one afternoon effort.

Additionally, as I feared, each author was very opinionated about how it should be setup and configured. Needless to say, my comfort factor was not high for this endeavor.

So here is how I ended up regrouping and getting it done.

I started at the Capistrano website. Reading through the documentation, it felt like I wasn't getting the whole picture. The documentation is very consise.

Don't get me wrong, I think for someone who knows what they are doing the docs are great, but for me (and my level of expertise at the time) it was confusing. I like to know things like "Why would I want to do this" and "How does this fit into the bigger context of what I am trying to do" but there is little, if any, of that in the docs.

It felt like I wasn't making any progress. What I did learn, however, was the lay of the land for the documentation, the different components involved (even though I didn't know how they connected) and this first nugget:

Nugget 1: Capistrano 3 is based on Rake

One post I read recommended to first get familiar with how Rake tasks are built. This turned out to be a good idea. It gave me a better understanding about the code I was seeing in the Capistrano 3 config files. This was the recommended post and it is well worth reading all the way through.

It was a great start.

After reading that post I came back to the Capistrano documentation. I read more. Specifically I read these sections:

• Installation
• Structure
• Configuration
• Preparing Your Application
• Cold Start
• Before/After Hooks
• Authentication & Authorisation

I also read the "Advanced Feature - Properties" section and then I looked at some of the plugins.

At this point I felt like I was ready to roll up my sleeves and get started.

A lot of the tutorials I ran across started with building the application and the server, and since I had already done that I won't go into those steps.

Installation

Capistrano installation was fairly straight forward, you just need to make sure you include the right gems. Since this is a RoR app, including the capistrano-rails gem was critical.

Nugget 2: Each gem adds to the list of Capistrano tasks you have available. You will need several of them.

Here is the list of gems I added to the Gemfile

gem 'capistrano'
gem 'capistrano-rails'
gem 'capistrano-bundler'
gem 'capistrano-rvm'

After installing the gems and running bundle update I then 'capified' the app:

bundle exec cap install

Having to do this seems to get glossed over in the Capistrano documentation.

I next added the require statements to the generated Capfile

require 'capistrano/rails'
require 'capistrano/rvm'

Note: To start with I purposefully did not add the nginx or unicorn capistrano gems so I could just concentrate on getting the app to install properly. I also chose to deploy the app to a test directory on the production server so I could compare it to the running application to know that it was working correctly.

At this point I could run cap -T to see the list of available tasks.

Configuration - Deploying the Code

Next, most of the tutorials, will take you into configuring the deploy.rb and production.rb files that got laid down when you 'capified' the app.

The documentation in both of these files is pretty good, but the problem was, at least for me, it was overwhelming. What EXACTLY did I need to set up to get going.

Nugget 3: config/deploy.rb is for common settings and configuration, config/deploy/production.rb is for settings specific to your production deployment and config/deploy/staging.rb is for settings for your staging environment (i.e. a pre-production/test environment). The settings in depoy.rb will be overridden by settings in production.rb or staging.rb depending on what you are deploying.

Next I wrote down the steps I would have to do if I was deploying by hand. These were the steps I wanted my Capistrano deploy setup to do. The steps I needed to accomplish were:

1. Run local tests
2. Stop nginx
3. Update the code
4. Run Bundle install (if needed)
5. Run rake db:migrate (if needed)
6. Compile assets
7. Link shared folders and files
8. Restart nginx
9. Restart unicorn
   

Knowing what I needed to do helped me figure out what settings I needed to set.

Here are the config settings I ended up with in the config/deploy.rb Note: anything listed in {} needs to be changed to match the server environment being run in.

set :application, '{my_app_name}'
set :deploy_user, '{deploy}'
set :scm, :git
set :repo_url, '{git@...}'
set :deploy_to, '{/home/deploy}'
set :format, :pretty
set :linked_files, %w{config/database.yml}
set :linked_dirs,  %w{bin log tmp/pids tmp/cache tmp/sockets vendor/bundle public/system}
set :keep_releases, 5
set :keep_assets, 2
set :tests [{a list of path+file names to the spec tests to run}]

Next I moved to the 'config/deploy/production.rb' file and set these attributes:

set :stage, :production
set :branch, 'master'
set :server_name, '{my server name}'
server '{my server name}', user: '{deploy}', roles: %w{web app db}, primary: true

set :rails_env, :production

set :ssh_options, {
    forward_agent: true,
    auth_methods: %w(password),
    user: '{deploy}'
}

I was finally ready to run the first deploy

cap production deploy

As expected it didn't work, but it was close.

There were some deprecation warnings I needed to fix, I found it was best just to look at the log and fix the issues.

The next problem I encountered was the deployment could not link to the 'database.yml' file from the shared directory. I expected this so I logged into the production server and put the file in place. Note: Capistrano has the ability to create/install the application on a new server. I chose not to go this route for this deployment as all the config files for nginx and unicorn were already in place.

Nugget 4: Review the capistrano output. It is important to fix as many problems as you see in the capistrano output. Some commands that are marked "failed" are ok as some directories may not exist in your project. You can use cap production deply --dry-run if you want to do a trial run of the deployment without actually doing it. This is good for when you make a large or potentially destructive change.

Basically you want to put anything that you can share between projects into the shared directory and symlink them over. One extra one I had to do was a hidden folder for my ssl certificates.

Once I cleaned as many issues up as I could, without doing any custom tasks I had items 3 through 7 working. Now on to the server stuff.

Installation - Controlling the Application Server

I first started by adding the nginx and unicorn gems to the Gemfile:

gem 'capistrano3-nginx'
gem 'capistrano3-unicorn'

I required both in the Capfile:

require 'capistrano/nginx
require 'capistrano3/unicorn' 

I also added the correct deploy steps to the deploy.rb file:

before :deploy, 'nginx:stop'
after 'deploy:publishing', 'deploy:restart'
task :restart do
  invoke 'nginx:restart'
  invoke 'unicorn:reload'
end    

At this point I started getting "sudo: no tty present and no askpass program specified" I fixed this by adding the NOPASSWD option to my deployers group on the server.

sudo visudo

#add this line to bottom of file
deployers ALL=(ALL) NOPASSWD: ALL

This allowed the Nginx service to be controlled by the deployer account.

At this point steps 2, 8 and 9 were complete on my list.

Installation - Running Tests Before Deployment

The final thing to complete was getting the tests running on the local development box prior to allowing the deploy to the server to happen.

To do this I created a custom task named run_tests.cap and placed it in the lib/capistrano/tasks directory. Here is it's contents:

namespace :deploy do
  desc "Runs test before deploying, can't deploy unless they pass"
  task :run_tests do
    test_log = "log/capistrano.test.log"
    tests = fetch(:tests)
    tests.each do |test|
      puts "--> Running tests: '#{test}', please wait ..."
      unless system "bundle exec rspec #{test} > #{test_log} 2>&1"
        puts "--> Tests: '#{test}' failed. Results in: #{test_log} and below:"
        system "cat #{test_log}"
        exit;
      end
      puts "--> '#{test}' passed"
    end
    puts "--> All tests passed"
    system "rm #{test_log}"
  end
end

I got this from a post I read. It should be fairly explanitory. It essentially fetches the tests from the array of tests I added to the deploy.rb file and runs them. If any test fails the deployment is halted.

To bring this custom task in I added these lines to the bottom of the Capfile:

# Load custom tasks from `lib/capistrano/tasks` if you have any defined
Dir.glob('lib/capistrano/tasks/*.rake').each { |r| import r }
Dir.glob('lib/capistrano/tasks/*.cap').each { |r| import r }
Dir.glob('lib/capistrano/**/*.rb').each { |r| import r }   

I am probably including way many more types of files than I should but it does cover all the bases so I went for it.

Finally I added this line to the deploy.rb file:

before :deploy, 'deploy:run_tests'     

I reran cap production deploy and all was good.

Of course this post glosses over many of the stops, starts, restarts that I did to get this to work. I bet I deployed the app more than 50 times before I got it all right.

Here are some other issues I ran into along the way:

I ran into a problem where on the production server the Gemfile.lock was out of sync with the actual gems I had installed. I tried to run bundle install on the server but it would fail until I moved into a real directory and not by running it in the current directory. It seems the symlink messed that up.

Another thing that bothered me was all the "failures" the Capistrano log showed while deploying. I found this post where the comment by bruno at the bottom on Nov 6 will give you some comfort. At least it did for me.

Another thing that was still very opaque to me was what tasks were really being run and what did they do? There was really no documentation I could find on that. The only way I got a clue of what was going on was looking at the source. That was pretty enlightening, but not for the faint of heart.

One other thing I learned, if you change the ":format" setting to ":dot" instead of ":pretty" you will see red and green dots as the deployment script runs instead of all the verbose output. This makes it "feel" better at least. Only do this once you know everything is working though.

Final Thoughts

It was a long process to get this going. I spent many hours 'googling' and reading posts to come up with this configuration. Was it worth it? I think so. I can reliably deploy the app in about 2 minutes all from my dev box. As a side affect I also can run tasks like nginx:restart at any time I want to which will help out in day-to-day server management.

I guess my next steps in this area is to better understand what the tasks are that are run, and configure the deployment so I could actually install the app with Capistrano on a new server. But that will have to wait for another day. It's time to get back to coding.

Till next time