Rob Seaman - Home

Detecting the number of processors with Ruby

2009-07-11T23:55:00Z

This post contains a cross platform Ruby method to return the total number of CPU cores. There are gems, such as facter, that will do the same thing and much more but if you just want a quick and dirty single purpose method this will probably suffice. The only tricky bit was picking the best way to do the determination in Windows. I ended up using a WMI approach described in Rick Minerich's There Counting Processors in .NET: The Pros and Cons of Five Different Methods post.

# Returns the number of processor for Linux, OS X or Windows.
def number_of_processors
  if RUBY_PLATFORM =~ /linux/
    return `cat /proc/cpuinfo | grep processor | wc -l`.to_i
  elsif RUBY_PLATFORM =~ /darwin/
    return `sysctl -n hw.logicalcpu`.to_i
  elsif RUBY_PLATFORM =~ /win32/
    # this works for windows 2000 or greater
    require 'win32ole'
    wmi = WIN32OLE.connect("winmgmts://")
    wmi.ExecQuery("select * from Win32_ComputerSystem").each do |system| 
      begin
        processors = system.NumberOfLogicalProcessors
      rescue
        processors = 0
      end
      return [system.NumberOfProcessors, processors].max
    end
  end
  raise "can't determine 'number_of_processors' for '#{RUBY_PLATFORM}'"
end

Adding Rubinius to multiruby Versions

2009-01-22T20:18:00Z

In previous posts I've covered using multiruby to switch between Ruby 1.8 & 1.9 and how to add JRuby to your multiruby environment. This post describes using multiruby as a way to install and experiment with Rubinius in Mac OS X.

In this post I assume you have git and a current version of multiruby, which is part of the popular ZenTest gem. If you do have it but your revision is older than 3.11.1 then you should update it. If ZenTest is not installed then see Switching Between Ruby 1.8 & 1.9 with multiruby for information about getting started with multiruby.

Installing Rubinius in `~/.multiruby`

If multiruby is ready to go, then the place to start is cloning Rubinius to the multiruby vendor directory:

$ cd ~/.multiruby/versions
$ git clone git://github.com/evanphx/rubinius.git

Rubinius support is built-in to multiruby but before letting multiruby do the build take a look at the ~/.multiruby/versions/rubinius/doc/getting_start.txt to see if you're missing any required software. In my case I was missing the ParseTree gem:

$ sudo gem install ParseTree

After getting any known missing prerequisites, the next step is to execute multiruby which will see rubinius in its versions directory and automatically try to build it:

$ multiruby -e "p 1+1"
building and installing rubinius
Running command: rake &> log.build

If you get an error check the log.build file in ~/.multiruby/build/rubinius. The most common problem will be a missing prerequisite, something that should be obvious in the log. If you don't get any errors then after a fairly lengthy build you should see a successful execution:

...
VERSION = rubinius
CMD     = ~/.multiruby/install/rubinius/bin/rbx -e p 1+1

2

RESULT = 0
...

TOTAL RESULT = 0 failures out of 5

Passed: 1.8.6-p287, jruby-1.1.6, 1.9.1-preview2, rubinius, 1.8.7-p72
Failed:

That's it, if executing Rubinius via multiruby is all you want to do then you're done. If you want to execute Rubinius independently of multiruby read on about adding it to your PATH.

Running Rubinious Outside of multiruby

Rubinius runs Ruby code with its own command name (rbx). To make the multiruby version of rbx available you need to add the rubinius/bin directory to the PATH in your login initialization file, i.e. your .bash_login file:

# manage ruby verisions
export PATH=$PATH~/.multiruby/install/rubinius/bin
export ORIGPATH=$PATH
...

I would recommend adding it to the end of your PATH variable, or at least past your existing ruby command path just in case you ever decide to add the ruby symlink that I'll discuss next.

Rubinius as Ruby via Alias Switching

In Switching Between Ruby 1.8 & 1.9 with multiruby I created aliases to set the path in the .bash_login to select which version of Ruby gets used:

# switch between ruby verisions
export ORIGPATH=$PATH
alias mr186p287='export PATH=~/.multiruby/install/1.8.6-p287/bin:$ORIGPATH'
alias mr187p72='export PATH=~/.multiruby/install/1.8.7-p72/bin:$ORIGPATH'
alias mr191pre2='export PATH=~/.multiruby/install/1.9.1-preview2/bin:$ORIGPATH'
alias mrjruby='export PATH=~/.multiruby/install/jruby-1.1.6/bin:$ORIGPATH'
alias mroff='export PATH=$ORIGPATH'

Since multiruby recognizes the rbx command you can use Rubinius anytime as long as you added it to your path and setting up an alias to switch over to Rubinius as your default Ruby implementation isn't required. However, if you have applications that launch ruby or you just want you to be able to execute Rubinius with the ruby command then you can still use the same alias technique I described in my previous post. First, create a ruby symlink with:

$ ln -fs ~/.multiruby/install/rubinius/bin/rbx \
> ~/.multiruby/install/rubinius/bin/ruby

Second, setup a new alias in your .bash_login with something like:

28	alias mrrub='export PATH=~/.multiruby/install/runius/bin:$ORIGPATH'

Now, after you reopen your terminal session, $ mrrub will use Multiruby whenever you execute the ruby command. You can switch back to your default Ruby implementation with $ mroff.

Again, since Rubinius executes via rbx and multiruby is aware of that, so symlinking and aliasing is totally optional. With JRuby the symlink was required for multiruby compatibility but the alias was optional. When I first setup JRuby I didn't bother with an alias but decided to add it later when I wanted to test scripts that launch ruby as a subprocess, i.e. `ruby -e .... This time I created the symlink and alias right from the start.

Upgrading to Mephisto 0.8.1

2009-01-15T01:51:00Z

Last month Mephisto 0.8.1: Drax was relaunched. Seeing activity on Mephisto was great news in itself. The update includes support for Rails 2.2.2 and several significant security fixes. The release is considered experimental but important because of the security fixes. That's a bit of a mixed message but I have no financial stake in my blog so decided to do an update.

This post describes my update process. It provides some general information about the changes and provides an example of using interactive git rebasing but in most cases it won't be a cookie cutter approach that you can follow for your own Mephisto implementation. In gitting Started with Mephisto and other posts I've described how I set-up Mephisto 0.8. Unless you've followed along and are maintaining Mephisto in the same way, or at least maintain your Mephisto blog in a git repository with customizations in your own branch, then your update procedure will probably be significantly different.

I should point out that I've been using the technoweenie version of Mephisto from github and not the emk version where all of the latest experimental work is going on. When I pulled from the master the latest commit in technoweenie branch was very close to the emk v0.8.1 (v0.8-stable) tag that marked the version that's available for download from the Mephisto site. The only additional commits in the official release were a safe_erb test update and emk-safe-erb was updated from 0.1.1 to 0.1.2. The main point is that when I talk about pulling the latest master, it's really a fairly stable version. At some point I may migrate over to emk but so far I haven't had a compelling reason to do so.

The Original Plan

In gitting Started with Mephisto I described my upgrade strategy for future releases using rebasing:

[mac]$ git checkout master
[mac]$ git pull
[mac]$ git checkout robseaman
[mac]$ git rebase master

The Updated Plan

When it came time to move to 0.8.1 I stuck with that basic concept but added a few details like a backup step and an interactive rebase instead of an automatic one. There were several reasons for going with an interactive rebase, the biggest ones were that I'd installed Rails 2.0.2 in vendor and used Cached Externals. Allowing the commits associated with those changes when I knew Mephisto had added support for Rails 2.2.2 and probably also changed other externals would result in a bunch of messy conflicts that I'd need to resolve, assuming it worked at all. Interactive rebasing would provide the control I needed to make the rebase reasonably clean. The first steps were to do a backup and pull down the latest Mephisto master branch:

[mac]$ cap data:load_from_prod
[mac]$ cd ..
[mac]$ cp -r mephisto mephisto_backup
[mac]$ cd mephisto
[mac]$ git checkout master
[mac]$ git pull

$ cap data:load_from_prod (see Production Data to Development for details) and the copy created a backup of the production database and 0.8 version. After the git-pull I ended up with revision e229865d6e63a6b6c0e6cb7aac21992ca303f94a of Mephisto.

With the latest master in hand I looked for changes that would impact my rebase process. More specifically I was looking for changes to any of the externals that I'd been caching (see Cached Externals). The first thing that stood out is that my pull of Mephisto included TZInfo-0.3.12 in the vendor/gems directory. TZInfo can't be vendorized so it shouldn't be there. In fact the first development commit (emk repository) made after the 0.8.1 release was to remove it. See Should TZInfo be included in vendor/gems? for details. Since the vendorized copy won't be used make sure you follow 0.8.1 download page instructions: "Before installing: Run sudo gem install tzinfo". Alternately you can make sure you have all the gems including TZInfo installed on your target by running:

[mac]$ rake gems:install

On with the rest of the inspection, I already knew I'd need to upgrade to Rails 2.2.2 and of course it wasn't embedded so the next step was to inspect the RSpec directories. The old 2962 versions in 0.8 were from subversion repositories. I knew that a lot of work went into updating the tests for Mephisto so I wasn't surprised to find the RSpec directories contained updated copies of git repositories. A search for "rspec" in gitk showed a checkin comment indicating that the latest github versions available on 12-4-2008 were used. Going over to github's rspec commits the last commit before 12-4-2008 was f85a53bb54b13e6461c8ffbded4933b90e40a86c and the commit for rspec-rails was f96eb3fd4d95eb400e87b4f1cf8e6a279f1dbcbd.

Interactive Rebasing

OK, now I had the basic information I knew I'd need. With that background I moved on to checkout my custom branch and start the interactive rebase:

[mac]$ git checkout robseaman
[mac]$ git rebase -i master

Interactive rebasing opens a file called git-rebase-todo in your editor. git-rebase-todo contains a list of the branch's commits with an action to be applied to each one. The default is to pick (apply) each one and apply them in order but they can be squashed (merged), edited after they are applied, deleted or reordered. Here was a pseudo edit copy of my list showing the changes I decided to make:

e f2b123a Changed to track changes for my blog.
DELETE> pick f25ef13 Froze to Rails 2.0.2.
DELETE> pick 286a82d Updated to Rails 2.0.2 boot.rb.
DELETE> pick c8efd30 Put tzinfo 0.3.12 in vendor dir.
pick dcddd47 Changed secret from distributed version.
pick 0c1579c Disabled multi-site support.
pick f78a7d3 Ran db:bootstrap
pick 7c4f203 Initial versions from capify.
pick d1d5efc First customized version of deploy.rb
pick 9a4a736 Added Railscasts CodeRay style.
pick 4cead81 Updated styles for coderay.
DELETE> pick b6e7478 Added doc directory.
pick b1ac0ef Restyled sidebar and meta fields.
e 5c0bdb4 Added Cached Externals plugin.
DELETE> pick f7308b9 Created externals YAML file.
DELETE> pick 4ff2557 Cached Externals: rails, tzinfo & rspec/rspec_rails
pick 4105041 Change the application user.
pick 5c29462 Added section article count, change tags to use linebreak as separator.
pick 40507f7 Added more coderay standard defs to help code match to linenums.
pick 0289fc4 Added standard coderay.css (from coderay_stylesheet) for reference.
pick 6ff1ba4 Increased top and bottom padding to give a little breathing room.
pick d978f48 Added git://github.com/caritos/action_mailer_tls.git for TLS mail support.
pick a874664 Added gmail blog account settings.
DELETE> pick 4d153be Added action_mailer_tls to the externals list.
pick 66bc24c Commented out display of links to other pages in the same section.
pick 0bfb9c8 Removed Search sidebar title, Added min-nav icon section.
pick df51dd7 Added email subscriptions.
pick 11905d1 Changed mini-nav icon to include text and increased site font size.
pick b1c246b Changed Contact to 'Contact Me'.
pick 58d48cd Changed Contact Me to just Contact.

# Rebase e229865..58d48cd onto e229865
#
# Commands:
#  p, pick = use commit
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

The line 1 commit contained changes to .gitinore and line 12 was a .gitinore for a doc directory where I save HTML edits of articles. I deleted line 12 and changed line 1 to an edit point so that I could recall the commit and do all .gitinore changes at once. I deleted lines 2-4 because I wanted to add Rails 2.2.2 and take care of TZInfo as part of the Cached Externals setup, something that I'd redo after the plugin was added after line 14. I deleted line 24 because I decided not to continue to include the action_mailer_tls plugin in my Cached Externals setup. I was on the fence about adding it in the first place; the plugin is so small that it didn't provide a performance benefit. My reason for adding it was simply documentation to keep track of its revision information but I have that recorded elsewhere. Leaving it out meant one less consideration the next time I updated Mephisto.

After saving the file there was an immediate conflict on the first commit:

warning: too many files, skipping inexact rename detection
Auto-merged .gitignore
CONFLICT (content): Merge conflict in .gitignore
Automatic cherry-pick failed.  After resolving the conflicts,
mark the corrected paths with 'git add ', and
run 'git rebase --continue'
Could not apply f2b123a... Changed to track changes for my blog.

A new .gitignore

Here's what .gitinore contained:

.DS_Store
<<<<<<< HEAD:.gitignore
.rake_tasks
Capfile
config/database.yml
config/deploy.rb
config/initializers/session_store.rb
db/*.sqlite3
log/*.log
public/assets
public/cache
public/plugin_assets
tmp
themes
*~
=======
config/database.yml
log
tmp
public/assets
>>>>>>> f2b123a... Changed to track changes for my blog.:.gitignore

I covered setting up the .gitinore in gitting Started with Mephisto but there were some new considerations with Mephisto's 0.8.1 .gitinore. Here were the additional things I considered when deciding how to resolve the conflict:

config/initializers/session_store.rb is a file that Mephisto now generates to avoid the weakness with using a common session secret. Something I had worked around in my original post by generating my own key. The file is generated as part of the bootstrap process or it can be created independently with a new db:bootstrap:session or config/initializers/session_store.rb rake task. It's obvious the file doesn't belong in Mephisto's project repository on github but I decided to leave it in my private git repository of Mephisto to make deployment easy.
db/*.sqlite3 is a new ignore. I'm not using SQLite but I decided I'd ignore these files in case they get generated automatically as part of running the test suite.
public/assets was something that I had already added. public/cache and public/plugin_assets are automatically generated by Mephisto and not part of the code base so I decided I'd also make them part of my .gitignore.
*~ is for Emacs backup files. I'm not using Emacs so I didn't bother with adding it.
log/*.log is a more specific ignore for log files. I don't see much significance but went with the new method to keep my implementation in sync if something important happens to get added to the log directory so I decided to accept this change.
The doc directory is something I'd originally added to .gitinore as a separate commit but deleted the line in interactive mode so I could add it after this first .gitignore commit. Since there was a conflict I decided I'd add it as part of this conflict resolution.

So that lead to this version of .gitignore:

.DS_Store
db/*.sqlite3
log/*.log
public/assets
public/cache
public/plugin_assets
tmp
config/database.yml
doc

Resolving Conflicts

Adding the resolution and continuing is the standard thing to do after resolving a conflict:

[mac]$ git add .gitignore
[mac]$ git rebase --continue
warning: too many files, skipping inexact rename detection
Auto-merged config/environment.rb
CONFLICT (content): Merge conflict in config/environment.rb
Automatic cherry-pick failed.  After resolving the conflicts,
mark the corrected paths with 'git add ', and
run 'git rebase --continue'
Could not apply dcddd47... Changed secret from distributed version.

I wasn't surprised by this config/environment.rb problem. I knew I'd updated the file to create my own session key and had just discovered there was a fix for this in Mephisto with the addition of config/initializers/session_store.rb to .gitignore. So now I needed to delete the following lines (my original change):

<<<<<<< HEAD:config/environment.rb
=======
  # Use the database for sessions instead of the file system
  # (create the session table with 'rake create_sessions_table')
  # config.action_controller.session_store = :active_record_store
  config.action_controller.session = { :session_key => "_mephisto_session", ....

>>>  >>> dcddd47... Changed secret from distributed version.:config/environment.rb

And move on:

[mac]$ git add config/environment.rb
[mac]$ git rebase --continue
warning: too many files, skipping inexact rename detection
Auto-merged config/initializers/custom.rb
CONFLICT (content): Merge conflict in config/initializers/custom.rb
Automatic cherry-pick failed.  After resolving the conflicts,
mark the corrected paths with 'git add ', and
run 'git rebase --continue'
Could not apply 0c1579c... Disabled multi-site support.

This custom.rb conflict occurred because I had disabled multisite support near another change that went into 0.8.1:

<<<<<<< HEAD:config/initializers/custom.rb
  # TODO - We need to wrap these options in Dispatcher.to_prepare, because
  # the underlying Site, etc., objects will be unloaded on each request in
  # development mode, and the preferences will be reset.  This is a flaw
  # in how we handle configuration, and to_prepare is just a workaround.
  Dispatcher.to_prepare do
    # Turn this on to get detailed cache sweeper logging in production mode
    # Site.cache_sweeper_tracing = true
=======
# Enable if you want to host multiple sites on this app
# Site.multi_sites_enabled = true
>>>>>>> Disabled multi-site support.:config/initializers/custom.rb

    # Enable if you want to host multiple sites on this app
    Site.multi_sites_enabled = true

Last spring Mephisto was changed to enable multisite support by default but most people will probably encounter it for the first time in 0.8.1. I discussed why I disabled it in gitting Started with Mephisto. Hopefully it's working fine now (if it wasn't before) but I still don't need multisite support and decided to keep it disabled so I deleted lines 8,16-19 and commented out line 22:

22	# Site.multi_sites_enabled = true

I do have a side note here. If you decide to leave Site.multi_sites_enabled set to true and you're using Phusion Passenger then there is one consideration that I've become aware of: Enabling multisite support causes your cache files to be placed under a cache directory (i.e. public/cache/site.com) instead of in public root. If you want Passenger to find the cached files may may need mod_rewrite modifications. Check out Paul Gross's Mephisto with Phusion Passenger post for details. Oh well, that's not an issue for me so on to the next stop:

[mac]$ git add config/initializers/custom.rb
[mac]$ git rebase --continue
Could not apply 0c1579c... Disabled multi-site support.
Robs-Laptop-2:mephisto Rob$ git add config/initializers/custom.rb
Robs-Laptop-2:mephisto Rob$ git rebase --continue
Created commit 47398d9: Disabled multi-site support.
 1 files changed, 1 insertions(+), 1 deletions(-)
Stopped at 5c0bdb4... Added Cached Externals plugin.
You can amend the commit now, with

	git commit --amend

Once you are satisfied with your changes, run

	git rebase --continue

Updating Cached Externals

Finally, I hit my first planned stop after the plugin was added. At this point I needed to complete the Cached Externals setup, which was pretty much a subset of what I had done in my Cached Externals post.

Although TZInfo was installed in vendor/gems in my copy of Mephisto, as I mentioned above it'll be gone in the next stable release. Vendorized versions of TZInfo won't be loaded, at least for now, because Rails (2.1.2+) loads its own TZInfo version that is unfortunately just a subset without support for TimezoneProxy which is required by Mephisto when you access Admin/Settings. Since it can't be vendorized I verified that 3.12 was installed on my production server and no longer plan to cache it. I was tempted to remove the gem from my Mephisto implementation but decided to live with it being there for now and wait for the next stable Mephisto release.

I did want to cache the new RSpec versions so the current vendor copies needed to be removed:

[mac]$ rm -rf vendor/plugins/rspec
[mac]$ rm -rf vendor/plugins/rspec_on_rails

There was no need to freeze Rails 2.2.2, delete it and then include it as a Cached External. Just making it part of config/externals.yml would effectively freeze it into vendor. So, using information from the 2.2.2 tag on github and what I'd found after the pull, here was my new config/externals.yml:

vendor/rails:  # v2.2.2
  :type: git
  # DH old git checkout (1.4.4.4) doesn't support -q
  :scm_verbose: true
  :repository: git://github.com/rails/rails.git
  :revision: ff561370739eb809d1d006ea842cd2b1e43f6950
vendor/plugins/rspec: # 12-4-2008
  :type: git
  # DH old git checkout (1.4.4.4) doesn't support -q
  :scm_verbose: true
  :repository: git://github.com/dchelimsky/rspec.git
  :revision: f85a53bb54b13e6461c8ffbded4933b90e40a86c
vendor/plugins/rspec_on_rails: # 12-4-2008
  :type: git
  # DH old git checkout (1.4.4.4) doesn't support -q
  :scm_verbose: true
  :repository: git://github.com/dchelimsky/rspec-rails.git
  :revision: f96eb3fd4d95eb400e87b4f1cf8e6a279f1dbcbd

With that file in place it was time to use the configuration to load up the cache and create the symbolic links:

[mac]$ cap local externals:setup

At this point I did a little verification using git-status and then committed my changes:

[mac]$ git add config/externals.yml
[mac]$ git commit config/externals.yml -m "Created externals YAML file."
[mac]$ git add .
[mac]$ git commit -a -m "Cached Externals: rails, rspec and rspec_rails."
[mac]$ git rebase --continue
Successfully rebased and updated refs/heads/robseaman.

Common 0.8.1 Update Stuff

Great, no more conflicts and that completed the rebasing. Just a couple of 0.8.1 update related tasks remained:

[mac]$ rake config/initializers/session_store.rb
[mac]$ git add config/initializers/session_store.rb
[mac]$ git commit config/initializers/session_store.rb -m \
> "Created session key for this Mephisto site."
[mac]$ rake db:migrate
[mac]$ git add db/schema.rb
[mac]$ git commit db/schema.rb -m "Ran migrations for Mephisto update."

To work with the new security scheme config/initializers/session_store.rb needed to be created and since I decided to put it in my git repository it also needed to be committed. 0.8.1 had a new migration so running migrations and checking in the updated schema was also required.

New Theme Locations

I figured I was done but the first time I ran script/server I got a 500 error on my first page load and found this entry in the log:

MissingThemesError (No themes found in 
'/Users/Rob/work/mephisto/themes/development/site-1/simpla'.  
This must be set correctly in the site settings. You may be 
able to fix this by running 'rake db:bootstrap:copy_default_theme'):

After looking around a little I discovered the location of themes for non-production environments was changed to reflect the environment. If you'd rather keep the code that way you could just run in production mode by making sure you have usable production database settings in the database.yml and running script/server -e production from the local environment, or you could keep a separate copy of your production theme in themes/development. I didn't like those solutions, I wanted to run my production theme(s) in the development environment and I didn't want to deal with making sure my development themes were always copied over to production. So, instead of living with the change, I chose to modify the line in Mephisto that sets the THEME_ROOT location in lib/mephisto/theme_root.rb from:

7	(Rails.env.production? ? "themes" : "themes/#{Rails.env}")

to:

7	(Rails.env.test? ? "themes/#{Rails.env}" : "themes")

Then I checked in the change and did some verification:

[mac]$ git add lib/mephisto/theme_root.rb
[mac]$ git commit lib/mephisto/theme_root.rb -m \
> "Made themes/ the default themes dir for all except test."
[mac]$ script/server

Deploying to Production

Once I was satisfied that things were working it was time to deploy to production. Deploying Mephisto with Capistrano to DreamHost describes my deployment setup, but I'm doing deployments in a standard way:

[mac]$ cap deploy:migrations

In the end the update wasn't too bad. The upgrade complexities associated with Cached Externals wouldn't have occurred in a normal project, they only existed because I decided to use Cached Externals on someone else's project in my custom branch. Even at that once I figured out what had changed the update went pretty quickly.

Updating Gmail SSL SMTP Support for Ruby 1.8.7

2009-01-14T06:03:00Z

In Adding Gmail SSL SMTP Support to Rails I described the use of a simple Rails plugin to provide SSL SMTP support. Today I upgraded to a new version of Mephisto and while testing it in production I discovered that I couldn't send emails. It turns out that it had nothing to do with the Mephisto upgrade; my DreamHost server had been updated to Ruby 1.8.7 and the action_mailer_tls plugin I was using wasn't compatible. Oh well, I guess I deserved that surprise for using shared hosting and adding a monkey patch plugin. This post describes a simple fix.

The problem occurred because the 1.8.7 version of Net::SMTP changed its private check_auth_args method. Fortunately, I had just removed the action_mailer_tls plugin from Cached Externals so I was able to do a quick code fix in the plugins directory. Here's the change I made to vendor/plugins/action_mailer_tls/lib/smtp_tls.rb.

In Net::SMTP.class_eval#do_start replace:

8	check_auth_args user, secret, authtype if user or secret

with:

RUBY_VERSION.match(/(\d+)\.(\d+)\.(\d+)/)
if ($1.to_i <= 1 && $2.to_i <= 8 && $3.to_i <= 6)
  check_auth_args user, secret, authtype if user or secret
elsif user or secret
  check_auth_method(authtype || DEFAULT_AUTH_TYPE)
  check_auth_args user, secret
end

Check it in:

[mac]$ git add vendor/plugins/action_mailer_tls/lib/smtp_tls.rb
[mac]$ git commit -a -m "Fixed auth check method for 1.8.7 compatibility."

And deploy it:

[mac]$ cap deploy

If you're starting from scratch and want something that already works I did notice there are other similar solutions with 1.8.7 support available on the Internet. Simplificator's tls-support gem on github is one example.

Adding JRuby to multiruby Versions

2008-12-21T06:53:00Z

In Switching Between Ruby 1.8 & 1.9 with multiruby I covered setting up and using Ruby 1.8 and 1.9 versions in multiruby. This post describes adding JRuby to the list of multiruby versions in Mac OS X.

Install JRuby in `~/.multiruby`

multiruby doesn't have built-in support for JRuby but it works by tracking its installations in the ~/.multiruby/versions and ~/.multiruby/install directories. Setting up the directory structure to look like a MRI build from a tarball does most of what's necessary:

$ curl -O http://dist.codehaus.org/jruby/1.1.6/jruby-bin-1.1.6.tar.gz
$ tar -C ~/.multiruby/install -zxf jruby-bin-1.1.6.tar.gz
$ mv jruby-bin-1.1.6.tar.gz ~/.multiruby/versions/jruby-1.1.6.tar.gz

That will provide access to the jruby command but multiruby only knows about using ruby so we'll need to fool it with a symbolic link:

$ ln -fs ~/.multiruby/install/jruby-1.1.6/bin/jruby \
> ~/.multiruby/install/jruby-1.1.6/bin/ruby

Now verify that it works:

$ multiruby -e "p 1+1"

Although that installation was from a binary distribution it wouldn't be much more difficult to do it from source. You just need to make sure the base names in the versions and install directories match when you're done.

Adding JRuby to the PATH

In Switching Between Ruby 1.8 & 1.9 with multiruby I created aliases to set the path in the .bash_login to selected version of Ruby:

# switch between ruby verisions
export ORIGPATH=$PATH
alias mr186p287='export PATH=~/.multiruby/install/1.8.6-p287/bin:$ORIGPATH'
alias mr187p72='export PATH=~/.multiruby/install/1.8.7-p72/bin:$ORIGPATH'
alias mr191pre2='export PATH=~/.multiruby/install/1.9.1-preview2/bin:$ORIGPATH'
alias mroff='export PATH=$ORIGPATH'

If we wanted to execute JRuby with the ruby command then we could add an alias for JRuby too but since JRuby has its own command names (jruby, jgem,jirb) we can allow them to always be available if we just make sure that the ruby symlink we created doesn't come before any real ruby commands by adding it to our path:

# manage ruby verisions
export PATH=$PATH~/.multiruby/install/jruby-1.1.6/bin
export ORIGPATH=$PATH
...

BTW: If you want an alias that you can execute to make the ruby command executes JRuby:

25	alias mrjruby='export PATH=~/.multiruby/install/jruby-1.1.6/bin:$ORIGPATH'

then you'll probably also want to setup symlinks for gem -> jgem and irb -> jirb. That setup could be done instead of or in addition to the PATH change on line 23 above. I didn't bother with the alias or the additional symlinks; to me using the JRuby commands in their native jXXX format seems like the way to go.

That's it, JRuby is now a first class citizen in multiruby and multiruby_setup and jruby, jgem, and jirb are all available.

Switching Between Ruby 1.8 & 1.9 with multiruby

2008-12-20T02:06:00Z

I just recently built and installed three different Ruby versions (see Installing Ruby 1.8.6, 1.8.7 and 1.9 on Leopard). Following that I wanted to do some benchmark testing between 1.8 and 1.9 on a separate project so I began to use ZenTest's multiruby. It worked great for the benchmarking and I started to think about how to use multiruby as a way to manage and switch between Ruby installations. As I started looking into it I came across Dr Nic's Future proofing your Ruby code. Ruby 1.9.1 is coming post, which turned on the rest of the lights for me. The Dr Nic post does a nice job of describing how to use multiruby including setting up specific Ruby versions, adding gem support and using it with sake tasks. I'll cover some of the same setup and usage information so that this post stands alone but I won't repeat it all so I recommend you read Dr Nic's post. In addition I'll provide information about how to overcome a RubyGems installation issue that I encountered and how to use the multiruby Ruby installations as a way to dynamically switch between Ruby versions. I'll also discuss how the static build and install from source approach used in Installing Ruby 1.8.6, 1.8.7 and 1.9 on Leopard could be modified to be provide a non-multiruby dependent way to do Ruby version switching. At the end of the post I'll mention using symlinks as an alternative and how that approach could be used with multiruby.

Set-Up

multiruby is part of the popular ZenTest gem so it's likely you already have it. If you do have it but your revision is older than 3.11.0 you should update it, if it's not installed then it's available on rubyforge:

$ sudo gem install ZenTest

When multiruby is run for the first time it sets up a hidden directory structure for Ruby versions in user home ( ~/.multiruby) and downloads and installs the latest 1.8 and 1.9 releases of Ruby:

$ multiruby -e "p 1+1"
  ...
  Put other ruby tarballs in ~/.multiruby/versions to use them.
  ...
VERSION = 1.8.7-p72
CMD     = ~/.multiruby/install/1.8.7-p72/bin/ruby -e p 1+1

/Users/Rob/.multiruby/install/1.8.7-p72/bin/ruby: no such file to load -- ubygems 
      (LoadError)

RESULT = 256

VERSION = 1.9.0-5
CMD     = ~/.multiruby/install/1.9.0-5/bin/ruby -e p 1+1

2

RESULT = 0

TOTAL RESULT = 1 failures out of 2

Passed: 1.9.0-5
Failed: 1.8.7-p72

I didn't show all of the output but left in the "Put other ruby tarballs in ~/.multiruby/versions" line because it's worthy of note. If the installation worked then you'll see "RESULT = 0" for each of the Ruby versions but in my case I had a "no such file to load -- ubygems" error on the 1.8.7 version. This is a result of a conflict with the current Ruby installation. If you get the same error just temporarily unset the RUBYOPT environment variable ($ unset RUBYOPT ) and continue following these instructions. Updating RubyGems (coming right up) will correct the problem.

multiruby_setup is used to manage the installed ruby versions. The next step is to use the command to install the latest RubyGems update:

$ multiruby_setup update:rubygems

Managing Installations

Typing "$ multiruby_setup" without any arguments will provide you with a usage list. One of the first things you'll want to know is how to control which versions of Ruby you use. The automatic version detection when multiruby_setup is run the first time makes it easy to get started but in the case of 1.9 that means v1.9.0-5 is installed and it's more likely you'll want the latest 1.9.1 preview release. Dr Nic showed installing using subversion ($ multiruby_setup mri:svn:tag:v1_9_1_preview2). So, just to be different, here's how to add preview2 from a tarball:

$ curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-preview2.tar.gz
$ mv ruby-1.9.1-preview2.tar.gz ~/.multiruby/versions
multiruby -e "p 1+1"
$ multiruby -e "p 1+1"

Dr Nic also mentioned how to get a list of Ruby tagged releases with subversion ($ svn list http://svn.ruby-lang.org/repos/ruby/tags). The equivalent for tarballs for 1.8 and 1.9 would be:

$ curl ftp://ftp.ruby-lang.org/pub/ruby/1.8/ 
$ curl ftp://ftp.ruby-lang.org/pub/ruby/1.9/

Now that we have the 1.9.1-preview2 release, here's the command to remove v1.9.0-5:

$ multiruby_setup rm:1.9.0-5

RubyGems

By default each version has it's own copy of RubyGems. If you want to leave it that way and install the same gems in all versions you can use Ruby's -S option with multiruby. multiruby also looks for the VERSIONS and EXCLUDED_VERSIONS environment variables with verions separated by a ":". For example, if we had three Ruby versions installed:

$ multiruby_setup list
Known versions:
  1.8.6-p287
  1.8.7-p72
  1.9.1-preview2

and wanted to install Rails in just the 1.8 versions we could use:

$ export VERSIONS=1.8.6-p287:1.8.7-p72
$ multiruby -S gem install rails

$ export EXCLUDED_VERSIONS=1.9.1-preview2
$ multiruby -S gem install rails

Allowing each version to have it's own copy of gems provides fine grained control. If you'd rather just have all versions share gems (1.8 & 1.9 will still be separate) that can be done too:

$ multiruby_setup rubygems:merge

Switching Between Ruby Versions

The gem install example shows how multiruby allows control of which versions it runs by setting environment variables but it adds output that you probably don't want when running a single version of Ruby. Dr Nic also points out how you can isolate and run a specific cucumber scenario against a specific ruby version by cutting and pasting the results of a multiruby run. It's simple to extend those concepts to allow any of the Ruby commands to change if we just modify the path. For example, to use the 1.9.1-preview2 version installed above:

$ export PATH=~/.multiruby/install/1.9.1-preview2/bin:$PATH
ruby -v
ruby 1.9.1 (2008-12-01 revision 20438) [i386-darwin9.5.0]

Having the path set is convenient since it also makes 1.9.1 irb, erb, gem and gem commands directly available.

Of course it's a pain to type the export command each time and it would be nice to easily get back to the default Ruby installation. A simple solution is to add add aliases to your login initialization file. Depending on your shell this may be done differently but for my bash shell I added the following to the end of ~/.bash_login:

# switch between ruby versions
export ORIGPATH=$PATH
alias mr186p287='export PATH=~/.multiruby/install/1.8.6-p287/bin:$ORIGPATH'
alias mr187p72='export PATH=~/.multiruby/install/1.8.7-p72/bin:$ORIGPATH'
alias mr191pre2='export PATH=~/.multiruby/install/1.9.1-preview2/bin:$ORIGPATH'
alias mroff='export PATH=$ORIGPATH'

So switching to the latest 1.8.6 Ruby versions becomes as easy as:

$ mr186p287

"$ mroff" reverts to the default install and "$ alias" provides a reminder of what's available. I haven't done a lot with this switching method yet but so far it seems to work for me. If it continues to work well at some point I may even remove my /usr/local Ruby installation and just add mr186p287 to the end of my .bash_login. Of course, I'll leave the Leopard 1.8.6 Ruby installation alone.

Other Ways to Switch Ruby Versions

I haven't tested this but the same alias-PATH method used above could be done without multiruby by doing /usr/local build and installs into a separate installation directory (instead of /usr/local/bin) and changing the configure portion of the build. For example, 1.9.1-preview2 could be build from source and aliased with:

$ sudo mkdir -p /usr/local/1.9.1-preview2
$ cd /usr/local/src
$ curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-preview2.tar.gz
$ tar xzvf ruby-1.9.1-preview2.tar.gz
$ cd ruby-1.9.1-preview2
$ ./configure --prefix /usr/local/1.9.1-preview2 --enable-shared \
> --enable-pthread CFLAGS=-D_XOPEN_SOURCE=1
$ make
$ sudo make install
$ cd ..

and:

# switch between ruby verisions
export ORIGPATH=$PATH
alias mr191pre2='export PATH=/usr/local/bin/1.9.1-preview2/bin:$ORIGPATH'
alias mroff='export PATH=$ORIGPATH'

in the .bash_login. Again, I haven't tested this so there may be some tweaking to get the installation correct but it's the same principle as we used with the multiruby solution. The upside is that you have complete control of builds and installations, but that seems minor there's no reason you can't do your own builds in the ~/.multiruby directories. On the downside you lose the sharing of Ruby versions with multiruby and you lose multiruby and multiruby_setup's abilities to manage the versions as a group.

Another way to switch, and I what I thought I'd use prior to thinking about using PATH to switch between Ruby installations is a symlink system like Daniel Manages describes in his Install Multiple Versions of Ruby on Leopard. It's very similar to the /usr/local method I just described but uses simlinks in /usr/local/bin instead of adding to PATH. The symlinks could also be used with the ~/.multiruby/install versions to make to work with multiruby as well.

Sending CTRL-C to a Subprocess with Ruby

2008-12-12T17:49:00Z

Ruby provides several simple ways to launch subprocesses. Ruby's popen method comes into play in situations where cross platform handling of subprocess IO is required. Ruby also provides mechanisms to trap and send signals, such as the SYSINT (Control-C) signal. Putting popen and signal handling together to launch and manage a subprocess is straight forward but there's a limitation; popen lacks support for capturing standard error. By default standard error output is just displayed on the screen as if the subprocess was launched from the command line. There are a couple of workarounds available, in some cases it may be acceptable to discard standard error by redirecting it as part of the subprocess launch command, i.e. "command 2>/dev/null"; another, and typically better, option is to redirect standard error so that it's merged with standard output, i.e. "command 2>&1". Unfortunately the redirection impacts the Process ID (pid) for the launched subprocess, which breaks the simple standard signal handling that I mentioned earlier. In addition to the pid issue, getting the Control-C signalling support to work in Windows can be challenging. This post walks through an example that shows popen usage and how to send Control-C signals to a subprocess in Windows and Unix when standard error is redirected. It concludes with a brief discussion of popen3 and popen4 as alternatives to popen.

I'll start with a simple Ruby program called sub.rb that will stand in for a program that we want to execute:

trap("INT") do 
    STDERR.puts "sub pid #{$$} Control-C"
    exit 2
end

puts "#{$$}"
STDOUT.flush
sleep 10

The program prints its pid ($$) and exits normally after 10 seconds if it doesn't receive a Control-C. If Control-C is pressed or sent from another process the trap handler is executed, it prints an error message and exits with an error code. Ruby allows traps to be specified by number or as a long or abbreviated string. For example, 2, 'SYSINT' or just 'INT' can be used to specify the Control-C trap. For a complete list of available traps open irb and type Signal.list. If you intend to use signals in a application targeted to multiple platforms, keep in mind that the list is OS dependent with Windows only supporting a small number of traps. Back to sub.rb, it's easy to check out the program by running it from the command line and pressing Control-C:

[mac]$	 ruby sub.rb
2994
^Csub pid 2994 Control-C

Unix without Standard Error

Next up is program called popen_sub.rb that will execute sub.rb, capture its output and send it a Control-C:

IO.popen("ruby sub.rb") do |pipe|
  puts "parent pid: #{$$}, popen return (child) pid: #{pipe.pid}"
  line = pipe.gets      # pid from child
  puts "child says it's pid is: "+line
  Process.kill 'INT', pipe.pid
  childs_last_word = pipe.gets
  if childs_last_word
    puts "child's last word: " + childs_last_word
  end
end
puts "child's exit code: #{$?.exitstatus}"

This first pass shows a basic way to use popen and capture the subprocess output. Process.kill sends a SYSINT (Control-C) signal to the subprocess and concludes by printing $?.exitstatus, which contains the exit code. So what happens when it's run:

[mac]$	ruby popen_sub.rb
parent pid: 3046, popen return (child) pid: 3047
child says it's pid is: 3047
sub pid 3047 Control-C
child's exit code: 2

It works, popen captured standard output from sub.rb, sent it a Control-C and captured its exit code.

Adding Support for Standard Error

The code worked but popen_sub.rb didn't capture sub.rb's standard error output as the childs_last_word. The standard error output just went directly to the display. Since we're already capturing standard output a simple way to capture standard error is to redirect the error stream when sub.rb is launched:

IO.popen("ruby sub.rb 2>&1") do |pipe|

That's easy and an Internet search will make it clear that it's a common practice. so what happens when the updated popen_sub.rb is run:

[mac]$	ruby popen_sub.rb
parent pid: 3060, popen return (child) pid: 3061
child says it's pid is: 3062
# 10 second delay
child's exit code: 0

That not what we wanted, obviously the SYSINT signal wasn't received and you can see why by looking at the pid. The redirection caused the pipe.pid to return 3061 but the actual pid for sub.rb was 3062. Now what? We can't just increment the pid returned by pipe.pid; assuming pids are assigned in certain ways by all OS's and that the pipe.pid from IO.popen will never handle redirection correctly is weak. But we could send the SYSINT to all processes in the current process group. There are a couple of ways to do that:

Process.kill 'INT', -Process.getpgrp

Process.kill 'INT', 0

Passing a negative pid indicates that it's a group ID. Using Process.getpgrp (or it's close relative -Process.getpgid($$)) to retrieve the process group ID is direct and obvious. So how does using zero work? Unfortunately Ruby's rdoc for Process.kill is incorrect regarding using a pid of zero. It indicates the signal is sent to the current process but that's not true. Looking at the rb_f_kill() function in Ruby's signal.c module there's some handling to support Posix and Berkley style kill() calls but other than that pids are just passed to the Unix OS call. Some Unix documentation states that using zero for the pid will send the signal to all processes, except system processes but I've only seen behavior consistent with this description:

"If pid is 0, sig shall be sent to all processes (excluding an unspecified set of system processes) whose process group ID is equal to the process group ID of the sender, and for which the process has permission to send a signal."

With either group ID solution the parent process will also get the SYSINT signal so it will need to have a trap handler in place when the signal is sent. The handler doesn't have to do anything so it can be as simple as this:

trap("INT") {}

If the parent doesn't setup a trap handler then Ruby's default SYSINT exception handling will take place, printing an error and terminating the program. That leads to the biggest caveat with this approach, since the signal goes to processes in the group, not just the current process and child processes, all parent processes need to set-up a handler and ignore the signal. This isn't an issue when you run the application directly from Ruby since it's the top-level parent. It can even be made to work as a child if you write the code for the ancestor processes but it is an issue when the Ruby application is spawned by a process. Unless you go through the trouble to patch the top-level parent so it handles signals the way you want then when the Ruby app decides to send a SYSINT, the parent will get the signal too and in most cases terminate. For apps that are intended to be standalone this isn't a show stopper but it's something to beware of. One common situation where it shows up is when running tests. It's not an issue with directly running unit or coverage tests (i.e. ruby test/test_xxx.rb or rcov test/test_xxx.rb) but it is a problem when running those tests with rake or rake coverage. In my test code I put code in to check for Process.getpgrp == Process.pid, if it's false I print a warning and skip tests that result in SYSINTs. Another workaround would be to place similar code in the application itself to avoid sending a SYSINT when there's a parent.

Making it work in Windows

There's been Process.kill support for Windows since Ruby 1.7.x but Ruby's win32.c kill() function has this check at the top:

if (pid <= 0) {
  errno = EINVAL;
  return -1;
}

That makes it pretty obvious that only individual process IDs are supported in Windows. Actually, I'm not sure if it would matter if there was group pid support since I wasn't able to get Ruby's Process.kill to work with standard pids in Vista or XP using the original non-redirect popen_sub.rb from above. That seems hard to believe so may be I did something wrong. I was focussed on group pids by the time I looked at Windows, so, I didn't dig into it but it definitely fell short of my expectation that the same code would just work in both Windows and Unix.

Win32 Utils provides an alternative with its win32-process gem. The gem is installed with the One-Click Ruby Installer for Windows gem pack so it's already part of most Windows installations. At first glance it doesn't appear to be much better than Ruby's built in support. Again, there's no group PID support so Process.getpgrp and -Process.getpgid($$) won't work. It does support using zero for the pid but unfortunately it implements the support as Ruby's rdoc says it should instead of the way the Ruby code implements it. Specifically, it intercepts the zero and translates it to the pid of the current process. And, just like with the standard Ruby implementation, Process.kill didn't work for me with normal pids on Vista or XP.

Ah, but there's a loophole that we can take advantage of in the win32-process gem. If we use nil for the pid it will get translated to NULL (0L in c/c++) before being passed as the dwProcessGroupId parameter to the GenerateConsoleCtrlEvent() Windows function. A zero for that parameter means that "the signal is generated in all processes that share the console of the calling process". Yeah baby! OK, it's a hack and depends on the implementation of win32-process but we don't have a lot of choices.

That leads to the final version of popen_sub.rb:

if RUBY_PLATFORM =~ /win32/
  require "win32/process"
  
  # GenerateConsoleCtrlEvent, dwProcessGroupId = 0L < NULL < nil
  def sysint_gpid; nil; end
else
  def sysint_gpid; -Process.getpgrp; end
end

IO.popen("ruby sub.rb 2>&1") do |pipe|
  trap("INT") do
    puts "parent pid: #{$$} Control-C"
  end

  puts "parent pid: #{$$}, popen return (child) pid: #{pipe.pid}"
  line = pipe.gets      # pid from child
  puts "child says it's pid is: "+line
  Process.kill 'INT', sysint_gpid
  childs_last_word = pipe.gets
  if childs_last_word
    puts "child's last word: " + childs_last_word
  end
end
puts "child's exit code: #{$?.exitstatus}"

The trap handler could have been placed in the top-level scope but I put it inside the popen block to make it clear that scoping rules apply when defining a trap handler. Also, in a real application that wanted to ignore SYSINTs the handler would be defined as 'trap("INT") {}', like I showed above, but for the test application I wanted to see when the parent received the signal. Here's a run of the program from Windows:

Z:\work\ptest>ruby popen_sub.rb
parent pid: 200, popen return (child) pid: 1468
child says it's pid is: 376
parent pid: 200 Control-C
child's last word: sub pid 376 Control-C
child's exit code: 2

That was a Windows run but it works fine in Unix too. That's it, both standard output and standard error went through the parent process and the parent successfully sent a Control-C (SYSINT) signal to the subprocess. BTW: If you don't care about standard error then the same code could also be used without the redirection. That would be helpful if you need cross platform support and you have the same problems I had getting Process.kill to work with pipe.pid in Windows.

Other Options

If redirecting standard error doesn't meet your needs then popen3 and popen4 are alternatives to popen that provide separate access to the standard error stream.

Unfortunately standard library support for popen3 is limited to Unix and it doesn't provide an way to get to the subprocess pid so I don't see a way to send signals without opening up the method. There is a Win32 Utils win32-open3 library that provides popen3 support. I gave it a quick try but I couldn't get it to work. May be I installed the wrong version or since I had issues with all Windows Process.kill implementations using standard pids, may be I ran into some related user error but I didn't bother to look into it very much because the project I'm working on requires signal support, and I was concerned about the issues I saw reported when googling for 'popen3 ruby windows' and finally, because it's not included in the Win32 Utils gems that are distributed with the One-Click Ruby Installer for Windows, and I wasn't crazy about making it a separate prerequisite.

popen4's advantage over popen3 is that it provides access to the subprocess pid along with the three IO streams. I didn't use it instead of popen because not part of the Unix standard library and I thought the Windows installation requirements were too complex. So, I didn't want it as a dependancy for any application that I might end up distributing.

Installing Ruby 1.8.6, 1.8.7 and 1.9 on Leopard

2008-12-10T08:02:00Z

Dec. 19, 2008 Update: This post covers setting up and switching the default Ruby installation in /usr/local. If you're looking for a more dynamic way to test and/or switch between Ruby versions see Switching Between Ruby 1.8 & 1.9 with multiruby.

I had originally installed Ruby 1.8.6 on Tiger from source using Dan Benjamin's Building Ruby, Rails, Subversion, Mongrel, and MySQL on Mac OS X instructions. I upgraded to Leopard but continued to run the /usr/local version of Ruby. It has been working fine for me but for various reasons I decided it was time to upgrade to 1.8.7. I was happy to see Dan had created updated instructions for Leopard in his Installing Ruby, Rubygems, Rails, and Mongrel on Mac OS X 10.5 (Leopard) post. So I decided that I'd just install over my 1.8.6 version. It didn't go off without a hitch, I ran into a problem compiling readline. This post documents what worked to resolve the problem, my journey back to 1.8.6, how to add 1.9 support and how to switch between versions.

ruby-1.8.7-p72

My first step, and original goal, was to upgrade to 1.8.7. Since I already had 1.8.6 in /usr/local I already had RubyGems so I thought I just needed to follow the Ruby installation instructions. Here's a repeat of Dan's information about how to do that:

$ cd /usr/local/src
$ curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz
$ tar xzvf ruby-1.8.7-p72.tar.gz
$ cd ruby-1.8.7-p72
$ ./configure --enable-shared --enable-pthread CFLAGS=-D_XOPEN_SOURCE=1
$ make
$ sudo make install
$ cd ..

But the instructions were for a fresh Leopard install and I ran into the following problem during the make:

compiling readline
gcc -I. -I../.. -I../../. -I../.././ext/readline -DHAVE_READLINE_READLINE_H ...
... -D_XOPEN_SOURCE=1  -fno-common -pipe -fno-common   -c readline.c
readline.c: In function ‘filename_completion_proc_call’:
readline.c:703: error: ‘filename_completion_function’ undeclared
     (first use in this function)
readline.c:703: error: (Each undeclared identifier is reported only once
readline.c:703: error: for each function it appears in.)
readline.c:703: warning: assignment makes pointer from integer without a cast
readline.c: In function ‘username_completion_proc_call’:
readline.c:730: error: ‘username_completion_function’ undeclared 
     (first use in this function)
readline.c:730: warning: assignment makes pointer from integer without a cast
{standard input}:358:non-relocatable subtraction expression, 
     "_mReadline" minus "L00000000007$pb"
{standard input}:358:symbol: "_mReadline" can't be undefined in a 
      subtraction expression
...
{standard input}:249:non-relocatable subtraction expression, "_completion_proc"
       minus "L00000000003$pb"
{standard input}:249:symbol: "_completion_proc" can't be undefined in a 
        subtraction expression
make[1]: *** [readline.o] Error 1
make: *** [all] Error 1

I went through a couple of failed correction attempts based on posts from others before finding Angelbob’s Musings on Programming, Compiling Ruby 1.8.7 on a PowerBook G4 with Mac OS X 10.4 post. Bob's post explains that the issue relates to which version of readline you're using and Ruby not knowing whether functions have an "rl_" prepended to function names but that you can explicitly provide the information in Ruby's config.h. Since I'd messed with the 1.8.7 source directories I deleted the whole thing, extracted it again and then, after re-running the configure command, I added the following to config.h:

193
194
195

#define HAVE_RL_FILENAME_COMPLETION_FUNCTION 1
#define HAVE_RL_USERNAME_COMPLETION_FUNCTION 1
#define HAVE_RL_COMPLETION_MATCHES 1

This time the make and installed worked fine. I could have left it at that but since Dan's instructions would have worked with a fresh Leopard computer I decided to get rid of the /usr/local readline version that I had installed back when I was running Tiger. I didn't want to deal with non-standard (meaning non-native to Leopard) readline issues again. First I made sure I didn't have other /usr/local/ readline dependencies, I didn't; in my case readline 5.1 was only installed to get the orignal Ruby 1.8.6 environment set-up, so I nixed it:

$ cd /usr/local/src
$ cd readline-5.1 
$ sudo make uninstall

From there I started again over with Dan's Leopard instructions. This time everything worked without modifying the config.h.

1.8.6-p287

Unfortunately I couldn't stick with 1.8.7. Mephisto is using 2.0.2 and starting it in 1.8.7 results in this error:

undefined method `[]' for #

So, I moved back to Ruby 1.8.6. This time I got the latest version and dropped the use of the /usr/local readline:

$ cd /usr/local/src
$ curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p287.tar.gz
$ tar xzvf ruby-1.8.6-p287.tar.gz
$ cd ruby-1.8.6-p287
$ ./configure --enable-shared --enable-pthread CFLAGS=-D_XOPEN_SOURCE=1
$ make
$ sudo make install
$ sudo make install-doc
$ cd ..

1.9.1-preview2

Other than the 1.8.7 and Rails incompatibility everything was fine and, since switching between installations was now fresh in my mind, I decided it was finally time to try the latest 1.9 release:

$ cd /usr/local/src
$ curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-preview2.tar.gz
$ tar xzvf ruby-1.9.1-preview2.tar.gz
$ cd ruby-1.9.1-preview2
$ ./configure --enable-shared --enable-pthread CFLAGS=-D_XOPEN_SOURCE=1
$ make
$ sudo make install
$ cd ..

After I'd done a little testing going back to 1.8.6 was easy:

$ cd /usr/local/src/ruby-1.8.6-p287
$ sudo make install
$ sudo make install-doc

So, with the readline issue gone, I can switch my default installation by just changing to appropriate build and running "sudo make install" (and "sudo make install-doc" for pre 1.8.7 releases) whenever I want.

I should add a caveat at this point, it's generally been fine to install Ruby over old versions and I didn't notice any problems but there is a chance that a file from a previous install won't get overwritten and could cause a problem. Also, my goal was just to upgrade, it wasn't to come up with an optimal or clean way to switch between versions.

Adding Gmail SSL SMTP Support to Rails

2008-12-04T17:21:00Z

Jan. 13, 2009 Update: If you're using Ruby 1.8.7 see Updating Gmail SSL SMTP Support for Ruby 1.8.7 for a compatibility fix to the plugin used in this post.

DreamHost provides it's own SMTP support but their default set-up uses Gmail, which only supports SSL based SMTP connections. Rails 2.2 with Ruby 1.9 provides SSL support in ActionMailer via Ruby's 1.9 Net::SMTP class. That's great but it's still a very new combination and I wanted support for DreamHost's Ruby 1.8.5 and Mephisto using Rails 2.0.2. Fortunately it's simple to get SSL (a.k.a. TLS) based SMTP connections to work. Within a few minutes of hitting the browser search button I saw three options: lift 1.9 Net::SMTP code, use msmtp, and, the method I chose, a small plugin that patches Net::SMTP. No, I didn't look into the merits of each solution and I didn't do any type of exhaustive search to see what else might be available, I just picked what looked like an easy way to get the job done. This post documents where to find the plugin and how to use it.

Monkey patching can be a slippery slope but in this case the solution in Steven Chu's How to use GMail SMTP server to send emails in Rails ActionMailer post was so simple and transparent that I decided it was the way to go. Steven provided the code to extend ActionMailer to support TLS but not the plugin packaging so before doing that myself I did quick search on github and found that Eladio Caritos had the code in an action_mailer_tls git repository.

With Rails 2.1.0 or greater you can install the repository as a plugin:

[mac]$ script/plugin install git://github.com/caritos/action_mailer_tls.git

With older versions you have to install it manually:

[mac]$ git clone --depth 1 git://github.com/caritos/action_mailer_tls.git \
> vendor/plugins/action_mailer_tls
[mac]$ rm -rf vendor/plugins/action_mailer_tls/.git

The README in the plugin is based on Steven's post, which is a bit dated. The recommendation is to add SMTP configuration to config/environment.rb:

ActionMailer::Base.server_settings = {
  :address => "smtp.gmail.com",
  :port => 587,
  :domain => "mycompany.com",
  :authentication => :plain,
  :user_name => "username",
  :password => "password"
}

Way back in Rails 1.2.2 the config/initializers directory was added to replace this type of use of config/environment.rb and the server_settings method was dropped in favor of smtp_settings. So, I created a config/initializers/mail.rb file that looks like this:

ActionMailer::Base.smtp_settings = {
  :address => "smtp.gmail.com",
  :port => 587,
  # :domain => "robseaman.com", # HELO domain (not req'd)
  :authentication => :plain,
  :user_name => "blog@robseaman.com",
  :password => "mypass"
}

From an email standpoint that's all there was to it, Gmail support was complete.

Housekeeping

Since I'm using Cached Externals I also added an entry to my config/externals.yml file:

vendor/plugins/action_mailer_tls:
  :type: git
  # DH old git checkout (1.4.4.4) doesn't support -q
  :scm_verbose: true
  :repository: git://github.com/caritos/action_mailer_tls.git
  :revision: 0bc370bbfcbb6c25485c11c13a1662f1728d97e5

and ran:

[mac]$ cap local externals:setup

to cache the plugin. There's very little code in the plugin so I didn't cache it expecting a performance benefit; I just wanted to create an explicit record of where I got it and which revision I used.

Cached Externals

2008-12-03T20:08:00Z

Jan. 14, 2009 Update: The techniques discussed in this post are still relevant but the externals only apply to Mephisto 0.8. See Upgrading to Mephisto 0.8.1 for information about the externals that I'm currently caching.

Cached Externals is a Rails plugin that extends Capistrano to speed up deployments by not deploying vendor gems and plugins unless they change. The way it accomplishes the performance gain also results in explicit management of the cached gems and plugins, something worthwhile in it's own right. In this this post I'll walk you through installation and use of the plugin by describing how I used it for this Mephisto blog application. gitting Started with Mephisto describes how the application was set-up and Deploying Mephisto with Capistrano to DreamHost provides the pre-Cached Externals deployment details. Basically the app is just a customized copy of Mephisto in a git repository. The vendor directory holds Rails 2.0.2 and TZInfo along with all the gems and plugins that are part of the Mephisto distribution. The repository is kept locally on a laptop so Capistrano's copy strategy is used to deploy it to the production server.

Using Cached Externals

The first step is is to install Cached Externals. It's a Rails plugin on github so if your using any Rails version greater than 2.1.0 you can install it directly:

[mac]$ script/plugin install git://github.com/37signals/cached_externals.git

but since the Mephisto implementation is using Rails 2.0.2, I had to install it manually:

[mac]$ git clone --depth 1 git://github.com/37signals/cached_externals.git \
> vendor/plugins/cached_externals
[mac]$ rm -rf vendor/plugins/cached_externals/.git

Once the plugin is installed it's just a matter of following the instructions in the README.rdoc and deciding which externals to cache. Capistrano 2 is the minimum requirement, this blog is deployed using 2.5.2, so no problem there. The next step is to check to make sure this line is in the Capify file:

Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) }

It was in mine. If it's not in yours then you'll need to add the line or rerun capify .

To support an external gem or plugin you need to let Cached Externals know about it by creating an entry for it in a config/externals.yml file. The gem or plugin needs to be in an external repository that's accessible locally and on the deployment target. The YAML entry for each external contains information about the type of repository, where it is and which revision is required.

If you know the repository and revision information for your gems and plugins then creating the YAML file should be easy. If you don't have or don't want to list all of the information then start with the big externals. It's a good idea to have an explicit list of this information for all of your gems and plugins but you won't see nearly as much of a performance benefit from the smaller externals. For my Mephisto application I'd installed the Rails and the TZInfo gems myself but the other gems and plugins came bundled as part of Mephisto so I didn't have any of the revision information and, since I'm only merging in the Mephisto master branch and not maintaining the application details, I was only interested in performance gains, not the management benefit. Digging up data for all the gems and plugins would be more effort than it was worth, especially for some of those unversioned plugins. So, instead of trying to cache everything, I did a check for low hanging fruit:

[mac]$ cd vendor
[mac]$ du -sk *
[mac]$ cd plugins
[mac]$ du -sk *
[mac]$ cd ../..

Of course Rails was the monster at about 9.8M, TZInfo was 3.8M, and the other 4 gems only totaled about 700K with the largest being 336K. All 25 plugins totaled 4.8M with RSpec weighing in at 2.1M, RSpec on Rails about 600K and the next largest plugin was only about 300K. So if I only cached 4 of 31 externals: Rails, TZInfo, RSpec and RSpec on Rails, I could cover 16.3 of 19.1 MB.

After finding the repositories on github and RubyForge (I'll describe how I found the exact revision information later) I created this config/externals.yml file:

vendor/rails:  # v2.0.2
  :type: git
  # DH old git checkout (1.4.4.4) doesn't support -q
  :scm_verbose: true
  :repository: git://github.com/rails/rails.git
  :revision: e454cf721ea2deca1b1b991339a5d30161dcfb9c
vendor/tzinfo-0.3.12:
  :type: subversion
  :repository: http://tzinfo.rubyforge.org/svn/trunk/tzinfo
  :revision: "210"
vendor/plugins/rspec: # pre 1.1.0
  :type: subversion
  :repository: http://rspec.rubyforge.org/svn/trunk/rspec
  :revision: "2962"
vendor/plugins/rspec_on_rails: # pre 1.1.0
  :type: subversion
  :repository: http://rspec.rubyforge.org/svn/trunk/rspec_on_rails
  :revision: "2962"

The file assumes that Subversion and git are installed on both the local deployment PC and on the deployment target. If you aren't sure if you have the correct source code control software on the client and target then check for it. For example, I used git --version and svn --version on my hosting service to see what was installed. If you're missing an SCM that you need then either install it, delete the appropriate entries in your externals.yml or move the gem or plugin to another external repository that you can support.

Here are some additional notes about the YAML file:

Cached Externals creates a new SCM instance for each repository. So any SCM option variables that are normally set in deploy.rb need to be set for each cached item.
In my case :scm_verbose had to be set because my DreamHost server has an old verion of git (1.4.4.4) and the -q (--quiet) flag that's used in git-checkout when :scm_verbose isn't set won't work with old versions of git.
I used the SHA1 code for the Rails revision but it's a tagged release so v2.0.2 could have been used instead.
I always use the http:// link for RubyForge subversion repositories. The svn:// transport to RubyForge isn't reliable for me but the http:// transport works fine. I've never investigated why but I do know it's a common issue.

The next step is to remove externals referenced in the externals.yml from the vendor directory. If your following along then at this point I'd recommend you're prepared for any problems by having a clean repository (everything's committed or stashed) or by making a backup copy of the vendor items before doing this:

[mac]$ rm -rf vendor/rails
[mac]$ rm -rf vendor/tzinfo-0.3.12
[mac]$ rm -rf vendor/plugins/rspec
[mac]$ rm -rf vendor/plugins/rspec_on_rails

Somehow that feels uncomfortable, but no worries, the files will get replaced when the local cached copies are retrieved. Well, actually the files will be placed in a ../shared/externals/vendor directory and symlinks to them will be created in the proper locations in the vendor directory. Yes, the shared directory will be at the same level as the application directory. Here's how to load up the cache:

[mac]$ cap local externals:setup

That will result in a big retrieval so after it's done it's a good time to make sure everything looks correct with git status. Here's what I saw with git status after loading the local cache using the externals.yml file from above:

[mac]$ show git status
# On branch robseaman
# Changed but not updated:
#   (use "git add/rm ..." to update what will be committed)
#
#	deleted:    vendor/rails/REVISION_5fa0457542b0ff541d0a80ff8c3561eec8e35959
#	modified:   vendor/tzinfo-0.3.12/lib/tzinfo/ruby_core_support.rb
#	modified:   vendor/tzinfo-0.3.12/test/tc_ruby_core_support.rb
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#	vendor/plugins/rspec
#	vendor/plugins/rspec_on_rails
#	vendor/rails
#	vendor/tzinfo-0.3.12
no changes added to commit (use "git add" and/or "git commit -a")

The untracked files were the symlinks, no problem. The deleted rails/REVISION_5fa045... was just an empty touch file that was created when Rails was frozen into the vendor directory. It doesn't indicate that the 5fa04575 revision should have used in the externals.yml file. The 5fa04575... revision number is the SHA1 of the latest commit on github at the time when Rails was downloaded. There were no other issues in the Rails directory so it looked good. Ah, but the TZinfo files looked like trouble, so I used git-diff to find out:

[mac]$ git diff
diff --git a/vendor/rails/REVISION_5fa0457542b0ff541d0a80ff8c3561eec8e35959 \
           b/vendor/rails/REVISION_5fa0457542b0ff541d0a80ff8c3561eec8e35959
deleted file mode 100644
index e69de29..0000000
diff --git a/vendor/tzinfo-0.3.12/lib/tzinfo/ruby_core_support.rb \
           b/vendor/tzinfo-0.3.12/lib/tzinfo/ruby_core_support.rb
old mode 100644
new mode 100755
diff --git a/vendor/tzinfo-0.3.12/test/tc_ruby_core_support.rb \
           b/vendor/tzinfo-0.3.12/test/tc_ruby_core_support.rb
old mode 100644
new mode 100755

Hmm, the only difference was the addition of execute permission on the files. I'm not sure why the settings were different but it didn't seem like a big deal to me so I considered it good.

I was waiting to discuss how I found external revision numbers until I covered cap local externals:setup, git status and git diff because they were a big part of my search. I started by picking likely candidates. For Rails this was just a matter of using the tags drop-down on github and selecting the revision for v2.0.2. For TZInfo I used svnX to browse the repository on RubyForge and the 0.3.12 release comments were near the top. For the RSpec plugins I examined CHANGES file in Mephisto's vendor copy which led me to 1.1.0. I found 1.1.0 and other old versions of RSpec and RSpec on Rails in a Subversion repository on RubyForge, so I used svnX again and plugged the 1.1.0 release revision in externals.yml. Then I ran cap local externals:setup and used a combination of git-status and git-diff to see what was different. Rails and TZInfo matched but the RSpec plugins didn't. There were enough differences that I ran git-diff on one file at time, trying to pick the file most likely to help me find the correct version. From the diff of the CHANGES file I could tell that Mephisto was using a pre-1.1.0 edge version. Looking at checkin history with svnX gave me a pretty good idea how far back to go, so I trashed the ../shared/externals/vendor/plugins directory, re-ran cap local externals:setup and checked the differences again. The CHANGES file didn't have every detail so I ended up having to do this a few times, narrowing it down each time until I found a match.

Given that I was setting this up for an application that I knew little about, I was lucky. If I'd have wanted to do this for a gem or plugin that had been customized I would have had to put the custom version in an external repository like github or left it out of the cache. Also, all of the externals I looked for were on github or RubyForge, were well tagged and were good about documenting version information in their distribution. Searching for the RSpec revision gave me an clue about how much of a pain this could have been if good tagging and release documentation practices hadn't been followed. The lesson here is to not wait until final deployment before you start to keep track of externals; do it as the externals are installed.

After you're satisfied with your shared cache you'll want to commit the results to source control. For me that meant:

[mac]$ git add .
[mac]$ git commit -a -m "Cached Externals: rails, tzinfo & rspec/rspec_rails"

cap local externals:setup will need to be run anytime externals.yml is updated. And, if you're using git, there's one last related housekeeping item. Cached Externals includes post-checkout and post-merge git hooks to automatically makes sure the correct symlinks are used. They work by running cap local externals:setup when switching between branches, pulling and merging. Before installing the hooks check (using ls .git/hooks) to make sure that you don't already have your own versions of the hooks. I didn't have either so I used the Cached Externals rake task to install the hooks:

[mac]$ rake git:hooks:install

If you already have one or both of the hooks then you'll need to merge in the Cached Externals versions from the plugin's script/git-hooks directory or decide whether you want your old hooks or the Cached Externals hooks.

At this point I did a git-tag to mark the application as a release and then did a new deployment:

$ cap deploy

It's not mandatory to deploy right after setting up Cached Externals but I wanted to isolate it as a separate release in case anything went wrong. The first deployment was slower because it has to clone the external repositories. I also noticed false error reporting on the git status messages. You'll probably see the same thing; there will be a lot of messages that look like this:

...
*** [err :: blog.robseaman.com] 11% (186/1690) done
*** [err :: blog.robseaman.com] 12% (203/1690) done
*** [err :: blog.robseaman.com] 13% (220/1690) done
...

It looks bad but it's not a problem. It just indicates git must be using the error stream for status reporting. Of course it's a good idea to do some verification that the symlinks and directories look correct after the deployment. When I checked my results everything was fine.

After the first run things should be fast until one or more externals need updating. For the Mephisto blog my deployment gzip file went from 3.2 MB to under 1 MB with what seemed to be a corresponding time savings.

Additional Information

Cached External's main purpose is to speed up deployments but the way it accomplishes this inherently helps you track and share the externals that you decide to cache. Externals management is a category where there are already several solutions. Piston, Braid, the subtree merge strategy, git submodules and svn:externals are some of the more popular ways to manage the vendor externals. Gary Lam has put together a github repository for his Dependancy Management Talk with a Keynote presentation, example implementations and notes that cover Cached Externals, the solutions I mentioned and others.

Gary's repository is a great resource and this note in his notes.markdown file caught my attention:

* script/generate has issues with symlinks?  
  did not detect generators from symlinked plugins

When I first read Gary's note I did try the rspec_model generator script from the rspec_on_rails plugin after the symlinks were in place and it worked fine. After looking into it more I found out that the problem Gary refers to is an issue with some versions of Rails. It looks like the problem was fixed in February 2007, inadvertently broken as part of another fix in March 2008 and finally fixed again in June, 2008. Going by dates it would mean generators in symlinked plugins worked from Rails 1.2.3 to 2.0.2, were broken in Rails 2.0.3 and have worked fine since 2.1.1. The Mephisto blog uses Rails 2.0.2, so it's not surprising that I didn't see the issue. If you run into this problem then you'll need to update your Rails version, eliminate the affected plugin from the cache or temporarily replace the symlink with the real thing when you use a generator.

Jamis Buck, the author of Cached Externals, discussed the importance of deployment performance and how Cached Externals is different than other Capistrano caching techniques in his CachedExternals: managing application dependencies post on the Signal vs. Noise blog. Finally, I should mention that I found out about Cached External from a Rails Envy podcast.

Production Data to Development

2008-12-02T01:01:00Z

It's often helpful to be able to bring MySQL production data down to a development machine. I wanted to do this for my Mephisto blog engine (see gitting Started with Mephisto and Deploying Mephisto with Capistrano to DreamHost) so that I could see how changes would look with existing data before deploying an update. Production data is also useful for tracking down bugs with production data and for backups. This post describes the Capistrano namespace I created for doing this.

Before writing my own code I googled and found that Peter Harkins had posted Capistrano Task to Load Production Data with an implementation that did basically what I wanted. I also found a Joyent CodeSnippets post from Jamie Wilkinson that was based on Peter's code but added support for Capistrano 2.0 and gzip before doing the download. I didn't need the gzip support so I started with Peter's code. I'm using Capistrano 2.5.2 and I added additional code to include downloading files in the public/assets directory, pull the production database connection information from the server (instead of the development database.yml), get the password off of the mysqldump command line, split the download from the load to provide cached loading and backup functionality, and use rsync delta transfers to speed up the transfers.

With those changes I ended up with a larger (but still not big) implementation with three tasks in a namespace called data. You could load this from a separate recipe file but I just added it to the end of my deploy.rb file. Here's the namespace:

# Derived from http://push.cx/2007/capistrano-task-to-load-production-data
namespace :data do
  require 'yaml'
  set :sql_dump_file, "dump.#{application}.sql"
  
  desc <<-DESC
    Get production data and load it into development. A new copy of the
    production database and production public/assets are cached and
    loaded into the development environment.
  DESC
  task :load_from_prod do
    cache_prod_data
    load_from_cache
  end
  
  desc <<-DESC
    Get production and cache it locally. The production database and 
    production public/assets directory are downloaded and cached.
  DESC
  task :cache_prod_data, :roles => :db, :only => { :primary => true } do
    on_rollback { run "rm #{current_path}/tmp/#{sql_dump_file}" }

    require 'yaml'

    database_yml = ""
    run "cat #{current_path}/config/database.yml" do |_, _, database_yml| end
    config = YAML::load(database_yml)['production']
    
    run "mysqldump #{mysql_options(config, true)}"+
        " > #{current_path}/tmp/#{sql_dump_file}" do |ch, _, out| 
      if out =~ /^Enter password: /
        ch.send_data "#{config['password']}\n"
      else
        puts out 
      end
    end
    
    FileUtils.mkdir_p("tmp/data/public/assets")

    logger.debug "rsyncing #{sql_dump_file} from #{application}"
    system "rsync -lrp "+
      "#{user}@#{application}:#{current_path}/tmp/#{sql_dump_file} tmp/data"
    run "rm #{current_path}/tmp/#{sql_dump_file}"
    
    logger.debug "rsyncing assets from #{application}"
    system "rsync -lrp --delete "+
             "#{user}@#{application}:#{shared_path}/assets tmp/data/public"
  end

  desc <<-DESC
    (Re)load cached production data into development. Use
    data:cache_prod_data or data:load_from_prod to create the cache.
  DESC
  task :load_from_cache do
    config = YAML::load_file('config/database.yml')['development']
    if File.exist?("tmp/data/#{sql_dump_file}") 
      mysql_load = "mysql #{mysql_options(config)} < tmp/data/#{sql_dump_file}"
      logger.debug %(executing "#{mysql_load.sub(/-p\S+/, '-px')}")
      system mysql_load 
      FileUtils.rm_rf("public/assets/*")
      FileUtils.cp_r("tmp/data/public/assets", "public")
    else
      abort "The data cache is empty, try 'cap data:load_from_prod'"
    end
  end
  
  # Return MySQL options for a specific configuration.
  def mysql_options(config, prompt_for_password=false)
    if config['password'] 
      password_opt = prompt_for_password ? " -p" : " -p#{config['password']}" 
    else
      password_opt = ""
    end
    "-u #{config['username']} -h #{config['host']} #{config['database']}"+ 
    password_opt
  end
  
end

And here's a breakdown of the tasks and their features:

cap data:cache_prod_data dumps and downloads the latest copy of the production database and the public/assets directory. The database configuration is pulled from the database.yml on the server so no account information is needed on the local machine. The MySQL password is kept out of the command line so it's left out of the Capistrano log and out of the process list (see Hiding MySQL Passwords with Capistrano). The download uses rsync to a cache in the local tmp/data directory. This is intended to make the transfer fast and eliminate the need for a creating new dump when reloading a previous dump is good enough. data:cache_prod_data is used as a step in data:load_from_prod but is also useful as standalone task for getting a backup copy of the database. It could be used in a crontab as is with backup software like Time Machine taking care of versioning. Alternatively, with some additional software you could create time stamped compressed versions of the tmp/data directory after download.
cap data:load_from_cache will reload a the last version of the production data, printing an error if the data doesn't exist in the local cache. This eliminates the dump and transfer when you're fine with just restarting with a fresh copy of the last downloaded version of production data. One scenario is that you've mucking with production data in the development environment and want to reload a fresh copy. Another is that you're doing regular back-ups using data:cache_prod_data during off hours and the cache contains current enough information for your development testing.
cap data:load_from_prod downloads the current production data into the cache and development environment by combining data:cache_prod_data and data:load_from_cache. If you always want the latest copy of production data you can use this task and forget about the other two.

Limitations

I'm sure there are lot more than this but the obvious ones are:

The code specific to MySQL and specific to moving from production to development.
It does a complete database dump. I think the rsync delta transfer and caching are sweet but if your database is quite big then some form of delta dumping and loading of the database will also be needed. Perhaps tracking changes since the last backup using some type of journaling or a similar technology along with smarter dumping.
The entire production database is made available on a development machine. That's fine for many applications but if you have privacy concerns about the contents of the production database then copying the production database to a development machine isn't a good idea and in some cases, such as the need to comply with HIPPA regulations, it may even be a legal violation. In those situations you'll need to be smarter on the server side and either obfuscate or limit the what data is transfered.
MySQL vulnerabilities to process list visibility are handled on the production server but not on the development client. Hiding MySQL Passwords with Capistrano describes the scope of the issue and suggestions for possible solutions if this is a problem in your environment.

Hiding MySQL Passwords with Capistrano

2008-12-01T22:16:00Z

I was using Capistrano 2.5.2 to run a MySQL dump on a deployment target and what started as a simple implementation to prevent the command line password from showing up in the Capistrano log evolved into a research effort about the security of using passwords on the MySQL command line. This post describes how to suppress run execution lines from showing up in the Capistrano log, the real risks with using passwords on the MySQL command line and how to use Capistrano to avoid command line passwords without putting the password in a ~/.my.cnf file.

I'll start from the beginning ... the Capistrano logger level is set to DEBUG by default and that caused MySQL command line passwords to be displayed/logged whenever MySQL was invoked via run. I didn't like that so I came up with a very direct solution to hide the password and leave everything else alone. Here's the code segment from a Capistrano task to do that:

require 'yaml'

database_yml = ""
run "cat #{current_path}/config/database.yml" do |_, _, database_yml| end
config = YAML::load(database_yml)['production']

mysql_dump = "mysqldump -u #{config['username']} -p#{config['password']} "+
  "-h #{config['host']} #{config['database']} > #{current_path}/tmp/dump.sql"

# surpress debug log output to hide the password
current_logger_level = self.logger.level
if current_logger_level >= Capistrano::Logger::DEBUG
  logger.debug %(executing "#{mysql_dump.sub(/-p\S+/, '-px')}")
  self.logger.level = Capistrano::Logger::INFO 
end

run mysql_dump do |ch, _, out| 
  puts out 
end

# restore logger level
self.logger.level = current_logger_level

It worked and I was satisfied for a while but a FIXME comment in a Joyent CodeSnippets post with similar code started bothering me:

2223	# FIXME pass shows up in process list, # do not use in shared hosting!!! Use a .my.cnf instead

I knew, or I thought I knew, that MySQL took care of hiding passwords on the process list but I decided to verify how they did it. I found a lot of posts from from people claiming there was no protection at all and several others that like me thought MySQL had the issue resolved. Eventually I looked at the C source and found some good posts. Here's what I found:

There's a window of time when MySQL command line passwords are visible on the process list. That's not to say that you can always just grab MySQL command line passwords with ps -ef as many posts claim. Despite posts to the contrary, MySQL is smart enough to try to hide command line passwords and, for the most popular Unix systems, this works. In general the window of vulnerability only exists between the time a MySQL tool is launched and the time the MySQL code does the hiding (see here). Since it processes command line arguments very early, it's a small window. I read claims that you could put ps in a loop to exploit the window and find passwords. I never saw any proof that something so simplistic would work and given the frequency of command execution and the small window size I have my doubts. On the other hand, I'm sure it's possible to come up with some programatic way to exploit the window and then there's always the chance that someone could have lucky timing with a single ps -ef.

The hiding is done by modifying the password argv like this: while (*argument) *argument++= 'x'; then if it's the last argument it gets truncated to 0 length. From my search on the MySQL 5.1.30 source this hiding technique appears to be done for all commands, it's definitely done for mysql, mysqldump and mysqladmin. I already mentioned the window of vulnerability but there are two additional issues: First, unless the password is the last command line argument, its length can be determined by counting the x's. I didn't look at source for older versions but the x'ing out seems to have been in place for a very long time. On a 4.0.23 implementation that I have on phpwebhosting.com the x'ing out worked but the truncation didn't, from this post it appears to have been fixed in 4.1.

The bigger issue is that the method won't work for all Unix systems. It seems to be fine for some systems like Linux and Darwin (OS X) but not for others like Slackware that don't overwrite the process list entries when argv's are changed. The theory is that it works for the BSD-ish Unix versions and not for the SysV-ish flavors. May be, but during my search I did see what looked like a credible report that one user could see passwords after moving from Linux to FreeBSD. I'm curious about that but not so curious that I'm willing to create a FreeBSD VM with MySQL to check it out. It's easy to verify whether or not password hiding works on your system by starting mysql and then opening up another process and running ps -ef. On most Unix system you'll see the password x'd out as in -px xxxxxxxx or just -px. -px will show up alone if the -p option is the last command line argument.

In many environments the issues I mentioned just aren't real problems. Even some shared hosting environments manage to keep the process list private, this is true on DreamHost for example, so it's only an issue within your own accounts. OTH in other environments it is a big deal. Even if the password hiding works, if your processes show up when someone else does a ps -ef at just the right time they'll get your password. Now, I'm not trying to say that the MySQL password hiding is bogus. Security is almost always a matter of degree and the technique used protects passwords far better then doing nothing. That said, it seems wise not to use a command line password in a shared environment if it can be avoided and fortunately that's easy.

Avoiding the MySQL --password Option with Capistrano

The general recommendation I found to avoid the --password option is in line with the comment from the Joyent CodeSnippets post, put the password in ~/.my.cnf. That allows you to avoid command line passwords with any scripting tool including Capistrano. The downside is it means another configuration step to setup ~/.my.cnf and you have a plaintext password in an additional file (it's already in database.yml for Rails). In my Capistrano script I'm already loading the production server's database.yml so I decided to handle it in the run block by interactively responding to MySQL's password prompt. Here's the code fragment:

require 'yaml'

database_yml = ""
run "cat #{current_path}/config/database.yml" do |_, _, database_yml| end
config = YAML::load(database_yml)['production']

run "mysqldump -u #{config['username']} -p -h #{config['host']} "+
  "#{config['database']} > #{current_path}/tmp/dump.sql" do |ch, _, out| 
  if out =~ /^Enter password: /
    ch.send_data "#{config['password']}\n"
  else
    puts out 
  end
end

Since we're using a SSH channel the password is protected and as a bonus the code for suppressing log output is no longer needed making the code segment shorter and cleaner. If you want to see the task that actually uses this snippet then take a look at the the data:cache_prod_data task in the Production Data to Development post.

Limitations

The run code block solution is for executing on a remote server. It doesn't address executing MySQL commands from the local deployment client. Before trying to solve it you should consider whether or not you really need to. There's no logging issue when executing local commands and the process list security limitations aren't as likely to be a problem; they weren't for me. But, if this is something you need to or want to take care of then you'll need to use ~/.my.cnf or investigate ways to respond to the MySQL password prompt. Handling the prompt would mean leaving the simple Ruby subprocess methods like %x, `, exec and system behind. PTY.spawn and expect would probably work and look very similar to the run code block but pty is Unix specific. You may also be able to get popen to work but I'm not sure how you'd handle the prompting with popen.

Deploying Mephisto with Capistrano to DreamHost

2008-12-01T07:32:00Z

This post describes how to use Capistrano to deploy a Mephisto based blogging engine to a DreamHost account running Phusion Passenger. The blog code is deployed from a local git repository. gitting Started with Mephisto describes how the repository was set-up. With some modifications the script and commands presented here should be adaptable to other hosting services and other Phusion based Rails applications.

This was my first use of Capistrano. To get a foundation I watched the PeepCode, Capistrano 2 screencast, read the From the Beginning page at the Capistrano site, and watched the Capistrano Tasks Railscast. All of those were good resources, the Railscast episode from Ryan Bates was particularly applicable due to Ryan's use of git and coverage of assets. The best coming up to speed tip that I can provide is to open up the gem source code and use it as a reference. Pay particular attention to the lib/capistrano/recipes/deploy.rb file. It contains the default tasks, contains a large chunk of the built-in variables and makes the deployment execution order clear.

Server Prerequisites

If you haven't set up a user account with SSH access that Capistrano can use for deployment then do it now. Any existing user account will do but one minor point that may affect the username you choose is that your web directory will eventually reside somewhere under one of your users home path, i.e. it'll end up some place like /home/username/mywebapp.com/current/public. I created a user called rsblog to house and deploy my blog application. Setting up a separate deployment user would also be a reasonable choice but DreamHost doesn't provide superuser (sudo) access. So if you go that way you'll need to configure group permissions to allow you to deploy files to the application user's directory tree.

Capistrano supports a :password variable that will allow you to put a password in your script but it's more secure to setup a public key instead. If you haven't done that then DreamHost SSH Access with Leopard.html walks you through the setup if you have Mac Leopard. For other systems you can start with the DreamHost SSH Wiki.

You'll also need to create a database for Mephisto on the DreamHost server. To do that open up your control panel to the Goodies > Manage MySQL section and create one using the Create a new MySQL database: section. The database name has to be unique so the default name in the database.yml file, mephisto_production, probably won't work. A common approach is to prefix the name you want with your domain name, for example, I used robseaman_mephisto_production. When you create the database it's a good idea to create a new user that only has database access rather than using a shell account. You can do this by selecting Create a new user now... from the First User: drop down in the Create a new MySQL database: section. The concept is that separate accounts will limit your vulnerability if one of your accounts is compromised (you know, stolen, hacked into, etc...). In general creating separate accounts for different roles with each account having just enough permissions to do its job is a common and solid practice. That (along with plagiarism) is the basis for why there's a separate deploy user account in so many of the Capistrano examples. If this is a new concept to you and you start applying it then be sure to use some common sense in your role breakdowns, too many accounts create administration headaches. Ah, but I digress ... back to the subject.

Capistrano Setup

The current version of Capistrano is 2.5.2 as I'm writing this. Upgrade if you have an older version. If you haven't installed Capistrano yet then the gem is installed in the usual way:

[mac]$ sudo gem install capistrano

Great, easy as Ruby, now it's time to "capify" the application. This will create the Capfile and config/deploy.rb files and keep our repository clean:

[mac]$ cd mephisto
[mac]$ capify .
[mac]$ git add .
[mac]$ git commit -a -m "Initial versions from capify."

Capfile doesn't need any changes. config/deploy.rb needs extensive overhauling. There's a config/deploy.example.rb that Mephisto provides but I didn't use it and if you're following along and using the same or similar environment then I think you'll end up with a fewer modifications if you just use mine. I'll show it in chunks so I can describe its contents. Here's part one:

# Settings that change from app to app
set :user, "rsblog"
set :application, "blog.robseaman.com"
set :scm, :git
set :branch, "robseaman"  # "master" would be more typical
set :repository, "file:///Users/Rob/work/mephisto/.git"
set :deploy_via, :copy
set :git_shallow_clone, 1
set :copy_exclude, [".git", ".gitignore"]
set :deploy_to, "/home/#{user}/#{application}"

You'll need to change some of this first part of deploy.com to your own settings. Here's an overview: The user² is your SSH user account that you have setup to do the deployment. The application³ is the domain name. I created a separate git ⁴ branch⁵ from master to keep my changes separate from the open source version of Mephisto. Replace it with whatever branch you want to deploy. Since I'm using a local file repository⁶ from my laptop I chose the copy⁷ deployment strategy. :git_shallow_clone ⁸sets the --depth to 1 to make the clone faster, we're not passing the git repository anyway. Eliminating the .git/.gitignore⁹ on the server results in a significant reduction in compression time. With this Mephisto repository it takes us from a 8.9 MB transfer down to 3.2 MB. My choice of deploy_to¹⁰ follows DreamHost defaults but if you created a separate user for deploying then you'll need to change #{user}¹⁰ to the name of DreamHost user that you want to contain the application. You can also change the #{application}¹⁰ part of the path to something different, i.e. mephisto or whatever you want. You could also set :copy_cache to true; I tried it but I didn't notice a significant difference either way so I left it out.

# App specific tasks:
namespace :deploy do
  desc <<-DESC
    Bootstrap the Mephisto production database.  Runs the Mephisto \
    db:bootstrap rake task for the production database.  Normally run \
    one time as part of the initial deployment:
    
      $ cap deploy:setup
      $ cap deploy:update
      $ cap deploy:bootstrap
      
    WARNING! This task will reinitialize your database.  If you run \
    it after the initial deployment all of your content will be lost.
  DESC
  task :bootstrap, :roles => [:db], :only => { :primary => true } do
    run "cd #{current_path}; rake RAILS_ENV=production db:bootstrap"
  end
end

I decided to create a separate section (starting a line 12 above) at the top of my deploy.rb file for application specific tasks. In this case I added code to run Mephisto's db:bootstrap task on the production server. The task description in lines 14-25 says it all. If you don't want a task to do the bootstrap from the client than you can leave this code out; it's no problem to ssh to the server and run rake db:bootstrap when you need to. The task is something I added early on while I was just figuring out how things worked and saw several implementations with the equivalent functionality. It saves a little typing but it's rarely done and the manual steps are so simple it doesn't really provide that much of a benefit.

# The rest is generic for DH a shared hosting deployment, assumming:
#   - Phusion Passenger is used
#   - An assets dir in public that's not part of the scm repository.
#   - The database.yml is not in the repository 
set :use_sudo, false

role :app, application
role :web, application
role :db,  application, :primary => true

Lines 31-40 contain pretty standard stuff. We can't use_sudo³⁵ on DreamHost and the roles^37-39 are all on the same server.

namespace :deploy do
  desc "Restart passenger."
  task :restart do
    run "touch #{current_path}/tmp/restart.txt"
  end

  desc "Initial setup of the shared_path."
  task :setup_shared do
    run "mkdir -p #{shared_path}/config #{shared_path}/assets"
  end

  desc "Gets the shared_path ready for use with the release."
  task :prepare_shared do
    run "ln -nfs #{shared_path}/assets #{release_path}/public/assets"
    run "ln -nfs #{shared_path}/config/database.yml "+
              "#{release_path}/config/database.yml"
  end
end
after 'deploy:setup', 'deploy:setup_shared'
before 'deploy:finalize_update', 'deploy:prepare_shared'

Lines 41-60 contain the deploy namespace tasks. Phusion Passenger is restarted^42-45 in the usual way. :setup_shared^47-50 is intended to create shared directories or files needed as part of the initial setup. In this case that's only the assets directory⁴⁹, which holds article assets so it's left out of the repository and a single copy is kept and shared between releases. It needs to exist on initial deployment when it's symlinked⁵⁴ for the first time. :prepare_shared^52-57 is intended for any setup required to prepare the shared path for use. In this case that just means symlink creation. Tasks like :prepare_shared are typically done after deploy:update_code. I chose to do it before deploy:finalize_update⁶⁰ instead of the usual time after reading a comment in Cached Externals about touching order and looking at the :finalize_update task code. Doing it right before :finalize_update will allow :normalize_asset_timestamps to work properly with anything that needs timestamping in the shared path. Although it won't matter in this deployment, doing it before :finalize_update seems like the correct execution order and I expect to reuse this code in other projects where it could make a difference. Because the database.yml file contains the production password, I opted to only keep it on the production server. I'll give instructions about how to create it before it's linked⁵⁵ to later, but if you'd rather just keep your production information in your local database.yml and copy it up as part of the deployment then insert:

50	top.upload("config/database.yml", "#{shared_path}/config/database.yml")

into :setup_shared. This will upload the file whenever you run cap deploy:setup, which is safe to run anytime. Alternatively, you could add it to the beginning of :prepare_shared to upload it with each deployment.

That's enough to give us a complete deployment script so it's a good time to commit the changes. And, since were about to deploying a release to production, I'd also recommend tagging it with a version or some other type label:

[mac]$ git commit -a -m "First customized version of deploy.rb"
[mac]$ git tag -m "First production blog release." v0.0.1

Initial Deployment

Originally cap deploy:cold was intended to be a single step initial deployment. It still exists but it is no longer recommended. It was recognized that generically automating initial deployments for web applications just isn't something that works well. For example, this blog deployment is pretty simple but db:bootstrap is used for database and server setup, migrations aren't needed, and I've made a security decision not to automate deployment of the database.yml. At the very least deploy:cold would need to be rewritten to get it to work. It might be worth exploring whether or not you can get deploy:cold to work if you have some type of cookie cutter deployment but it's not worth it if the deployment environment varies or for the one time setup of a web application. As it turns out even without cap deploy:cold Capistrano makes it easy for a first time deploy. The first step is to prepare the directory structure on the server:

[mac]$ cap deploy:setup

I chose to only keep separate database.yml files on the client and server in order to keep database account settings off the deployment client. Unless you added the top.upload() code in :setup_shared or you did something similar to get your database.yml file to the server then we'll need to create the database.yml on the server. It doesn't matter how you get it there, you can edit a yaml file locally and put it on the server then delete it, create it on the server or whatever. I scp'd the Mephisto example database yaml file and edited on the server with vim :

[mac]$ scp config/database.example.yml \
> rsblog@robseaman.com:blog.robseaman.com/shared/config/database.yml
[mac]$ ssh rsblog@robseaman.com
[dh]$ cd blog.robseaman.com/shared/config
[dh]$ vim database.yml # edit production settings and save
[dh]$ logout

Your production section contents will vary but my DreamHost version is close to this:

production:
  adapter: mysql
  database: robseaman_mephisto_production
  username: robsblogdbuser
  password: YeahSure
  host:     mysql.robseaman.com
  encoding: utf8

BTW: If you did decide to put the top.upload() code in the :setup_shared deployment task but ran cap deploy:setup before updating the production section of your database.yml then just run cap deploy:setup again after the database.yml update is complete.

Now move the application to the server with:

[mac]$ cap deploy:update

At this point the database exists but lacks a schema so it's time for our Mephisto specific task. Run the task for that:

[mac]$ cap deploy:bootstrap

If you see a line that looks like this:

*** [err :: robseaman.com] mkdir -p /home/rsblog/xxxx/releases/20081129044823/log

don't worry about it. It's occurs because we already created the assets directory. If you didn't add the deploy:bootstrap to deploy.rb then just ssh to the server, change to the current directory and run rake db:bootstrap. For other applications this would be the point where you'd usually run rake RAILS_ENV=production db:schema:load.

Now the web application needs to be started. For DreamHost this means changing the hosting settings for the domain to use Phusion Passenger and to point to the application. Go to Domain > Manage Domains in your control panel and select Edit in the Web Hosting column for the domain. Set the FTP user / CGI-runs-as user to your application user, for me that was rsblog. The web directory should default to the domain name, for me it was /home/username/blog.robseaman.com. If you used the same :deploy_to variable in deploy.rb as me just add /current/public, for example I ended up with /home/username/blog.robseaman.com/current/public. Now check the box next to Ruby on Rails Passenger (mod_rails)?. If you want other changes go ahead and make them. When you're done press the Change fully hosting settings now! button.

Give it a minute or two and you should be able to pint your browser to http://your_domain/admin, i.e. http://blog.robseaman.com/admin for me. Login with Login: admin and Password: test, change the password, and change the site settings. I did notice at this point that I needed to go to Settings > Caches (admin/cached_pages) and clear the cache to see my changes but I'm not yet sure if that's always necessary. Now you should be able to start blogging.

Subsequent Deploys

Subsequent deployments is where Capistrano really shines. When you want to make a change you can do it in your development environment and after you make sure it works and git commit it you can move it to your web server and restart the application with a single command:

[mac]$ cap deploy

It's probably a good idea to add a git tag step, like we did before the initial deploy above, before each deploy but that's up to you. If you can't remember what version you used last, just by type git tag with no options and you'll get a list.

Is that it?

At this point we have a working maintainable blogging system. The blog software and its templates can be tracked and updated and it's easy to deploy changes from a development machine. From a workflow standpoint it would be very helpful if it were easy to copy the production database into the development environment to see what impact changes will have on existing content before an update is deployed. I'll cover that in the Production Data to Development post. Once that's in place I'd consider it a functionally complete working environment but doesn't it seem inefficient that with every update Rails and all that stuff in the vendor directory that hardly ever changes gets compressed and uploaded? That issue can be addressed using Cached Externals, which I've covered in another post.

gitting Started with Mephisto

2008-12-01T02:01:00Z

Jan. 14, 2009 Update: This post applies to Mephisto 0.8. See Upgrading to Mephisto 0.8.1 for information about moving to version 0.8.1

I took what I bet is a pretty common approach to setting up Mephisto. I put a copy on my laptop, played with it just enough to get it to work and then found and followed some fairly simple instructions to install it directly on my DreamHost account. It was an easy and straight forward approach but in my case by the time I finished my installation I knew I wanted to start over. The problem was that I quickly found myself modifying too many things to get Mephisto to work and look the way I wanted. I could tell it was going to be easier for me in the long run to get the blog engine in source code control with a development and deployment environment that made it to easy tweak and update from my laptop. Since Mephisto is a Rails application I went with the common choices: maintaining the blog code in a local git repository and deploying it using Capistrano.

In this post I cover setting up a custom git branch for Mephisto along with a couple of Mephisto modifications I recommend to prepare it for deployment to a DreamHost account. Actually there's really nothing DreamHost specific here but other hosting services could require additional changes to environment files. I'll cover deployment to DreamHost using Capistrano in a my next post. If you want to take this approach then I'm assuming you're fairly technical and already have git and Rails and MySQL installed on your computer. The use of git and Rails in this post is basic but going down this maintenance and deployment path is only worthwhile if you're reasonably comfortable with both technologies.

Prep the Blog Application Repository

There are a couple of versions of Mephisto out there; the last official release was 0.8 (Drax). I started with the current master branch of the technoweenie version from github. To do the same cd to some place on your hard drive where you want mephisto installed and type:

[mac]$ git clone git://github.com/technoweenie/mephisto.git
[mac]$ cd mephisto

After the download completed, I ended up with a copy that had a last commit SHA1 ID of 14dc7e9b27ceaafa8b26627eaf20e6bafa80b3b1. If there have been any updates you'll end up with something newer than mine. I'd stick with whatever you clone but if it's newer than mine it does mean there's a chance that a change could impact the instructions in this post. If you decide you want to be sure to have the same version you can reset to that commit:

[mac]$ git reset --hard 14dc7e9b27ceaafa8b26627eaf20e6bafa80b3b1

At this point you have the Mephisto master branch. Instead of treating my copy as a fork, I set up and switched to a separate branch to keep my changes separate. If you just want to pull in and merge updates with your code then you can just work from the master. If you decided to keep your changes in custom branch like I did then you can call it whatever you want. Since my version is intended for my website, I called mine robseaman:

[mac]$ git checkout -b robseaman

When I want to apply updates from the master my plan is to switch to the master, pull the updates and then go back to the custom branch and rebase my changes to the updated master, dealing with conflicts as necessary:

[mac]$ git checkout master
[mac]$ git pull
[mac]$ git checkout robseaman
[mac]$ git rebase master

Fine, but that's for the future, let's get back to the here and now. After the clone you have a git repository that's ready to use but it's setup to maintain a blogging application for reuse by anyone on the web, independent of themes and deployment information. While you could make that work, as individual deploying a blog I felt it would be simpler to treat the whole thing as one single Rails application. To customize it that way you'll need to change the .gitignore file. For example I changed my .gitignore from:

.rake_tasks
config/database.yml
config/deploy.rb
log
tmp
themes/*

to:

.DS_Store
config/database.yml
log
tmp
public/assets

Here are the reasons for my choices:

I Deleted .rake_tasks. It's not something I have, so I didn't leave it in.
I added .DS_Store. These are hidden files containing Mac GUI information. I'm on a Mac, if you aren't then you don't need this line.
Just like the original, I left config/database.yml out of the repository because it contains private account and password information. While it makes no sense for the database.yml to be in the Mephisto master project, for a private blog where the intention is to keep the repository local it's probably OK to check it in, especially if you only have the development and test account information. All that said, there's nothing here that really needs tracking and I like the idea that the repository can be moved around and shared without having to consider what password information it may contain.
I Added config/deploy.rb. It contains the Capistrano deployment script. This script is something that changes so tracking it with a source code manager is beneficial. An exception would be if you end up storing passwords in the deploy.rb. In that case I'd recommend leaving it out of the repository but I do think you should avoid passwords in deploy.rb if possible. For example, if you have a password in the script because you're using it to connect using Capistrano's :password variable then switch to SSH with public key signatures or put the password in a separate recipe or the ~/.caprc file. Or, if you're dynamically building the production database.yml and including a password in your deployment script then consider not automating that part of the deployment and creating the database.yml on the server or at least consider simply uploading the production database.yml as a separate file.
I added public/assets. In Mephisto this directory contains things like images for articles, which are more closely aligned with the database than with code. While it makes sense to maintain code locally, data and assets are created on the production server so they're not things that belong in the repository.
I removed themes. I wanted to be able make and track changes to themes locally as part of the application and push those changes up to a web server.

When you're done commit the .gitignore changes to the new branch:

[mac]$ git commit .gitignore -m "Changed to track changes for my blog."

Put Everything in Vendor

Unfortunately Mephisto isn't keeping pace with Rails. The master branch of the 0.8 version works with Rails 2.0.2. Many hosting servers have multiple versions of Rails. If yours does and 2.0.2 exists then it's possible to set RAILS_GEM_VERSION in config/environment.rb but my preference is more in line with the vendor everything camp. Meaning I prefer to keep gems including Rails in the vendor directory to avoid potential environment issues. Here are the commands to set-up Mephisto for 2.0.2 using the vendor directory:

[mac]$ rake rails:freeze:edge RELEASE=2.0.2
[mac]$ git add .
[mac]$ git commit -a -m "Froze to Rails 2.0.2."
[mac]$ cp vendor/rails/railties/environments/boot.rb config/boot.rb
[mac]$ git commit config/boot.rb -m "Updated to Rails 2.0.2 boot.rb."

The TZInfo gem is required by Mephisto. Again, I chose to I install it into the vendor directory:

[mac]$ sudo gem install tzinfo
[mac]$ cd vendor
[mac]$ gem unpack tzinfo
[mac]$ cd ..
[mac]$ git add .
[mac]$ git commit -a -m "Put tzinfo 0.3.12 in vendor dir."

Minor Code Changes

The first code change I recommend is to change the session secret in config/environment.rb. It's the line that looks like this:

3738	config.action_controller.session = { :session_key => "_mephisto_session", :secret => "....big random string...." }

The :secret is an application wide key string used for the cookie store and should be set to something fairly long, at least 30 characters, consisting of random numbers and letters. Rails generates a unique string for each application but since Mephisto is distributed broadly with a common key if you don't change it then Mephisto (like most open source Rails apps) is vulnerable to session hijacking. There was a discussion of this in the Ruby on Rails: Core group. I generated my :secret replacement by creating a temporary Rails application:

[mac]$ cd ..
[mac]$ rails temp

and copying the secret from temp/config/environment.rb into Mephisto's environment.rb. After the edit I deleted the temp application and committed the change:

[mac]$ rm -rf temp
[mac]$ cd mephisto
[mac]$ git commit -a -m "Changed secret from distributed version."

The only other change I recommend before using Mephisto is to comment out the multi-site enabling line in config/initializers/custom.rb unless you need multi-site support:

# Site.multi_sites_enabled = true

The multi-site feature enables a single Mephisto implementation to support multiple blogs. I think to get it to work properly you need to leave this line uncommented and do some additional setup steps. I phrased this as I think because it's not completely clear to me what the current state of this feature is. Originally the enabling line in custom.rb was commented out by default but that was changed for some reason last spring. I discovered the setting early on while playing with Mephisto on my production server when I noticed image support wasn't working. When I investigated I found my images existed but were saved in public/assets/unusedfornow.com instead of in public/assets as they were on my development server. As far as I could tell my set-up was the same in both environments, a little investigation led me to this line and after a commenting it out I had thumbnails and consistent behavior. I was likely tinkering with other things at the same time so I'm not 100% sure if I needed to do more setup or if I had the issue for some unrelated reason but since I had no need multi-site support and commenting this line out worked for me, I never looked back:

[mac]$ git commit config/initializers/custom.rb -m "Disabled multi-site support."

Program and Database Initialization

Mephisto leaves it up to you to create your database.yml but it does have an example intended to provide a starting point:

[mac]$ cp config/database.example.yml config/database.yml

Edit the database.yml and change the development database connection settings to something that will work on your local system. As always the development, test and production databases could use different DBMS software but I made sure to use the same database manager for both. SQLite may be fine for development but I like to have as much consistency as possible between development and production and I wanted to make sure it would be easy to pull production data down to my development system. DreamHost provides MySQL support so I stuck with the Mephisto's default choice of MySQL and the root user and just changed the password.

You can also set-up the test database configuration as well, I did but haven't used it yet. I don't recommend setting the production settings on your development box, although a it's little more work, it's more secure to only keep this information on the production server. I'll cover production database set-up for DreamHost in a separate deployment post. Below the production database settings are example connection set-ups and conversion connection templates. I just left them in for reference but you could delete them if you want. After the edits you can create the development database. The Rails db:create rake task needs to access the log directory, which doesn't exist yet, so do it the old fashion way using your DBMS:

[mac]$ mysqladmin -u root create -p mephisto_development

If you don't have a root password you can leave of the -p, otherwise respond with your password when prompted. Mephisto comes with it's own bootstrapping task to create the necessary folders, initialize the database and setup the default theme. For a first time set-up use this program and forget about migrations. It generates a lot of files so a git update is also required:

[mac]$ rake db:bootstrap
[mac]$ git add .
[mac]$ git commit -a -m "Ran db:bootstrap"

Go Man Go

At this point the Mephisto blogging application is ready to run on your development system. It's also in a git repository that's ready to be moved to a web hosting service. It would be a good idea before deploying to make sure it works:

[mac]$ script/server

Assuming your using Mongrel for your development environment, login to the administration portion of the site at http://localhost:3000/admin with Login: admin and Password: test and play around.

SSH with Leopard

2008-11-30T00:05:00Z

I recently set-up a secure shell account for a DreamHost shared hosting account that I intend to use for automating deployment and backup tasks from a Mac OS X Leopard client. While not overly complicated there were a couple of gotchas that I ran into. This post consolidates my findings and gives you a brief background for what's going on when you set up SSH access. The information applies to Mac OS X 10.5.1 or greater.

Secure Shell Access

The first step is to make sure you have SSH and/or SFTP access for one of your hosting service accounts. How you accomplish this will vary from one service to another. For DreamHost it's very easy, go to the Manage Users > Users section of your DreamHost web panel, select Add A New User or Edit an existing user and fill in the account information. Be sure the User Account Type is set to Shell account, which includes support for both SSH and SFTP. If you have a shared hosting account then you'll need to make sure the Username is unique. The error messages should lead you through any issues. If you want more detail check out the DreamHost Enabling Shell Access Wiki.

For the rest of this post I'll assume your domain is called mydomain.com and you have an SSH user account named mydom_user. Now open Terminal in OS X and try using SSH. Here's the typical format for the ssh command:

[mac]$ ssh mydom_user@mydomain.com

You should be prompted for your password. If you aren't prompted then check your hosting service documentation to see if you need to use something other than the user@hostname format to connect. If you are prompted and you enter your password correctly then you should see a greeting followed by the command prompt from your server. At this point you have secure shell access working. The main limitation is that the password gets in the way for automated tasks such as deployment scripts and client initiated cron jobs. If you don't mind being prompted for a password when you use SSH and you either don't have automation needs or don't mind setting up your password in automation scripts then congratulations, you're done!

Digital Signature Setup

Assuming you don't want to deal with your SSH password then the solution is to use a public/private key pair for authentication. The first step is to create a directory to hold your Mac's public key on your domain server. For SSH to work this directory must be named .ssh and it must have owner only read, write and execute privileges. If you logged out then log back in to your shell account and type:

[dh]$ mkdir .ssh
[dh]$ chmod 700 .ssh
[dh]$ logout

That's it from the server side. You should be back to your regular OS X prompt. Unless you already have an existing private/public key pair you'll need to create one and copy the public key to your server. First check to see if any keys are already defined:

[mac]$ ls -l ~/.ssh

If you see an id_rsa and id_rsa.pub file or an id_dsa and id_dsa.pub file then you have a key pair already defined. For example, I have:

...
-rw-------@  1 Rob  Rob  1743 Nov 11 15:16 id_rsa
-rw-r--r--@  1 Rob  Rob   405 Nov 11 15:16 id_rsa.pub
...

The file without the extension contains the private key; the file with the .pub extension is the public key that can be shared. They don't have to be named id_rsa or id_dsa but those are the default names. ID is used because the keys establish an identity, while RSA and DSA are associated with the two public key signature algorithms that SSH supports. I'll describe how to generate a key pair later but first I want to cover the situation that I had with existing keys.

In my case an RSA key pair already existed but I didn't remember creating it. At this point creating a new set of keys or overwriting the existing pair may have been a good idea but I didn't want another identity and I didn't want to break something that I'd set up previously but couldn't remember at the moment. So, I decided to try to use the existing key pair. The first issue I had was with the privilege settings on my private key. The file should only have owner read, write privileges ( -rw-------). Initially, the permissions were wrong on my file. I have no idea why but at some point they got messed up. When I tried using the key it was ignored and I got the following warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0622 for '/Users/Rob/.ssh/id_rsa' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /Users/Rob/.ssh/id_rsa

If your private key file has -rw-r--r-- or any permissions other than -rw-------, you'll need to fix it:

[mac]$ chmod 600 ~/.ssh/id_rsa

The next issue I had was that I was prompted for my "passphrase" and I couldn't remember it. You can think of a passphrase as a password that's used to protect your private key. The difference is that spaces and tabs are allowed in passphrases. If you have a key pair and remember the passphrase you used when you created it then your set. Since I didn't remember mine, I decided to open the Keychain Access application in the Applications/Utilities folder to see if I could find it. If you're in the same situation checking your keychain is probably worth a try. If the passphrase is there you'll find it in the Passwords/Application Category of the login Keychains. It'll have a name starting with SSH:, but don't be surprised if you don't find it. Passphrases were not stored in the keychain until OS X 10.5.1 and even then only if you chose to store it when you used the key. You also won't find passphrases in your keychain if you're using the MacPorts OpenSSH implementation. I'll get to MacPorts later but if you don't find your passphrase and want to try a few guesses at it then you can use the ssh-keygen command with the change password option:

[mac]$ ssh-keygen -f ~/.ssh/id_rsa -p

Ultimately, if you can't remember or find your passphrase then you'll need to generate a new key pair or overwrite your existing pair. Generating new keys or overwriting existing keys can be accomplished the same way:

[mac]$ ssh-keygen

will generate an RSA key and prompt you for a passphrase. It will also prompt for a file name, defaulting to ~/.ssh/id_rsa. If you have an existing key pair with the same name it'll make sure it's OK with you before overwriting an existing key pairs. RSA is the default, if you want DSA then add the -t dsa option when you run ssh-keygen.

The passphrase is optional and if you just hit enter twice when prompted for it then you won't set one. Don't do that. It may seem that having a passphrase gets you back to square one with a password prompt but that's an easy limitation to get around, especially with Leopard. For now take my word for it and enter a passphrase. You should also pick something different than your password. I don't want to get too sidetracked with why a passphrase is so important but keep in mind that your private key is stored in a text file on disk and without a passphrase for protection anyone who gets your key could access anything that key is used for.

At this point you should have a private/public key pair and a known passphrase. I'll assume your private key is named id_rsa but you can make substitutions if you chose another name. Now you need to copy the public key to your server. You should also make sure it just has owner read, write privileges if it's a new file:

[mac]$ cat ~/.ssh/id_rsa.pub | ssh mydom_user@mydomain.com \
> "cat >>  ~/.ssh/authorized_keys; chmod 600 .ssh/authorized_keys"

If you enter your password correctly then the copy should work. I should note that instructions from other sites will often say to use authorized_keys2 instead of authorized_keys, that's fine but authorized_keys2 has been depreciated so I went with authorized_keys. You'll also see a lot of instructions using the secure copy (scp) command to move the file but I like the cat with the pipe and redirection format because it makes it clear how to append new public keys. Now it's time to give your public key signature a try:

[mac]$ ssh mydom_user@mydomain.com

If you're luckly you'll be prompted for your passcode with a dialog prompt like this:

If you get the dialog just enter the passphrase and check the box to save it in your keychain. From now on you should be able to use SSH without being prompted. You're done!

Don't Blame Apple

If you're still reading then I'm assuming things didn't go according to plan. Perhaps you got the following error:

percent_expand: NULL replacement

or, instead of being prompted by a GUI dialog box you get prompted for your passphrase on the command line:

Enter passphrase for key '/Users/Rob/.ssh/id_rsa':

These issues typically occur because you're not using the Leopard version of the ssh program. First, verify that's the problem:

[mac]$ which ssh

If you get back /usr/bin/ssh then you're using the Leopard version of ssh and I don't know what's wrong so you'll need to look elsewhere for a solution. On the other hand, if you get back/opt/local/bin/ssh then you're running the MacPorts version of ssh and I'll walk you through a solution. If you get a different path then there's still a good chance you're using the MacPorts ssh but it was installed somewhere other than the default location. The following instructions will probably still work but you'll need to adjust the paths.

OK, at this point I'm assuming you're using the OpenSSH implementation from MacPorts and your either getting the percent_expand: NULL replacement error or your continuing to see passphrase prompts when you use ssh. If so, the easiest solution is just to switch to Leopard's SSH commands. Just renaming the MacPorts SSH commands so the Leopard version is found instead will work but I like this technique taken from from github better:

[mac]$ sudo -s # To get a root prompt
bash-3.2# cd /opt/local/bin
bash-3.2# mv ssh{,_macports}
bash-3.2# ln -s /usr/bin/ssh ssh
bash-3.2# exit

It renames ssh and leaves a symbolic link behind allowing any software that looks specifically for the program in the MacPorts directory to find them. The github guide, and every other Internet post I've seen, renames scp as well but as far as I can tell that's not necessary. At least on my Mac only ssh has problems, scp and sftp don't exhibit the NULL replacement problem and the Leopard version of ssh-agent is automatically launched to take care of the passphrases. That said, you could also use the same renaming technique for scp, sftp and any other SSH commands that you want. For example, the Leopard version of ssh-add has additional options for keychain support that don't exist in the MacPorts version. Anyway, after you're done with renaming then when you use the ssh command you should get the desired Leopard dialog box and behavior described above.

Other Ideas and Resources

While I prefer switching to the Leopard ssh, if for some reason you want to stick with the MacPorts version then you may be able to solve the percent_expand: NULL replacement error by updating to the "openssh 5.0p1_0+darwin_9" version of MacPorts SSH (see bottom of this). I haven't tried the update so I'm not sure if it works. Even if it doesn't you can definitely work around the problem by creating/updating your ~/.ssh/config file (see here). To avoid the continuous passphrase prompting, you could just run sftp or the Leopard version of ssh once to cause Leopards ssh-agent to save it in your keychain. I suppose it would also be possible to switch over to the MacPorts ssh-agent and use it along with a utility such as SSHChain or SSHKeychain. You can also find plenty of generic Unix passphrase prompting solutions by googling for ssh-agent. Many of them aren't Mac compatible but I'm sure you can find something that works.

If you want to dig more into SSH on the Mac then Dave Dribin's has a couple of great blogs. As a starting point take a look at his ssh-agent on Mac OS X 10.5 (Leopard) article where he describes, among other things, how the Leopard version of ssh-agent is automatically launched as necessary.

Rob Seaman - Home

Detecting the number of processors with Ruby

Adding Rubinius to multiruby Versions

Installing Rubinius in ~/.multiruby

Running Rubinious Outside of multiruby

Rubinius as Ruby via Alias Switching

Upgrading to Mephisto 0.8.1

The Original Plan

The Updated Plan

Interactive Rebasing

A new .gitignore

Resolving Conflicts

Updating Cached Externals

Common 0.8.1 Update Stuff

New Theme Locations

Deploying to Production

Updating Gmail SSL SMTP Support for Ruby 1.8.7

Adding JRuby to multiruby Versions

Install JRuby in ~/.multiruby

Adding JRuby to the PATH

Switching Between Ruby 1.8 & 1.9 with multiruby

Set-Up

Managing Installations

RubyGems

Switching Between Ruby Versions

Other Ways to Switch Ruby Versions

Sending CTRL-C to a Subprocess with Ruby

Unix without Standard Error

Adding Support for Standard Error

Making it work in Windows

Other Options

Installing Ruby 1.8.6, 1.8.7 and 1.9 on Leopard

ruby-1.8.7-p72

1.8.6-p287

1.9.1-preview2

Adding Gmail SSL SMTP Support to Rails

Housekeeping

Cached Externals

Using Cached Externals

Additional Information

Production Data to Development

Limitations

Hiding MySQL Passwords with Capistrano

Avoiding the MySQL --password Option with Capistrano

Limitations

Deploying Mephisto with Capistrano to DreamHost

Server Prerequisites

Capistrano Setup

Initial Deployment

Subsequent Deploys

Is that it?

gitting Started with Mephisto

Prep the Blog Application Repository

Put Everything in Vendor

Minor Code Changes

Program and Database Initialization

Go Man Go

SSH with Leopard

Secure Shell Access

Digital Signature Setup

Don't Blame Apple

Other Ideas and Resources

Installing Rubinius in `~/.multiruby`

Install JRuby in `~/.multiruby`