Freezing a gem that has native extensions

January 13, 2010

I like to freeze all the gems I use as we run in a shared hosting environment and need to our apps isolated from each other. Deployments are also handled by an operational team that does not intimately understand our applications so keeping our deployments to a single capistrano command cap deploy:migrations has been a big win for us. Freezing most gems is pretty straightforward and has been built in since Rails 2.1. When dealing with a gem that requires native extensions to be built there's only one additional step to add to your Capfil.

Let's say we want to localize hpricot which does include native C extensions.

First tell Rails about your gem by adding a config.gem line to your environment.rb


Rails::Initializer.run do |config|
...
config.gem 'hpricot'
...
end



Now we can ask rails about its configured gems


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [I] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)



The 'I' means its hpricot is installed on my system but not frozen in the application. If you see '[]' instead you need to run sudo gem install hpricot (add '--source http://gemcutter.org' if necessary). At this point you could write some code to use hpricot and your application will work. But if hpricot (or the version you're expecting) is not installed on your production server you'll be in trouble.

To freeze the gem into your vendor directory run rake gems:unpack (optionally you can add 'GEM=hpricot' if you just want to unpack one gem).


$ rake gems:unpack
(in /Users/alexrothenberg/ruby/my_project)
Unpacked gem: '/Users/alexrothenberg/ruby/my_project/vendor/gems/hpricot-0.8.2'



We can ask rails again...


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
The following gems have native components that need to be built
hpricot

You're running:
ruby 1.8.6.287 at /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
rubygems 1.3.2 at /Users/alexrothenberg/.gem/ruby/1.8, /Library/Ruby/Gems/1.8, /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8

Run `rake gems:build` to build the unbuilt gems.



Oops our vendored gem is missing hasn't built the native extensions. Not to worry the message tells us what to do and we run rake gems:build


$ rake gems:build
(in /Users/alexrothenberg/ruby/my_project)
Built gem: '/Users/alexrothenberg/ruby/mars-admin/vendor/gems/hpricot-0.8.2'
alex-rothenbergs:mars-admin alexrothenberg$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [F] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)



We can ask rails again to see that the gem is now frozen and also look in our vendor folder


$ rake gems
(in /Users/alexrothenberg/ruby/my_project)
- [F] hpricot

I = Installed
F = Frozen
R = Framework (loaded before rails starts)

$ ls vendor/gems/hpricot-0.8.2/
total 72
-rw-r--r-- 1 alexrothenberg staff 4672 Jan 13 12:33 CHANGELOG
-rw-r--r-- 1 alexrothenberg staff 1048 Jan 13 12:33 COPYING
-rw-r--r-- 1 alexrothenberg staff 9216 Jan 13 12:33 README
-rw-r--r-- 1 alexrothenberg staff 8242 Jan 13 12:33 Rakefile
drwxr-xr-x 4 alexrothenberg staff 136 Jan 13 12:33 ext/
drwxr-xr-x 3 alexrothenberg staff 102 Jan 13 12:33 extras/
drwxr-xr-x 6 alexrothenberg staff 204 Jan 13 12:39 lib/
drwxr-xr-x 11 alexrothenberg staff 374 Jan 13 12:33 test/



Everything looks good and you can check this into git and now have a frozen version of the hpricot gem stored with your application.
But if we stop here, when we deploy to our production server we'd be using the native extensions we built on your laptop which may not work on the server if you have one is 32bit and the other 64bit or you have different OS libraries installed or any number of other reasons.

To be safe, we need to rebuild the native extensions on the server when we deploy. This is not as hard as it sounds as rails gave us the rake task rake gems:build. We can ask capistrano to run that command on the server by adding the following to your Capfile.


after "deploy:finalize_update" do
# build the native extensions for hpricot gem
run "cd #{release_path} && #{rake} RAILS_ENV=#{rails_env} gems:build GEM=hpricot"
end



Now when capistrano deploys in with all the other messages you'll see something like


...
* executing "cd /opt/apps/my_project/releases/20100108185109 && rake RAILS_ENV=production gems:build"
servers: ["your.server.com"]
[your.server.com] executing command
** [out :: your.server.com] (in /opt/apps/my_project/releases/20100108185109)
** [out :: your.server.com] Built gem: '/opt/apps/my_project/releases/20100108185109/vendor/gems/hpricot-0.8.2'
command finished
...



So rails give us a few simple patterns to follow to freeze our gems in the vendor folder and with a few lines in you Capfile you can use this pattern to vendor a gem with native extensions.