Hi everybody!

RailsConf is under way this week. We love Rails as much as you do, so we’re totally there.

Our own Jesse Newland will be giving a talk, the coolness of which rock your world. Continous (Production) Integration: Ruby on Rails Application Monitoring with Cucumber on Wednesday at 4:25PM . His words sooth wild mares and pacify wars. Do not miss it.

I’ll also be around and hacking Moonshine throughout RailsConf. If you’d like to pitch in, or have any moonshine-related questions in general, definitely find me by (look for the jar of moonshine) and say hi! In particular, I’m planning to work on:

• Rails 3 support for latest beta
• Full Ubuntu 10.04 (Lucid) support
• Really full stack integration testing, including deploying and verifying the system is up and running

We love meeting and remeeting our customers and other friends of Rails Machine. If you’re going to be around, we’re organizing a dinner, so definitely drop us a line (bradley at railsmachine dot com).

At Rails Machine, we have recently seen problems on a few deploys where Apache’s workers become overwhelmed and unable to serve new requests, even static ones. While we haven’t identified the root cause yet, we suspect either a recent update to The Browser Which Shall Remain Unnamed causes it to abuse the length of time connections stay alive or there may be something lurking in the recent 2.2.10 Passenger release.

Don’t worry though, because I have updates from the still about Moonshine’s Apache management. Previously, moonshine did not need to change anything in /etc/apache2/apache2.conf, so it defaulted to Ubuntu’s apache-common package… until now.

The primary motivation for doing this was specifically to turn off KeepAlive by default.

If you want to turn KeepAlive BACK on, you need to add something like this to your moonshine.yml:

:apache:
  :keep_alive: 'On'

Note that On needs to be quoted or YAML will convert it to true or false value, which would not be valid for the configuration. As a side note, I should probably find a fix for that.

If you are running a high-traffic site, and still experience problems not having idle Apache workers with KeepAlive Off, you may get some mileage out of tweaking :max_clients and :server_limits too.

To take advantage of these changes, you just need to update moonshine:

# if managed with script/plugin
script/plugin install 
# if installed using a submodule
git submodule update

For reference, here’s the commit:

commit 3f46140641fb22ce25e0b2b81497d4aec88e90b2
Author: Joshua Nichols <josh@technicalpickles.com>
Date:   Fri Feb 26 13:49:20 2010 -0500

    Added management of /etc/apache2/apache2.conf

    You can now specify the following apache directives using configuration:

     * Timeout => :timeout (default 300)
     * KeepAlive => :keep_alive (default 'Off')
     * MaxKeepAliveRequests => :max_keep_alive_requests (default 100)
     * MaxKeepAliveTimeout => :keep_alive_timeout (default 15)
     * MaxClients => :max_clients (default 150)
     * ServerLimit => :server_limit (default 16)
     * Timeout => :timeout (default 300)

Rails Machine is growing and we’re looking for some great additions to our talented team. All positions require an in depth knowledge of Ruby, Rails, and Linux.

Deployment and Operations Engineer

• Application deployment
• Configuration and automation
• Scaling and performance tuning
• Monitoring and response

VP of Consulting Services

• Scaling, performance, configuration, deployment
• Contract-based and on-demand consulting
• Scheduling/task management/Kanban

VP of Application Engineering

• Application design and development
• Application integration
• Business support

VP of Cloud Operations

• Data center management
• Infrastructure automation
• Virtualization services

Contact Bradley for more details. Please include GitHub account name and resume.

Here at Rails Machine, we have written, on occasion, about moonshine, our open source configuration management and deployment system. But now, I’m happy to announce it works with the Rails 3.0.0.beta.

Lots of people have written about getting up and running on Rails 3, or about upgrading from Rails 2 to Rails 3. I’m not going to repeat that here, but here are some links that might interest you:

• rails-upgrade: Automating a portion of the Rails 3 upgrade process
• Live coding Rails 3 upgrade
• Upgrading to Rails 3

Instead, I will focus on getting an existing Rails 3 application deployed. For narrative purposes, I’m going to spin the tale in which I deploy a fictional application.

The background

Here’s a little background you should be aware of before I begin my tale

• I have an application, zomgblog, which works with rails-3.0.0.beta
• I’m hosting the code on GitHub at http://github.com/technicalpickles/zomgblog)
• I have a freshly installed Ubuntu 8.10 server (the version supported by moonshine)
• I have a domain name setup, zomgblog.technicalpickles.com, which points at my freshly setup server
• I have a ‘rails’ user on the server, with my SSH public key setup in ~/.ssh/authorized_keys for password-less access
• I setup sudo to allow ‘rails’ to allow sudo without a password by adding rails ALL=(ALL) NOPASSWD:ALL to /etc/sudoers on the server
• I have installed capistrano locally: gem install capistrano

Adding moonshine to my application

I started in my local checkout. First thing I did was install moonshine itself:

    rails plugin install git://github.com/railsmachine/moonshine.git

Next, I used the moonshine-provided generator to setup moonshine for my application. It has a few options, so I reviewed `rails generate moonshine—help`. Ultimately, I decided on:

    rails generate moonshine --domain zomgblog.technicalpickles.com --user=rails --repository git@github.com:technicalpickles/zomgblog.git

I don’t plan to deviate from the default moonshine stack, so there’s not much else to change. I do need to my config/database.yml for production though. Moonshine assumes MySQL will be used, and will handle installing it, creating the database, and running migrations, so I need a production stanza that looks like:

    production:
      adapter: mysql
      encoding: utf8
      reconnect: false
      database: zomgblog_production
      pool: 5
      username: zomgblog
      password: zomgsosecretandsecure
      socket: /var/run/mysqld/mysqld.sock

I knew my application was working with the gem dependencies I had installed locally. I didn’t want any surprises though, so I should use the same versions on production. bundler made this easy:

    bundle lock

Before proceeding, I needed to commit and push the changes:

    git add .
    git ci -m "Moonshined application and locked dependencies" 
    git push origin master

The first deploy

At this point, I’m ready to bootstrap my server. This takes about 15 minutes, and will create the minimal environment that moonshine can run in.

    cap deploy:setup

I’m ready to deploy when that’s done. Moonshine does a lot of heavy lifting during deployment which makes sure the server is in compliance with what’s defined in app/manifests/application_manifest.rb. Since I didn’t change anything, I end up with the default stack of Apache, MySQL, and Passenger. This usually takes about 10 minutes.

    cap deploy

I may now rejoice and reap the rewards of a fully deployed Rails 3 application.

Subsequent deploys

Application development, like life, goes on. From now on, I can continue to develop and deploy just like any other Rails application out there

    # insert development & testing
    git push origin master
    cap deploy

In cases of changing dependencies, I need to do a little extra work before deploying because of the way bundler works:

   # make changes to Gemfile
   bundle unlock
   bundle install
   bundle lock
   git add Gemfile Gemfile.lock
   git ci -m "Updated dependencies" 
   cap deploy

In the case of having to changing the configuration of my server, I don’t even need to ssh there. Moonshine (and the underlying shadow_puppet) provides an automated way to manage it. See the Shadow Puppet overview and examples for more details.

Some caveats with Rails 3 and moonshine

There are a two points worth noting, particularly if you are used to using moonshine with Rails 2.

First, before bundler was introduced, moonshine had it’s own mechanism for installing gems on the server:

• Add dependencies to config/environment.rb
• Run rake moonshine:gems to update config/gems.yml
• Commit config/gems.yml.

When deploying, moonshine uses config/gems.yml to install the gems, in addition to system dependencies they might need (ie libxml2 for nokogiri).

But now Rails 3 has the bundler, so moonshine will try to use bundler if you have a Gemfile in your application. One side effect of this is that, because of a bug in rubygems-1.3.5, moonshine isn’t able to install system dependencies of gems managed by Gemfile. This is fixed in 1.3.6 when released

The second is that Rails 3 is Rack based, so Passenger’s Rack support is used. According to the 2.2.9 release notes, Passenger does not support it’s smart spawn mode for Rack applications yet though. Contrary to some reports, removing config.ru will not allow Passenger’s Rails support to be used instead of its Rack support because removing config.ru will render your application unbootable.

Future Moonshine developments

It’s been a long road to Rails 3, and we at Rails Machine fully intend to keep moonshine up and running with it. Don’t worry though: Rails 2 support is not being deprecated.

Supporting Rails 3 does open some interesting doors. In particular, by supporting bundler, any application with a Gemfile should benefit from the improvement. In addition, Rails 3 is a first class Rack citizen, so anytime there’s a config.ru file, Passenger will default to it’s Rack mode.

Between bundler support and detecting config.ru to turn on Passenger’s rack support, this moves moonshine in the right direction to support any Rack application in the future, not just Rails.

Jesse Newland and I are back from the 2009 Ruby Hoedown, the South’s regional Ruby conference. We both had a blast! The talks were all enjoyable and very informative, and we had an awesome time hanging out with folks like John Nunemaker, Steve Smith, Matt Todd, and many, many others.

It was great seeing all of you there who did come by and say “Hi” at some point this weekend—we hope you enjoyed the conference (and your spiffy lanyards!) as much as we did.

The talks that really stuck out out in my mind were those given by Ben Mabey (Writing Software, Not Code With Cucumber), Leon Gersing (couldn’t find a link to his slides, but it was on Appcelerator Titanium), and Jim Weirich (Source Control for People Who Don’t Like Source Control). The talk on threading in Ruby 1.9 was also really interesting—it’s definitely something we’ll revisit when more people are using 1.9 (ie. right around the release of Rails 3.0). The lightning talks were particularly amusing, both for the fact that they’re very impromptu and also because the speakers were given props to play with during their presentations. Anyone who was there can tell you all about the guy with the kazoo. All of the talks were very well done, and I’d like to thank everyone who did speak this year for doing a great job.

If you’d like to read up on specifics of the talks themselves, take a look at one of the first recaps I’ve seen posted or at the hashtag on twitter.

While Jesse and I had a blast at the conference, I think I can speak for both of us when I say that I’m glad to be back home, working on your rails machines to make sure you get the best experience possible.

Jesse Newland and I are currently on our way up to Nashville for this year’s Ruby Hoedown, the South’s regional Ruby conference. We’re both looking forward to good talks, demos, and of course hacking on some awesome stuff. We’ll definitely be at the Smash-Up Derby, as well as a few other things in the area. For a list of things to do while in Nashville, we recommend taking a look at this list compiled by Rick Bradley.

If you’re a customer, are interested in hosting at Rails Machine, or just want to ask us some questions, come find us (or hit us @railsmachine) and we’ll grab a beer and chat. And remember, our “ask us anything” support is always available in person!

We hope we’ll see you there, but if we don’t, we hope you have a good time at the Hoedown!

We are excited to announce that Scout monitoring is now available for free (via a $14 off coupon) to all Rails Machine customers. Created by our pals at Highgroove Studios, Scout was recently updated with a slew of awesome new features including deep Rails instrumentation, triggers, and a more robust agent. With Scout’s extensible plugin system, you can monitor any data that you want and track any trends over time. We’re big fans of the plugin system for catching tricky issues that elude basic system health metrics. There are a number of community-supported plugins available for your own use.

If you are a customer of our Ruby on Rails managed hosting and not currently on Scout, we will be in touch and take care of it without worry. If you have a traditional account, then please request a coupon code via the support request form. This coupon code will provide $14 off any of Scout’s affordable plans ($7/server) if you need more than the Basic account. We even have a new plugin for Moonshine for automating Scout’s installation.

If you have a server at Rails Machine, please head over to the Rails Machine Status Blog for our detailed take on the Ruby BigDecimal security vulnerability that was announced late last night.

If you’re not already subscribed to our Status Blog, now is an excellent time to do so. We post post news and updates about system, security, and network issues affecting multiple Rails Machine customers over there, and generally reserve this blog for other posts, such as job postings and software releases. Here’s the RSS URL for the Status Blog:

http://status.railsmachine.com/rss

Love helping people with their Rails deployments? Come work for The Machine! Rails Machine is hiring a part-time or full-time Ruby on Rails Support Engineer in Atlanta, GA or the surrounding area.

As a Ruby on Rails Support Engineer, you would work with the existing Rails Machine team to:

• Support existing customers’ Ruby on Rails deployments
• Deploy new customers’ Ruby on Rails applications
• Write documentation covering common problems
• Write software to automate anything you do more than twice

In addition to the above work, which can all be performed from the comfort of your home office, your back patio, our coworking space, or anywhere you can find the internets, the following work will need to be occasionally performed onsite at our two Atlanta data centers.

• Unbox, rack, cable, and configure new servers
• Perform basic server maintenance (replace hard drives, etc)

You should be comfortable with all of these things:

• Linux
• Ruby
• Rails
• Passenger
• Mongrel
• Apache
• MySQL
• Capistrano
• Git
• Subversion

You should be excited to learn about these things:

• Xen
• KVM
• Nagios
• LVS
• Puppet
• Moonshine
• Sinatra
• TDD/BDD with RSpec

Rails Machine is a small business, so this will be anything but a boring job. We all wear many hats, but have a damn good time doing it. Oh, and if you like strange beers, that helps too, as we’re running out of names for our servers.

If this sounds like fun to you, please send your resume, a link to your GitHub account, and a note explaining why you’d be perfect for the job to Jesse Newland. Thanks!

We hope by now that you’ve had a chance to check out Moonshine and maybe even give it a go on your own server. We’ve been using Moonshine to deploy our customer’s servers for a while now, and we’ve started extracting little pieces of manifests into Moonshine plugins. I’m going to show you how to add Moonshine plugins to your manifest, and give you the lowdown on writing your own.

Installing and using plugins

Moonshine plugins are installed just like any other Rails plugin:

script/plugin install git://github.com/railsmachine/moonshine_ssh.git

Then add a few lines to your application’s manifest- app/manifest/application_manifest.rb, by default.

configure( :ssh => { :allow_users => ['rails'] } )
plugin :ssh
recipe :ssh

Not all plugins will need to be configured before they’re used and the recipe name doesn’t need to be the same as the plugin name- in fact, a plugin could provide as many recipes as you like. The next time you deploy your application, Moonshine will run the recipes you specified from the installed plugins.

Writing your own

Now here’s how you can really shine while you ‘shine. If you’ve added functionality in your manifest that you think others could use, create a new Moonshine plugin for it and put it up on github. Here’s how we do it.

Generate the plugin stub

Start by running the plugin generator from your application directory. Just switch out “ssh” for whatever it is you’re plugin-ifying.

script/generate moonshine_plugin ssh

This creates the following structure in the vendor/plugins directory of your app:

  |--moonshine_ssh
     |-- README.rdoc
     |-- moonshine 
     |  `-- init.rb
     |-- lib
     |  `-- ssh.rb
     |-- spec
         |-- spec_helper.rb
         |-- ssh_spec.rb

Add recipes

The lib/ directory is where we’ll put puppet recipes. For our SSH plugin we add this:

def ssh(options = {})
  package 'ssh', :ensure => :installed
  service 'ssh', :enable => true, :ensure => :running

  file '/etc/ssh/sshd_config',
    :mode => '644',
    :content => template(File.join(File.dirname(__FILE__), '..', 'templates', 'sshd_config'), binding),
    :require => package('ssh'),
    :notify => service('ssh')
end

This is straightforward- we’re making sure that SSH is installed, that the service is running and will start on system boot. We’ll create a templates/ directory with an ERB template called sshd_config. It will be rendered and saved as /etc/ssh/sshd_config. The SSH service will be notified to restart when it changes. The template will have lines like this sprinkled throughout:

LoginGraceTime <%= options[:login_grace_time] || 30 %>
PermitRootLogin <%= options[:permit_root_login] || 'no' %>

The options come from our configure line in the first example. If you recall, we defined a property called :ssh, which we set to a hash. This hash will be passed to the ssh recipe by Moonshine. This little convention helps keep the custom configuration of your server separate from its list of parts.

Do the safety dance

Now we’ve got a working plugin that can manage your SSH settings. Just one thing- what if someone configured ssh like this?

configure(:ssh => {:port => 'two'})  

This clearly isn’t valid and we don’t want them to be locked out, so let’s test the file before we update it.

def ssh(options = {})
  package 'ssh', :ensure => :installed
  service 'ssh', :enable => true, :ensure => :running

  file '/etc/ssh/sshd_config.new',
    :mode => '644',
    :content => template(File.join(File.dirname(__FILE__), '..', 'templates', 'sshd_config'), binding),
    :require => package('ssh'),
    :notify => exec('update_sshd_config')

  exec 'cp /etc/ssh/sshd_config.new /etc/ssh/sshd_config',
    :alias => 'update_sshd_config'
    :onlyif => '/usr/sbin/sshd -t -f /etc/ssh/sshd_config.new',
    :refreshonly => true, # do nothing until notified
    :require => file('/etc/ssh/sshd_config.new'),
    :notify => service('ssh')
end

There you have it. We upload the new configuration to a temporary location and replace the previous one only if the syntax check returns 0. This plugin is ready to be put online for all to enjoy.

The first plugins

So far, we’ve released plugins for managing iptables, SSH, and god. Look for more in the coming days. If you’ve written one, let us know about it in the comments!

Update: Check out the list of plugins and add your own on the plugin wiki page.