We are excited to announce that Scout monitoring is now available for free (via a $14 off coupon) to all Rails Machine customers. Created by our pals at Highgroove Studios, Scout was recently updated with a slew of awesome new features including deep Rails instrumentation, triggers, and a more robust agent. With Scout’s extensible plugin system, you can monitor any data that you want and track any trends over time. We’re big fans of the plugin system for catching tricky issues that elude basic system health metrics. There are a number of community-supported plugins available for your own use.

If you are a customer of our Ruby on Rails managed hosting and not currently on Scout, we will be in touch and take care of it without worry. If you have a traditional account, then please request a coupon code via the support request form. This coupon code will provide $14 off any of Scout’s affordable plans ($7/server) if you need more than the Basic account. We even have a new plugin for Moonshine for automating Scout’s installation.

If you have a server at Rails Machine, please head over to the Rails Machine Status Blog for our detailed take on the Ruby BigDecimal security vulnerability that was announced late last night.

If you’re not already subscribed to our Status Blog, now is an excellent time to do so. We post post news and updates about system, security, and network issues affecting multiple Rails Machine customers over there, and generally reserve this blog for other posts, such as job postings and software releases. Here’s the RSS URL for the Status Blog:

http://status.railsmachine.com/rss

Love helping people with their Rails deployments? Come work for The Machine! Rails Machine is hiring a part-time or full-time Ruby on Rails Support Engineer in Atlanta, GA or the surrounding area.

As a Ruby on Rails Support Engineer, you would work with the existing Rails Machine team to:

• Support existing customers’ Ruby on Rails deployments
• Deploy new customers’ Ruby on Rails applications
• Write documentation covering common problems
• Write software to automate anything you do more than twice

In addition to the above work, which can all be performed from the comfort of your home office, your back patio, our coworking space, or anywhere you can find the internets, the following work will need to be occasionally performed onsite at our two Atlanta data centers.

• Unbox, rack, cable, and configure new servers
• Perform basic server maintenance (replace hard drives, etc)

You should be comfortable with all of these things:

• Linux
• Ruby
• Rails
• Passenger
• Mongrel
• Apache
• MySQL
• Capistrano
• Git
• Subversion

You should be excited to learn about these things:

• Xen
• KVM
• Nagios
• LVS
• Puppet
• Moonshine
• Sinatra
• TDD/BDD with RSpec

Rails Machine is a small business, so this will be anything but a boring job. We all wear many hats, but have a damn good time doing it. Oh, and if you like strange beers, that helps too, as we’re running out of names for our servers.

If this sounds like fun to you, please send your resume, a link to your GitHub account, and a note explaining why you’d be perfect for the job to Jesse Newland. Thanks!

We hope by now that you’ve had a chance to check out Moonshine and maybe even give it a go on your own server. We’ve been using Moonshine to deploy our customer’s servers for a while now, and we’ve started extracting little pieces of manifests into Moonshine plugins. I’m going to show you how to add Moonshine plugins to your manifest, and give you the lowdown on writing your own.

Installing and using plugins

Moonshine plugins are installed just like any other Rails plugin:

script/plugin install git://github.com/railsmachine/moonshine_ssh.git

Then add a few lines to your application’s manifest- app/manifest/application_manifest.rb, by default.

configure( :ssh => { :allow_users => ['rails'] } )
plugin :ssh
recipe :ssh

Not all plugins will need to be configured before they’re used and the recipe name doesn’t need to be the same as the plugin name- in fact, a plugin could provide as many recipes as you like. The next time you deploy your application, Moonshine will run the recipes you specified from the installed plugins.

Writing your own

Now here’s how you can really shine while you ‘shine. If you’ve added functionality in your manifest that you think others could use, create a new Moonshine plugin for it and put it up on github. Here’s how we do it.

Generate the plugin stub

Start by running the plugin generator from your application directory. Just switch out “ssh” for whatever it is you’re plugin-ifying.

script/generate moonshine_plugin ssh

This creates the following structure in the vendor/plugins directory of your app:

  |--moonshine_ssh
     |-- README.rdoc
     |-- moonshine 
     |  `-- init.rb
     |-- lib
     |  `-- ssh.rb
     |-- spec
         |-- spec_helper.rb
         |-- ssh_spec.rb

Add recipes

The lib/ directory is where we’ll put puppet recipes. For our SSH plugin we add this:

def ssh(options = {})
  package 'ssh', :ensure => :installed
  service 'ssh', :enable => true, :ensure => :running

  file '/etc/ssh/sshd_config',
    :mode => '644',
    :content => template(File.join(File.dirname(__FILE__), '..', 'templates', 'sshd_config'), binding),
    :require => package('ssh'),
    :notify => service('ssh')
end

This is straightforward- we’re making sure that SSH is installed, that the service is running and will start on system boot. We’ll create a templates/ directory with an ERB template called sshd_config. It will be rendered and saved as /etc/ssh/sshd_config. The SSH service will be notified to restart when it changes. The template will have lines like this sprinkled throughout:

LoginGraceTime <%= options[:login_grace_time] || 30 %>
PermitRootLogin <%= options[:permit_root_login] || 'no' %>

The options come from our configure line in the first example. If you recall, we defined a property called :ssh, which we set to a hash. This hash will be passed to the ssh recipe by Moonshine. This little convention helps keep the custom configuration of your server separate from its list of parts.

Do the safety dance

Now we’ve got a working plugin that can manage your SSH settings. Just one thing- what if someone configured ssh like this?

configure(:ssh => {:port => 'two'})  

This clearly isn’t valid and we don’t want them to be locked out, so let’s test the file before we update it.

def ssh(options = {})
  package 'ssh', :ensure => :installed
  service 'ssh', :enable => true, :ensure => :running

  file '/etc/ssh/sshd_config.new',
    :mode => '644',
    :content => template(File.join(File.dirname(__FILE__), '..', 'templates', 'sshd_config'), binding),
    :require => package('ssh'),
    :notify => exec('update_sshd_config')

  exec 'cp /etc/ssh/sshd_config.new /etc/ssh/sshd_config',
    :alias => 'update_sshd_config'
    :onlyif => '/usr/sbin/sshd -t -f /etc/ssh/sshd_config.new',
    :refreshonly => true, # do nothing until notified
    :require => file('/etc/ssh/sshd_config.new'),
    :notify => service('ssh')
end

There you have it. We upload the new configuration to a temporary location and replace the previous one only if the syntax check returns 0. This plugin is ready to be put online for all to enjoy.

The first plugins

So far, we’ve released plugins for managing iptables, SSH, and god. Look for more in the coming days. If you’ve written one, let us know about it in the comments!

Update: Check out the list of plugins and add your own on the plugin wiki page.

If you’re in Vegas for RailsConf, we’d love to meet you! Jesse Newland will hanging out in the Caboose Conf room during the day, hacking on Moonshine. Feel free to drop by if you’d like to learn more about Moonshine, chat about Rails deployment, or anything else. Remember: our Ask Us Anything Support is always valid in person.

Oh, and if you’re a Rails Machine customer, Jesse would love to buy you a beer. Just @jnewland him. Mmmm beer.

We’re happy to announce the release of Moonshine, the system we’re using to make the deployment and configuration management of our new customers’ Rails applications a no brainier. Moonshine combines all of the good parts of Puppet, Shadow Puppet and Capistrano into an amazingly simple solution for deploying your Rails application.

Look at all the choices I’m not making

One of the things that separates Moonshine from other solutions like Chef and Sprinkle is that out of the box, Moonshine comes with recipes for the same Ubuntu/Ruby Enterprise Edition/Apache/Passenger/MySQL stack that’s in production use at Rails Machine. We’re open sourcing this stack as a part of Moonshine for a couple reasons:

• To make it easier for Rails developers to deploy an application to any host, not just Rails Machine.
• To allow our customers to have a say in what they want in a Rails stack. If there’s something missing or a choice we’ve made that you don’t agree with, fork Moonshine, make your changes, and send us a pull request.

Since Rails Machine was founded in 2006, we’ve been dedicated to making the deployment of Rails applications easier for our customers. With the release of Moonshine, we’re taking that one step further by making Rails application deployment and configuration management easier for everyone!

So, how do I get started?

Moonshine is distributed as a Rails plugin. To get started with Moonshine, install this plugin and run the included generator:

ruby script/plugin install git://github.com/railsmachine/moonshine.git
ruby script/generate moonshine

You’ll notice a couple new files:

app/manifests/application_manifest.rb
config/moonshine.yml

The first file is the Moonshine::Manifest for your application. A Moonshine Manifest is a Ruby class that contains ShadowPuppet recipes – essentially just instance methods – that install packages, create configuration files, and run system commands. The manifest that’s been generated for your application is actually subclass of Moonshine::Manifest::Rails, which contains the entire Rails Machine production stack as recipes.

The second file is a hash serialized to YAML that contains the configuration for your application’s deployment. You’ll notice that this file contains variables you may be used to configuring in Capistrano – the location of your source code repository, the user to execute commands on the server as, and the deploy_to location on your server. Configure these just as you would in Capistrano – this configuration hash will be available to both Capistrano and Moonshine.

If your application doesn’t have any requirements in addition to the gems specified in your config/environment.rb (you have specified all of your gem requirements using config.gem calls, right? If not, stop reading and do this now), then you’re ready to deploy your app!

Deploying your Application with Moonshine

Once you’ve installed the Moonshine plugin, generated a manifest, configured Moonshine, and committed all of these changes to your repo, you’re ready to deploy! To work with Moonshine, your server needs to satisfy these requirements:

• Ubuntu 8.10
• Has a user with sudo privileges that can access your applications’ repository. (Moonshine uses rails as the default user – this is configurable in config/moonshine.yml)

This server doesn’t need Ruby, MySQL, or anything installed on it. That’s what Moonshine is for!

Once your server has been provisioned, capify your application and replace the stock config/deploy.rb with this one-liner:

server "myubuntuserver.com", :app, :web, :db, :primary => true

Once that’s in place, run the following command:

cap deploy:setup deploy

First, in the deploy:setup task, Moonshine will install Ruby Enterprise Edition, Git, and Shadow Puppet and setup your deploy_to directory. What happens next, during the deploy task, is what makes Moonshine so awesome.

On the first deploy of your application, all requirements defined by recipes are installed, configuration files are created, and services started. In repeated tests on a Ubuntu 8.10 server running in VMWare, this first deploy takes between 10-15 minutes to install everything needed for a Rails application. You read that right – less than 15 minutes after running cap deploy:setup deploy on a stock Ubuntu 8.10 server, your Rails app will be up and running.

Idempotent Deployment

On each and every deploy of your Rails application, your Moonshine Manifest is applied and verified. This means no more failed deploys because of a missing gem or package, and more importantly, you’ll never have to SSH to your server to install a package again.

For example, say you have a Rails app in production using Moonshine and you’ve added file uploading using Paperclip. After adding a config.gems 'paperclip' line to config/environment.rb, run rake moonshine:gems to cache your gem dependencies, commit the changes to your repo, and then deploy. Moonshine will automatically install Paperclip and ImageMagick (!!!) during the deploy. Done. Awesome, right?

Brew Your Own

So the default Rails stack we’ve included is just the tip of the iceberg. As we use Moonshine more and more at Rails Machine, we’ll be adding built in support for things like Memcached to the recipes in Moonshine::Manifest::Rails. And the plugin support in Moonshine makes it ridiculously easy for you to create reusable of recipes to share between apps, or even to include Moonshine recipes as a part of your plugin! If none of you beat me to it, I’ll be forking Thinking Sphinx and pouring some Moonshine on it pretty soon.

Use the Source, Luke

So what are you waiting for? The source and RDocs are up on GitHub – go nuts.

Moonshine at Rails Machine

Moonshine is currently available for all customers of our Application Management services at Rails Machine. If you’re an existing customer on a CentOS server and are interested in using Moonshine without the expert management, 24/7 monitoring, and proactive support that comes with our Application Management services, you will be able to redeploy to a new Ubuntu-based system very soon. Stay tuned!

On Saturday, February 7th 2009, Dan Benjamin, Rails Machine Evangelist, gave the closing keynote at this year’s Acts as Conference in Orlando Florida.

Both the video and slideshow are available for your viewing pleasure below.

Slideshow

Video

While our Moonshine is still distilling, I’d like to present some examples of using existing automation tools to deploy Rails applications easily.

Taskify

When Jamis Buck released Switchtower (now known as Capistrano) around 3 years ago, automated Rails deployment become standardized. It was a historic moment in Rails history. Since then, Capistrano has been used for a number of tasks beyond code deployment related to system configuration and automation. The railsmachine gem provides tasks for configuring an application stack. However, our gem has always assumed that you were deploying against a golden image (or foilball) and only addressed configuration. Other Capistrano-based projects took the additional step of automating calls to package installers to setup an empty machine. All of these projects provided a quick way to deploy a Rails application on a new server. Despite their utility, the approach has a number of shortcomings: increased code complexity, not idempotent, difficult to model complex dependencies, only executable via cap, difficult to manage existing systems, etc. When all you have is Capistrano, everything looks like a cap task.

Automate

With the rise of the clouds, automation tools are getting a lot attention recently. Unbeknownst to most developers, sysadmins and operations folks have been using an automation tool called Puppet for the past few years. Puppet is written in Ruby and provides its own external DSL and services for managing infrastructure. In addition, the new, Puppet-inspired Chef is helping spread the gospel of automation to Rails developers. More awareness is a good thing. Automation is a best practice that all Rails developers should add to their development process.

Cheese and Crackers

By using using Puppet with Capistrano, we can use the right tools for the right job. The issue of how to partition the work between the two tools has been been recently discussed by Andrew Shafer of Reductive Labs. Although I think we can push more onto Puppet then Andrew suggests, we both agree when Adam Jacob writes: Capistrano and Puppet go together like cheese and crackers.

One of our current goals is to easily integrate server automation into every Rails deployment. The challenge is that existing tools are just frameworks and have no knowledge or opinion on deploying Rails applications. Some developers will take the time to learn the tools and others won’t. This is understandable since spending time learning about automation frameworks is time not spent developing revenue-generating features.

shadow_rails

To jump start the process, I’ve written an example called shadow_rails that can be used as a starting point for introducing automation into your existing deployment process. The code is written using ShadowPuppet to install packages and configure the system. The tricky part is that Ruby-based automation tools require ‘ruby’. Therefore, there is a little bootstrapping script to install Ruby Enterprise Edition from source or the Ubuntu 8.10 ruby packages. In addition, shadow_rails installs Apache, MySQL, Passenger, and Rails. It configures Apache, creates your database, and sets up the Capistrano directories. From the comfort of your development box, you can go from nothingness to a deployed app in one command: cap deploy:bootstrap deploy:migrations.

This project is meant to help you get going with using automation and not the final word in deployment. To get started, read the README in the GitHub repository. Clone and browse around the code to get familiar with the syntax. Fork the code and try to make it work for your application. There are even a few rspec examples written using ShadowFacter for those interested in exploring behavior driven automation.

You’ll need a fresh install of Ubuntu 8.10 with an admin user named ‘rails’ and ssh installed. I highly recommend using a virtualization tool like VMware Fusion on OS X or VMware Player on Ubuntu for testing. Using snapshots and rollbacks, testing your manifest is super easy for both clean installs and existing ones as the manifest evolves.

Feedback

Please let me know what you think and ping me with any questions. If you are a Rails Machine customer and would like to try it on a new virtual machine, please submit a ticket.

As we’d hinted at a short while back, we’ve started up a podcast, hosted by our broadcaster-in-residence, Dan Benjamin. Each episode we’ll be talking with a different member of the Rails community about software development, business, being an entrepreneur, and more.

Episode one, where Dan talks about Rails Machine’s history with Bradley Taylor, is already live. You can check it out at http://podcast.railsmachine.com, and you can subscribe directly in iTunes (or your favorite newsreader) using this RSS feed.

We’d love to get your feedback and your suggestions for interviewees. Let us know what you think in the comments.

We’ve been using Puppet here at Rails Machine to manage our internal infrastructure for quite a while now, and it’s been great solution. Unfortunately, Puppet’s external DSL can be unfamiliar to Rubyists used to other Ruby-based DSLs like Rails, Rake, and Capistrano. But fortunately, Puppet actually has had a proof-of-concept Ruby DSL since July of 2006.

Last week we announced Moonshine, an opensource configuration management and deployment system being released in February (you can download the slides from the presentation Jesse Newland gave at ATLRUG here). Rather than re-implement those parts of Puppet, we decided to improve on the DSL included in Puppet, and build upon it.

While we’re hard at work finishing up the Moonshine release, we wanted to extract the work we’ve done on the Puppet Ruby DSL and release it so that people can experiment with it right now. We’re sharing ShadowPuppet with the Puppet development community early-on with the hope of improving and extending the existing, best of breed tools that are already available today.

Enter: ShadowPuppet

ShadowPuppet is a re-implementation of Puppet::DSL designed for use by Rubyists. There’s actually not very much DSL in ShadowPuppet at all: manifests are Classes, groups of resources are instance methods, and ‘plugins’ are implemented as Modules.

You can review, checkout, and fork the ShadowPuppet project on Github, and you can see the RDoc here.

You can install the ShadowPuppet gem with the gem install shadow_puppet command.

There are a few difference worth mentioning between ShadowPuppet and Puppet itself:

1. ShadowPuppet doesn’t use a puppetmaster (Puppet’s central server), but you can use Capistrano for remote execution.
2. You configure ShadowPuppet with a manifest (like this one)
3. You execute ShadowPuppet with the included shadow_puppet binary

ShadowFacter

In addition to the Ruby DSL for Puppet, we’ve also been working to make it easier for Ruby developers to integrate the gathering of system and application facts. So today we’re also introducing ShadowFacter, a DSL wrapper around the pre-existing project, Facter, used by Puppet.

As with ShadowPuppet, you can install the gem with the gem install shadow_facter command, checkout and the ShadowFacter project on Github, and you can see the RDoc here.

Let us know what you think, we’d love your feedback on these projects.

Update: You can learn a bit more about ShadowFacter from our post in the Puppet group.