Posted by kohsuke
on May 18, 2009 at 4:27 PM PDT
I released Hudson EC2 plugin, which enables you to use EC2 as Hudson slaves on-demand, without human intervention.
Continuous Integration often requires a heterogeneous environments; for example, the GlassFish build requires Linux, Solaris, and Windows, and the JDK build requires something like 10 different environments, each carefully created so that we can test what we need to test.
Unfortunately, heterogeneous environments reduce the resource utilization — you can easily have some Windows slaves idle while all the Solaris slaves are busy, just because you happen to have more jobs that require Solaris. So doing some kind of virtualization makes a lot of sense.
Similarly, CI jobs tend to be spiky. When you need computring resources, you need them a lot (think about a month before a release, where all kinds of tests run like crazy for almost every commit), but when you don't need them, you don't. These charateristics of CI makes it a great fit with the cloud computing technology. You can have a good resource utilization and faster turn-around time (if your tests can run in parallel.)
Toward this end, I just released the Hudson EC2 plugin . This plugin enables Hudson to automatically provision new instances on EC2, based on the system demand. That is, if Hudson notices that your system is overloaded, it will provision new slaves on EC2, and when those instances go unused for a certain time period, it will shut them down. You can run all your slaves on EC2 if you want, or you can maintain your local build cluster and use EC2 as a reserve capacity.
And here's how to use the plugin. First, go to EC2 and sign up for the service. Then enter the username/password information. Because of the way EC2 works, you also need to have an RSA private key. If you have already been using EC2 and have your own key, you can paste it here. Otherwise, you can have Hudson generate one. If you let Hudson generate one, you should save this private key in your file system as well, as you'll need this to interactively logon to EC2 instances. Use "Test Connection" button to verify that Hudson can successfully talk to EC2.
Next, configure AMIs that you want to launch. For this, you need to find the AMI IDs for the OS of your choice. ElasticFox is a good tool for doing that, but there are a number of other ways to do it . Hudson can work with any Unix AMIs. Windows is currently unsupported.
Configuring labels allow Hudson to pick the right AMI to start. For example, if all your existing slaves labeled "solaris" are fully busy and you have more builds that are tied to the "solaris" label, Hudson will start the AMIs that have the "solaris" label.
Init script is the shell script to be run on the newly launched EC2 instance, before Hudson starts launching a slave agent. If the AMI doesn't have Java pre-installed, you need to do so in this script. This is also a good place to install additional packages that you need for your builds and tests.
The load monitoring part is in the core, so the EC2 plugin really just contains some glue code that talks to EC2 to provision a new system. This means we can easily write simialar plugins for different clouds and virtualization technologies. For example, someone in the community is writing a plugin that do this with VMWare labmanager. VirtualBox lets us do something similar, with its web services (for which I wrote a JAX-WS client library some time ago, so might be fun to eat my own dogfood, so to speak.) Vivek and I also installed Eucalyptus on our spare machines, so I hope to be able to extend this plugin to work with that, too.
I'm going to talk more about this (and other things) in my JavaOne talk, so if you are coming to JavaOne, please consider dropping by!