Posts Tagged CI
Continuous delivery implementations differ due to company testing culture
Posted by Morten in Uncategorized on July 25, 2011
I just finished reading a post about continuous delivery at Outbrain. It’s quite interesting they made a trip similar to what we have done at RemoteX. However their setup is some what different than in the our. They use a tool called Glu to do deploy to the their different target environments and deliver RPM’s for their services.
At RemoteX we produce .exe files as well as a set of web services so I guess there will be significant differences when looking at the actual deployment.
What I find interesting is that they seem to go directly from the builds in TeamCity to starting deployment, where we at RemoteX have several steps after the commit stage to package and verify our release before its pushed out.
Outbrain also seems to be able to target specific environments to deploy to in a greater extent than we do at RemoteX. At RemoteX we instead categories our system installations to gradually deploy to all our installations.
This brings up something that seems to be common when reading Continuous Delivery war stories. They all agree (mostly) what what Continuous Delivery is all about, but they all implement it differently and have different areas where the solution is stronger or weaker.
The pattern seems to be that once the product goes out to customers there are different cultures at the companies that requires different solutions. The example of deploying to specific environments or deploy to categories of customers for example.
These cultural differences seem to all be focused around faith in the quality of the product. With in turn boils down to faith and investment in automated testing vs. manual testing.
In theory it would be possible to see what the culture at a company is like regarding to their faith in their own deliverables, by looking at their continuous delivery solution.
Making Hudson (Jenkins) update it self
Posted by Morten in Uncategorized on March 21, 2011
A funny thing happens when you start delivering software through an automated system. You need to find a way to upgrade the automated system, and yet it is in use all of a sudden.
Luckily for us we have everything checked into Git!
So we create a job that does a Git pull in the root hudson directory, and the use curl to tell hudson to reload the configuration from disk.
Some tweaks to doing this is to create a restart job, and have the update job call that job instead of doing the reload in the same job as the Git pull operation. The reason for this is that the reload doesn’t wait for the job to finish, so the build history starts to lag by one build for the job that does the reload.
Some thoughts on triggers
Having Hudson checked in into GIT gives you a distributed build system. We have our system set up so we have a master CI server that hosts the latest version of our system on github. We can have build triggers set up to update the CI every time we push. However, due to the reload configuration we aren’t doing that instead we ask the system to update it self at specific periods. We don’t want to interrupt current builds to do an update.
Any tips on getting the restart job to wait for all other builds to finish are appreciated.
Next, we use the system for deployment as well. This means that there is a version up and running in our production environment as well. For us the manual updating by running a job works great for the production system, since we do not have to open up access to Hudson in the production environment instead letting the system update itself from github.
MSBuild projects prebuild step can fail when building with Hudson
Posted by Morten in Uncategorized on March 7, 2011
I’ve recently ran into a problem where Hudson wouldn’t compile our solution since it will fail when were doing some special work in a prebuild step.
The reason was that we had some project with a prebuild step that used the “find” command line tool. However if you install mysisgit on windows it too contains a find tool, and these two tools differ.
I added a build step that used where to see which “find” hudson was using and I quickly found out that it was indeed using the mysisgit’s version of find. This caused the following output:
C:\dev\remotex\continuous-delivery-with-hudson\hudson\home\jobs\Package\workspace>where find
C:\dev\remotex\continuous-delivery-with-hudson\hudson\home\jobs\Package\workspace\find
C:\dev\Git\bin\find.exe
c:\Windows\System32\find.exe
As you can see “where” finds two different locations for the command find, and they work differently.
To fix this I had to change all our post build tasks to run with the specific find tool I wanted to use.
Instead of calling just “find” in my prebuild step I had to specify it like this:
%COMSPEC:cmd=find%
Thanks to Johan Andersson whom came up with the idea.
Git plugin for Hudson hangs when fetching
Posted by Morten in Uncategorized on February 28, 2011
I had issues with getting Git up and running in a Hudson that was hosted as a Windows Service. The Git Commands would simply hang, the console output stopped at the line where the Git command was executed.
If I took the service down and started hudson from the command-line everything worked out great.
It turns out that when Hudson is started as a windows service there is no HOME environment variable set. And this is where Git finds the keys for SSH identification.
I found information regarding this here:
http://stackoverflow.com/questions/3188710/hudson-git-plugin-not-working-on-windows
http://stackoverflow.com/questions/3182012/problem-with-hudson-git-gitosis-on-windows
The fix for me was to make a simple change to the hudson.xml that is located in HUDSON_HOME.
<id>hudson</id> <name>Hudson</name> <description>This service runs Hudson continuous integration system.</description> <env name="HUDSON_HOME" value="%BASE%"/> <env name="HOME" value="C:\Documents and Settings\hudson"/>
This adds the environment variable for the home directory when the service starts up. Now Git can fetch when I run Hudson as a service. Naturally you have to change the path to correspond to the location of the .shh/ directory to use. In our case there is a dedicated user running hudson.
Creating ZIP files using Powershell
Posted by Morten in Uncategorized on February 14, 2011
I’ve been working with a RemoteX version of the Continuous Delivery pipeline that is available at github. At RemoteX we have all our deployment done using PowerShell so to maintain as much familiarity as possible between builds and deploys were using the PowerShell plugin for Hudson (Jenkins).
The set up that we are working towards is this. The CI builds creates output. There is a packaging build that takes the CI output and creates a deployable package and that starts the rest of the pipeline. You can see the structure as it is outlined in the github project.
Now from the package I want a zip file as the delivery. It’s suiting to have an archive as an artifact that is tested and added on in the later steps of the pipeline.
The trouble is that PowerShell has no simple available command-line functions for creating and managing zip files. Not out of the box. In fact any archiving would be appreciated.
These are the solutions I looked at for working with zip-archives using PowerShell:
- David Aiken did an article on how to compress zip files in PowerShell using COM interop.
- This didn’t work out for. The size and amount of files in our archive was simply to much, a bunch of popup started to popup with read/write access issues to the zip-archive.
- Another example I found was using .Net and ICSharpCode.SharpZipLib to create zip files.
- This solution caused a lot of problems when I tried to dot-source the PowerShell files. The path to the dll screwed up and I didn’t want to introduce any hardcoded paths.
- Another solution is to use the Community PowerShell Extensions. These extensions have a lot of options for common tasks, one of them being the handling of zip-files.
- This wasn’t an option to us since it would require the entire extension suite to be checked in in source-control. I didn’t want to have our delivery pipeline reference an entire collection of functions for just working with zip files.
- Last solution was to introduce 7zip’s command line program for managing zip-files. I tried GZip first but there were issues with file system access on Windows 7.
- To us this was a perfect fit. It turned zip-handling into one line commands and there were not issues with dot-sourcing in PowerShell.
- I checked it in under hudson/bin to keep a collection of tools that we use with our pipeline.
- To us this was a perfect fit. It turned zip-handling into one line commands and there were not issues with dot-sourcing in PowerShell.
Book review: Continuous Delivery
Posted by Morten in Uncategorized on February 8, 2011
I recently finished reading Continuous Delivery by Jez Humble and David Farley. I have to say that the book was quite exiting. The first parts showed how to implement a continuous delivery pipeline and where automated tests fit in. It also explained how to setup the automated tests and the importance of continuously testing your applications.
The book however lost some of it’s flare at the end. The last chapters describing how to divide your application into components, manage continuous delivery and source control didn’t quite live up to the same quality earlier in the book. These chapters were a bit vague and repeated points made in earlier chapters.
I still recommend reading this book. To be honest I think this book gave me a better picture of how to fit acceptance tests into development than for example the Agile Testing book gave me. The more hands on practical approach to Continuous Delivery gives the option of first setting up the development environment to support acceptance tests, and then add the tests.
This being said the book was so inspiring that I started a template project to implement a continuous delivery pipeline using Hudson and Git. You can find it as a project on github. We’re adopting this template to be used at RemoteX.
I recommend this book to any team that wants to improve their build and delivery models.
Deploying updates for RemoteX Applications using Hudson
Posted by Morten in Uncategorized on January 24, 2011
We’ve been doing some work with Hudson over the past weeks. Mainly We’ve managed to create the last part of RemoteX delivery pipeline on a hudson build-server.
Since the first release of RemoteX Applications we’ve been releasing each release as a “package” containing everything needed to setup applications for a customer. I briefly mentioned this in my previous post.
In the beginning the final delivery was done by our operations dude. However as we’ve deployed more and more versions it quickly became apparent that he could not keep up without assistance. So we start to create more and more elaborate scripts to set up each customer. Today everything is done through the execution of a single script that sets everything up using other modular scripts.
What we’ve done over these last week is to set up a Hudson server to do the final delivery automatically. I calls this central script with its arguments and deploys our upgrades to all our customers.
We set up some builds that will connect to RemoteX Applications using PowerShell and create deployment jobs to upgrade different installations based on different criteria.
Next we started to categories the different installations for different targets. We have three different targets that does automatic deployment:
- A testing target that sets up the release for manual testing
- A “early adopters” target who get the version that passes the manual testing
- The bulk, which consists of all customers who will be upgraded automatically
As we work through a release we will have many deploys to the testing installations, a few more to the early adopters, and once we have a final release it will be rolled out on the bulk of our customers. All this is done with the click of a button.
With a single Hudson service deploying a new version of RemoteX Applications to the bulk should take roughly 3 hours. Each customer should have only a few seconds of service interruption during the process, but might be required to upgrade their clients. This process can be parallelized by adding Hudson slaves to run several deployments at once. Effectively cutting the deployment time by O(1/N) where N is the number of build executors available.
We can make it faster but that requires some infrastructure work, and this will work sufficiently for the coming months.
Doing Continuous Delivery with Hudson and Git
Posted by Morten in Uncategorized on January 17, 2011
Right now I’m reading the book Continuous Delivery. Excellent reading and even though I’ve only read around 46% percent of it, it’s quite thought provoking.
I’ve written earlier on how we at RemoteX have set up our build environment with a concept I then called The Independent Trunk (terrible name, right?). Continuous Delivery describes a similar approach of handling your source-control and build system, though in greater detail.
Recently we’ve been considering switching build environments to Hudson, since we’ve been quite happy with the results Hudson has been given us when we’ve tried it out for different scenarios.
Hudson however does not have its configuration checked in under source-control. Most people seem to be happy with just doing backups of the configuration with the motivation that it seldom changes. However I firmly believe that the configuration of the build-environment follows the same lifecycle as the software, and it seems the authors of the Continuous Delivery book do as well.
So I set out to do some experiments with Hudson to see if I could model an arbitrary delivery pipeline using Hudson and yet have it checked in.
I wanted the following to hold:
- All configuration is checked in
- Developers run the same builds locally as they do on the build-server
- It’s easy to run the complete pipeline on a developer machine
- It should be fast to get up and going
I started thinking about this in the middle of December and about a week ago I started to set things up. The idea I got was to simply check the entire Hudson system into source-control. I put the war-file that is executed in source-control along with Hudson’s home directory. I spent quite some time tinkering with the .gitignore file to get things set up how I wanted it.
The result is available on git-hub at: https://github.com/morkeleb/continuous-delivery-with-hudson feedback is highly appreciated. Also if anyone wants to use it as a boilerplate for new projects go right ahead.
The only difference in the pipeline I’ve created and the one that the book describes is that I’ve added a packaging step that is supposed to take all artifacts from the CI builds and add them up into a deliverable package. There are two stub CI builds creating files and shows the intended structure.
To make it easy to start Hudson on a developer machine I created a .bat file that starts the CI engine locally. All a developer has to do is get the latest version and run the script to have Hudson setup locally.
What I’ve found thus far is that this approach seems to work quite well. Sure it takes some disc space but that is affordable.
Upgrading Hudson can be done easily and have the changes replicated across developer machines as the fetch the latest version from the origin.
Now there are some things that differ when you run Hudson locally and on the main build-server. The main build-server will have all its artifacts and fingerprints backed up, as well as following the source-control version.
There are however some things that I’ve yet to try out. Changes to the configuration needs to be updated on the build machine, this can be accomplished by logging on to the machine and the do a git pull. However since this is tiresome a web hook approach could be interesting for automatically restarting Hudson and getting the latest version of the configuration from source-control. This however is yet to be experimented with.
Feel free to check it out on git-hub and any feedback is great. Especially considering that this might seem a bit crazy.
Hudson build server + Grails + AppEngine
Posted by Morten in Uncategorized on February 22, 2010
I just set up a build server for my Grails project. I chose Hudson since it was the one mentioned to work well, and it did. I’m quite happy with Hudson thus far. It hasn’t crashed (actually it crashed my browser once, more on that later), and been very easy to work with.
First of I installed it on my Ubuntu machine in the closet. It was straight forward installed using apt-get as per instruction. This was a clean install and the first use of that Ubuntu installation for any kind of development work.
With Hudson installed I started going through its configuration. I knew there was a Grails plug-in so I figured I had to install a lot more software to get things going.
Imagine my surprise when I found out I didn’t even have to install the JDK. All I had to do was to choose which JDK I wanted, accept the terms for usage, and Hudson installed it for me.
I found the plug-in page, marked the Grails plug-in and installed it.
I created a job, configured it to fetch the code from my subversion repository. Already I could run the job, and watch the output from my normal development machine.
I love the console out page, compared to watching the build-logs and build progress in MSBuild this page gives good feedback, doesn’t crash that much, and works as both progress and log all wrapped up in one.
With the job running I pinpointed the missing parts in the job. Running the grails targets. So it was time to configure the Grails Hudson plug-in. It required the Grails installation path, so time to set up the libraries. Download and install Grails, update the plug-in with the path.
When running a Grails job the first time you need to run the “grails compile –non-interactive” command. This will install all other plug-ins as it is the Grails way. This will make plug-in specific commands available.
I ran it first without the –non-interactive flag, which caused the app-engine plug-in installation to ask questions. Hudson didn’t mind, but my browser in which I was looking at the console out didn’t like it. It kept updating the page with “do you want to use JPA or JDO” questions, until it finally crashed.
In my case I wanted my build to upload the compiled application to Google AppEngine. I added the APPENGINE_HOME environment variable to Hudson’s main configuration. With the APPENGINE_HOME variable set to my newly downloaded SDK.
This is what my grails build target looks like:
To get the deploy command to work I needed to add the login details for app-engine. After some research I found that the details are configurable. All you have to do is add the following to you Config.groovy file in the grails project.
google.appengine.email = "email" google.appengine.password ="password"
But with the correct email and password of course. In my case I used the user I’ve created for sending emails as the user that will deploy using the build.
I now have a working build that deploys to AppEngine. What I’m currently missing are running tests, as I’m still searching for a functional testing framework that will run as a Grails plugin with the AppEngine plugin. None of the two (easyb, functional-testing) Grails plugins I’ve tried so far have worked.