Posts Tagged Hudson
PowerShell logs even caught errors
Posted by Morten in Uncategorized on July 18, 2011
An interesting PowerShell feature is that all errors are collected in an array called $error.
so for example if you throw an error you can echo $error and find it later:
However the error is also logged even if you catch it. Consider the following code snippet, which I’ve called errorcollectiontest.ps1
When running the script it outputs nothing, but when we check the errors array we find something interesting:
What does this matter?
Well if you have a system running PowerShell a convenient way to see if there were errors in the execution of a script is to check the $error array for any errors. However this array will also contain errors that were caught during the execution, which will introduce false-positives.
The PowerShell plugin for Hudson/Jenkins does exactly this, so if you want to use a PowerShell script that catches errors in Hudson/Jenkins the way to prevent false positive build results is to make the following change to the script:
This will remove the latest occurred error from the list, which just happens to be the error just caused since we entered the catch block.
I’d like to mention Johan Andersson, in this post, since without him it wouldn’t be written.
Job dependencies in Hudson for our Continuous Delivery effort
Posted by Morten in Uncategorized on May 30, 2011
I’m continuing our work with our Continuous Delivery effort. Were migrating from our previous build system to Hudson since it makes builds much more accessible.
The work started with the continuous delivery with hudson project I have on github and were slowly adapting a fork of that to work with out scenarios.
We realized that the amount of builds that we will be hosting in Hudson might make it hard to get an overview of what depends on what. After some research I found out that you could add the following to the URL of your hudson to get a dependency graph of all the builds:
/dependencyGraph/graph
So if you host Hudson locally it would look like:
http://localhost:8080/dependencyGraph/graph
This is how our current dependency graphs looks like:
The package output build takes the result from our old package build and migrates it over to the delivery pipeline. The Template Schedule Deploy job (which I now realize is a bad name) takes a set of “job files” and run the deploy installation job with the given settings.
The Create job files based on search creates the job files that the template deploy job works with, and the upgrade early, test, and bulk installations does a search in our installation of RemoteX Applications to find out which installations to upgrade based on the units in RemoteX Applications.
Each job then run the deploy installation build (not visible in the graph) to deploy the upgrade for each customer. In essence running the same build for upgrading that we use every time we deploy anything.
Making Hudson (Jenkins) update it self
Posted by Morten in Uncategorized on March 21, 2011
A funny thing happens when you start delivering software through an automated system. You need to find a way to upgrade the automated system, and yet it is in use all of a sudden.
Luckily for us we have everything checked into Git!
So we create a job that does a Git pull in the root hudson directory, and the use curl to tell hudson to reload the configuration from disk.
Some tweaks to doing this is to create a restart job, and have the update job call that job instead of doing the reload in the same job as the Git pull operation. The reason for this is that the reload doesn’t wait for the job to finish, so the build history starts to lag by one build for the job that does the reload.
Some thoughts on triggers
Having Hudson checked in into GIT gives you a distributed build system. We have our system set up so we have a master CI server that hosts the latest version of our system on github. We can have build triggers set up to update the CI every time we push. However, due to the reload configuration we aren’t doing that instead we ask the system to update it self at specific periods. We don’t want to interrupt current builds to do an update.
Any tips on getting the restart job to wait for all other builds to finish are appreciated.
Next, we use the system for deployment as well. This means that there is a version up and running in our production environment as well. For us the manual updating by running a job works great for the production system, since we do not have to open up access to Hudson in the production environment instead letting the system update itself from github.
MSBuild projects prebuild step can fail when building with Hudson
Posted by Morten in Uncategorized on March 7, 2011
I’ve recently ran into a problem where Hudson wouldn’t compile our solution since it will fail when were doing some special work in a prebuild step.
The reason was that we had some project with a prebuild step that used the “find” command line tool. However if you install mysisgit on windows it too contains a find tool, and these two tools differ.
I added a build step that used where to see which “find” hudson was using and I quickly found out that it was indeed using the mysisgit’s version of find. This caused the following output:
C:\dev\remotex\continuous-delivery-with-hudson\hudson\home\jobs\Package\workspace>where find
C:\dev\remotex\continuous-delivery-with-hudson\hudson\home\jobs\Package\workspace\find
C:\dev\Git\bin\find.exe
c:\Windows\System32\find.exe
As you can see “where” finds two different locations for the command find, and they work differently.
To fix this I had to change all our post build tasks to run with the specific find tool I wanted to use.
Instead of calling just “find” in my prebuild step I had to specify it like this:
%COMSPEC:cmd=find%
Thanks to Johan Andersson whom came up with the idea.
Git plugin for Hudson hangs when fetching
Posted by Morten in Uncategorized on February 28, 2011
I had issues with getting Git up and running in a Hudson that was hosted as a Windows Service. The Git Commands would simply hang, the console output stopped at the line where the Git command was executed.
If I took the service down and started hudson from the command-line everything worked out great.
It turns out that when Hudson is started as a windows service there is no HOME environment variable set. And this is where Git finds the keys for SSH identification.
I found information regarding this here:
http://stackoverflow.com/questions/3188710/hudson-git-plugin-not-working-on-windows
http://stackoverflow.com/questions/3182012/problem-with-hudson-git-gitosis-on-windows
The fix for me was to make a simple change to the hudson.xml that is located in HUDSON_HOME.
<id>hudson</id> <name>Hudson</name> <description>This service runs Hudson continuous integration system.</description> <env name="HUDSON_HOME" value="%BASE%"/> <env name="HOME" value="C:\Documents and Settings\hudson"/>
This adds the environment variable for the home directory when the service starts up. Now Git can fetch when I run Hudson as a service. Naturally you have to change the path to correspond to the location of the .shh/ directory to use. In our case there is a dedicated user running hudson.
Creating ZIP files using Powershell
Posted by Morten in Uncategorized on February 14, 2011
I’ve been working with a RemoteX version of the Continuous Delivery pipeline that is available at github. At RemoteX we have all our deployment done using PowerShell so to maintain as much familiarity as possible between builds and deploys were using the PowerShell plugin for Hudson (Jenkins).
The set up that we are working towards is this. The CI builds creates output. There is a packaging build that takes the CI output and creates a deployable package and that starts the rest of the pipeline. You can see the structure as it is outlined in the github project.
Now from the package I want a zip file as the delivery. It’s suiting to have an archive as an artifact that is tested and added on in the later steps of the pipeline.
The trouble is that PowerShell has no simple available command-line functions for creating and managing zip files. Not out of the box. In fact any archiving would be appreciated.
These are the solutions I looked at for working with zip-archives using PowerShell:
- David Aiken did an article on how to compress zip files in PowerShell using COM interop.
- This didn’t work out for. The size and amount of files in our archive was simply to much, a bunch of popup started to popup with read/write access issues to the zip-archive.
- Another example I found was using .Net and ICSharpCode.SharpZipLib to create zip files.
- This solution caused a lot of problems when I tried to dot-source the PowerShell files. The path to the dll screwed up and I didn’t want to introduce any hardcoded paths.
- Another solution is to use the Community PowerShell Extensions. These extensions have a lot of options for common tasks, one of them being the handling of zip-files.
- This wasn’t an option to us since it would require the entire extension suite to be checked in in source-control. I didn’t want to have our delivery pipeline reference an entire collection of functions for just working with zip files.
- Last solution was to introduce 7zip’s command line program for managing zip-files. I tried GZip first but there were issues with file system access on Windows 7.
- To us this was a perfect fit. It turned zip-handling into one line commands and there were not issues with dot-sourcing in PowerShell.
- I checked it in under hudson/bin to keep a collection of tools that we use with our pipeline.
- To us this was a perfect fit. It turned zip-handling into one line commands and there were not issues with dot-sourcing in PowerShell.
Deploying updates for RemoteX Applications using Hudson
Posted by Morten in Uncategorized on January 24, 2011
We’ve been doing some work with Hudson over the past weeks. Mainly We’ve managed to create the last part of RemoteX delivery pipeline on a hudson build-server.
Since the first release of RemoteX Applications we’ve been releasing each release as a “package” containing everything needed to setup applications for a customer. I briefly mentioned this in my previous post.
In the beginning the final delivery was done by our operations dude. However as we’ve deployed more and more versions it quickly became apparent that he could not keep up without assistance. So we start to create more and more elaborate scripts to set up each customer. Today everything is done through the execution of a single script that sets everything up using other modular scripts.
What we’ve done over these last week is to set up a Hudson server to do the final delivery automatically. I calls this central script with its arguments and deploys our upgrades to all our customers.
We set up some builds that will connect to RemoteX Applications using PowerShell and create deployment jobs to upgrade different installations based on different criteria.
Next we started to categories the different installations for different targets. We have three different targets that does automatic deployment:
- A testing target that sets up the release for manual testing
- A “early adopters” target who get the version that passes the manual testing
- The bulk, which consists of all customers who will be upgraded automatically
As we work through a release we will have many deploys to the testing installations, a few more to the early adopters, and once we have a final release it will be rolled out on the bulk of our customers. All this is done with the click of a button.
With a single Hudson service deploying a new version of RemoteX Applications to the bulk should take roughly 3 hours. Each customer should have only a few seconds of service interruption during the process, but might be required to upgrade their clients. This process can be parallelized by adding Hudson slaves to run several deployments at once. Effectively cutting the deployment time by O(1/N) where N is the number of build executors available.
We can make it faster but that requires some infrastructure work, and this will work sufficiently for the coming months.
Doing Continuous Delivery with Hudson and Git
Posted by Morten in Uncategorized on January 17, 2011
Right now I’m reading the book Continuous Delivery. Excellent reading and even though I’ve only read around 46% percent of it, it’s quite thought provoking.
I’ve written earlier on how we at RemoteX have set up our build environment with a concept I then called The Independent Trunk (terrible name, right?). Continuous Delivery describes a similar approach of handling your source-control and build system, though in greater detail.
Recently we’ve been considering switching build environments to Hudson, since we’ve been quite happy with the results Hudson has been given us when we’ve tried it out for different scenarios.
Hudson however does not have its configuration checked in under source-control. Most people seem to be happy with just doing backups of the configuration with the motivation that it seldom changes. However I firmly believe that the configuration of the build-environment follows the same lifecycle as the software, and it seems the authors of the Continuous Delivery book do as well.
So I set out to do some experiments with Hudson to see if I could model an arbitrary delivery pipeline using Hudson and yet have it checked in.
I wanted the following to hold:
- All configuration is checked in
- Developers run the same builds locally as they do on the build-server
- It’s easy to run the complete pipeline on a developer machine
- It should be fast to get up and going
I started thinking about this in the middle of December and about a week ago I started to set things up. The idea I got was to simply check the entire Hudson system into source-control. I put the war-file that is executed in source-control along with Hudson’s home directory. I spent quite some time tinkering with the .gitignore file to get things set up how I wanted it.
The result is available on git-hub at: https://github.com/morkeleb/continuous-delivery-with-hudson feedback is highly appreciated. Also if anyone wants to use it as a boilerplate for new projects go right ahead.
The only difference in the pipeline I’ve created and the one that the book describes is that I’ve added a packaging step that is supposed to take all artifacts from the CI builds and add them up into a deliverable package. There are two stub CI builds creating files and shows the intended structure.
To make it easy to start Hudson on a developer machine I created a .bat file that starts the CI engine locally. All a developer has to do is get the latest version and run the script to have Hudson setup locally.
What I’ve found thus far is that this approach seems to work quite well. Sure it takes some disc space but that is affordable.
Upgrading Hudson can be done easily and have the changes replicated across developer machines as the fetch the latest version from the origin.
Now there are some things that differ when you run Hudson locally and on the main build-server. The main build-server will have all its artifacts and fingerprints backed up, as well as following the source-control version.
There are however some things that I’ve yet to try out. Changes to the configuration needs to be updated on the build machine, this can be accomplished by logging on to the machine and the do a git pull. However since this is tiresome a web hook approach could be interesting for automatically restarting Hudson and getting the latest version of the configuration from source-control. This however is yet to be experimented with.
Feel free to check it out on git-hub and any feedback is great. Especially considering that this might seem a bit crazy.
Rolling back failed AppEngine deploys in Hudson
Posted by Morten in Uncategorized on October 30, 2010
In a previous post I mentioned how to set up a build that deployed to AppEngine using Hudson. I’ve been using such a build for a while for JS-Analytics but its frustrating when the deploy operation fails midway.
When this happens you need to issue a Rollback command at the same place you started to deploy form, else AppEngine will say that there is an update in progress. Meaning all builds will fail until it is corrected.
Until recently I’ve been logging into my build server to correct this problem every time it happens. However now I’ve set up another Hudson build to handle the rollback, a clean-up build if you wish.
I set the build to execute in the same workspace as the build that deploys to AppEngine and all the build does currently is to issue the rollback command.
I blurred out the part that contains the password, but as you can see it’s still pretty straight forward.
Now every time my deploy build starts acting up, I simply run my rollback build which takes a few seconds. Then I’m good to go again.
Hudson build server + Grails + AppEngine
Posted by Morten in Uncategorized on February 22, 2010
I just set up a build server for my Grails project. I chose Hudson since it was the one mentioned to work well, and it did. I’m quite happy with Hudson thus far. It hasn’t crashed (actually it crashed my browser once, more on that later), and been very easy to work with.
First of I installed it on my Ubuntu machine in the closet. It was straight forward installed using apt-get as per instruction. This was a clean install and the first use of that Ubuntu installation for any kind of development work.
With Hudson installed I started going through its configuration. I knew there was a Grails plug-in so I figured I had to install a lot more software to get things going.
Imagine my surprise when I found out I didn’t even have to install the JDK. All I had to do was to choose which JDK I wanted, accept the terms for usage, and Hudson installed it for me.
I found the plug-in page, marked the Grails plug-in and installed it.
I created a job, configured it to fetch the code from my subversion repository. Already I could run the job, and watch the output from my normal development machine.
I love the console out page, compared to watching the build-logs and build progress in MSBuild this page gives good feedback, doesn’t crash that much, and works as both progress and log all wrapped up in one.
With the job running I pinpointed the missing parts in the job. Running the grails targets. So it was time to configure the Grails Hudson plug-in. It required the Grails installation path, so time to set up the libraries. Download and install Grails, update the plug-in with the path.
When running a Grails job the first time you need to run the “grails compile –non-interactive” command. This will install all other plug-ins as it is the Grails way. This will make plug-in specific commands available.
I ran it first without the –non-interactive flag, which caused the app-engine plug-in installation to ask questions. Hudson didn’t mind, but my browser in which I was looking at the console out didn’t like it. It kept updating the page with “do you want to use JPA or JDO” questions, until it finally crashed.
In my case I wanted my build to upload the compiled application to Google AppEngine. I added the APPENGINE_HOME environment variable to Hudson’s main configuration. With the APPENGINE_HOME variable set to my newly downloaded SDK.
This is what my grails build target looks like:
To get the deploy command to work I needed to add the login details for app-engine. After some research I found that the details are configurable. All you have to do is add the following to you Config.groovy file in the grails project.
google.appengine.email = "email" google.appengine.password ="password"
But with the correct email and password of course. In my case I used the user I’ve created for sending emails as the user that will deploy using the build.
I now have a working build that deploys to AppEngine. What I’m currently missing are running tests, as I’m still searching for a functional testing framework that will run as a Grails plugin with the AppEngine plugin. None of the two (easyb, functional-testing) Grails plugins I’ve tried so far have worked.