Posts Tagged Python

Setting up CI on Hudson for Python based AppEngine apps

Posted by Morten in Uncategorized on September 13, 2010

This post describes how I for JS-Analytics use a Hudson build server for Continues Integration, publishing to Google AppEngine on successful builds.

The build currently does just two things:

  • It runs all my unit-tests and calculate code coverage on them. This step is to ensure that the application isn’t broken before the second step is executed.
  • It publishes the latest changes to Google AppEngine

Running the unit tests

imageThis is a straight forward execution of my unit tests. With some assisting tools. I use XML-runner to generate xml reports that Hudson can read to produce some nice output, and I use Coverage to calculate the metrics.

To get coverage to run I had to install it manually for python2.5. To run the coverage tool I run the following build step. 

#!/bin/bash -ex
cd jsAnalytics
coverage run manage.py test
coverage xml --omit=/usr/

It calculates the code covered during the test-run and omits the result as XML.

I then have the following setting set in the build to publish the results:

image

 

and

image

 

Publishing to AppEngine

#!/bin/bash -ex
rm ~/.appcfg_cookies

cd jsAnalytics

echo <password> | python2.5 .google_appengine/appcfg.py -e <user email> --passin update .

The above snippet first removes the appcfg_cookies from the user directory. This is to allow builds the publish to different AppEngine projects to run on the same Hudson installation, and can be omitted if you wish.

The snippet then executes the appcfg.py update command with a user and passes a password. The user for me is a special Google Apps account created for publishing to AppEngine.

What the build gives me

Basically I’m just looking at two trends, I want the amount of tests to increase and I monitor the coverage to ensure I’ve not added something without tests. The only problem right now is that the code coverage report is somewhat tainted with the app_engine helper files. Something I’m not too fond of.

That and the added value that my check-ins are continually evaluated, and the next version is always available on AppEngine.

,

3 Comments

Running Django tests automatically on save in Eclipse

Posted by Morten in Uncategorized on April 14, 2010

I’m re-installing my environment, while doing so I’m fixing a few things I’ve been missing in my environment.

Running tests using Python is really fast, so I wanted to add something like Autospec but for my tests. So I did the following:

  1. Right click the project you want to use
  2. Click properties
  3. Click on the builders item in the left pane
  4. Press New
  5. Select to add a Program builder
  6. Set python as program to run, your workspace as working directory and the command to run the tests as arguments. In my case I run tests using “manage.py test” from Django.image
  7. On the Build Options page check all the options for when to run the builderimage
  8. Now every time I save a file, my test command is run in the console. I get the output in the console window which I’ve moved to the right of my editor,window.image
    Hope it helps someone, the are probably improvements to this. You are welcome to suggest them in the comments section.

,

1 Comment

Grails vs. Python on Google AppEngine

Posted by Morten in Uncategorized on April 7, 2010

A few weeks ago I got fed up with the bad performance I was getting on Google AppEngine for my Grails project. I was also having trouble figuring out how to combine references for Entities to build more complex structures.

As a result I finally lost it and started experimenting with Python on Google AppEngine. So here are some of my conclusions.

Framework

I still wanted to have a framework to build on for developing my application. Since Grails has a lot of dependencies that are sent to GAE I wanted something that was already supported by GAE. Partially because I heard that sending all these dependencies around in Google’s datacenter was one of the reasons for the bad performance I was experiencing.

So I chose Django. The Django framework is already installed on the web servers hosting the applications on Google AppEngine, which would reduce the load cost for dependencies.

There is also a Django helper that will help you use GAE features through the normal Django API.

Static files

Something I was missing from the Grails solution was the use of static files, most of the client side java script I server were static, so I just wanted a high expiration date and static serving of the files. This is inherent in Python GAE app.yaml file.

To be honest I am quite impressed by what I can do with the app.yaml file. I can set expiration dates on content, declare parts of the urls I define as secure. I can even reserve some parts of my site for me personally. Allowing me to build Administration interfaces for the application.

I will however miss the UI-performance plugin, as it handles my java script files the way I want, potentially even better. Now I have to manually set up something that during deployment compress and bundle my java script files.

Working with the API

Working with the python API for GAE gave me a better sense of control over what I was doing. The concepts of the Datastore and other features made much more sense. All of a sudden making applications with more complex data structure wasn’t an issue. How things connect to each other was inherent and the documentation was there to assist me when I missed something.

I also had to write less code to achieve the same features, however part of this could be because I had already solved the problem once and could produce a better solution with this.

What I really liked was testing, where I earlier had had problems with getting tests to run for Grails with the AppEngine plugin. The tests not only worked out of the box, but they took roughly 0.1 second to run 10 tests.

Performance

So finally, the performance part. Remember this was one of the reasons I made the switch.

Before I get to the actual numbers, I have to mention that there are some changes that affect the results. When hosting on Python I could serve most of my pages as static HTML files, where using Grails I was depending on the virtual machine to boot up before content was delivered. Performance for the static files could probably be achieved if the Java app is configured to serve the files as static files through GAE.

And now the numbers

Avg. Response time
Grails 4985 ms
Python 252 ms
Uptime
Grails 53.48%
Python 100%

These numbers are captured using Pingdom, with checks running every 5 minutes on the website from all over the world.

Conclusion

Since I don’t require any plugins for Grails for this project and Python serving so well I’ll go with Python for this project.

This being said as I know Grails will get performance improvements for Google AppEngine in coming versions.

, , ,

6 Comments