Building Blocks

A few weeks ago I got fed up with the bad performance I was getting on Google AppEngine for my Grails project. I was also having trouble figuring out how to combine references for Entities to build more complex structures.

As a result I finally lost it and started experimenting with Python on Google AppEngine. So here are some of my conclusions.

Framework

I still wanted to have a framework to build on for developing my application. Since Grails has a lot of dependencies that are sent to GAE I wanted something that was already supported by GAE. Partially because I heard that sending all these dependencies around in Google’s datacenter was one of the reasons for the bad performance I was experiencing.

So I chose Django. The Django framework is already installed on the web servers hosting the applications on Google AppEngine, which would reduce the load cost for dependencies.

There is also a Django helper that will help you use GAE features through the normal Django API.

Static files

Something I was missing from the Grails solution was the use of static files, most of the client side java script I server were static, so I just wanted a high expiration date and static serving of the files. This is inherent in Python GAE app.yaml file.

To be honest I am quite impressed by what I can do with the app.yaml file. I can set expiration dates on content, declare parts of the urls I define as secure. I can even reserve some parts of my site for me personally. Allowing me to build Administration interfaces for the application.

I will however miss the UI-performance plugin, as it handles my java script files the way I want, potentially even better. Now I have to manually set up something that during deployment compress and bundle my java script files.

Working with the API

Working with the python API for GAE gave me a better sense of control over what I was doing. The concepts of the Datastore and other features made much more sense. All of a sudden making applications with more complex data structure wasn’t an issue. How things connect to each other was inherent and the documentation was there to assist me when I missed something.

I also had to write less code to achieve the same features, however part of this could be because I had already solved the problem once and could produce a better solution with this.

What I really liked was testing, where I earlier had had problems with getting tests to run for Grails with the AppEngine plugin. The tests not only worked out of the box, but they took roughly 0.1 second to run 10 tests.

Performance

So finally, the performance part. Remember this was one of the reasons I made the switch.

Before I get to the actual numbers, I have to mention that there are some changes that affect the results. When hosting on Python I could serve most of my pages as static HTML files, where using Grails I was depending on the virtual machine to boot up before content was delivered. Performance for the static files could probably be achieved if the Java app is configured to serve the files as static files through GAE.

And now the numbers

These numbers are captured using Pingdom, with checks running every 5 minutes on the website from all over the world.

Conclusion

Since I don’t require any plugins for Grails for this project and Python serving so well I’ll go with Python for this project.

This being said as I know Grails will get performance improvements for Google AppEngine in coming versions.

I was browsing around the issue lists on Google AppEngine, just some midnight light reading before going to bed…

I stumbled on the following issue, marked as critical:

http://code.google.com/p/googleappengine/issues/detail?id=1970

What’s interesting about it is that this is also what is causing all the problems with running Grails development on Windows machines. If this is fixed, then building Grails for GAE on Windows machines might work without any workarounds. Lets hope the Critical focus remains on this issue, and that it is resolved.

I just set up a build server for my Grails project. I chose Hudson since it was the one mentioned to work well, and it did. I’m quite happy with Hudson thus far. It hasn’t crashed (actually it crashed my browser once, more on that later), and been very easy to work with.

First of I installed it on my Ubuntu machine in the closet. It was straight forward installed using apt-get as per instruction. This was a clean install and the first use of that Ubuntu installation for any kind of development work.

With Hudson installed I started going through its configuration. I knew there was a Grails plug-in so I figured I had to install a lot more software to get things going.

I found the plug-in page, marked the Grails plug-in and installed it.

I created a job, configured it to fetch the code from my subversion repository. Already I could run the job, and watch the output from my normal development machine.

I love the console out page, compared to watching the build-logs and build progress in MSBuild this page gives good feedback, doesn’t crash that much, and works as both progress and log all wrapped up in one.

With the job running I pinpointed the missing parts in the job. Running the grails targets. So it was time to configure the Grails Hudson plug-in. It required the Grails installation path, so time to set up the libraries. Download and install Grails, update the plug-in with the path.

When running a Grails job the first time you need to run the “grails compile –non-interactive” command. This will install all other plug-ins as it is the Grails way. This will make plug-in specific commands available.

I ran it first without the –non-interactive flag, which caused the app-engine plug-in installation to ask questions. Hudson didn’t mind, but my browser in which I was looking at the console out didn’t like it. It kept updating the page with “do you want to use JPA or JDO” questions, until it finally crashed.

In my case I wanted my build to upload the compiled application to Google AppEngine. I added the APPENGINE_HOME environment variable to Hudson’s main configuration. With the APPENGINE_HOME variable set to my newly downloaded SDK.

This is what my grails build target looks like:

To get the deploy command to work I needed to add the login details for app-engine. After some research I found that the details are configurable. All you have to do is add the following to you Config.groovy file in the grails project.

google.appengine.email = "email"
google.appengine.password ="password"

But with the correct email and password of course. In my case I used the user I’ve created for sending emails as the user that will deploy using the build.

I now have a working build that deploys to AppEngine. What I’m currently missing are running tests, as I’m still searching for a functional testing framework that will run as a Grails plugin with the AppEngine plugin. None of the two (easyb, functional-testing) Grails plugins I’ve tried so far have worked.

I just added a Cron job to “ping” my application every 1 minute. After a short discussion with Paul Cusch, whom commented on one of my previous posts regarding the load request issues Grails on AppEngine is seeing. I decided to give it a try. This solution is entirely based on Paul’s idea I’m just writing it up.

Grails AppEngine plugin is missing a feature to use cron out of the box. I hope to submit a patch for it. This is how it works. You need to have a cron.xml file next to your appengine-web.xml file. In the cron file you describe which url and how often a GET operation should be done on the specified url.

Details can be found here.

I simply added a request to a specified url, which I added to my URL mapping:

"/today"(controller: "user", action = "ping")

Then added a cron file to ping it every minute:

<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
    <cron>
        <url>/today</url>
        <description>Keep the app alive by ping every 1 minutes</description>
        <schedule>every 1 minutes</schedule>
    </cron>
</cronentries>

The cron.xml file I keep in my grails-app/conf directory, following the conventions used by the AppEngine plugin, for similar files.

The user controller unfortunately got the honor for hosting the ping request.

def ping = {
  render "ping ${new Date().toString()}"
}

And there it is. Next time that application is uploaded to the AppEngine, the ping method on the user controller will be executed every 1 minute. Helping to keep the application loaded.

This method comes with a few drawbacks however. The application will still hit loading requests every now and then, and those requests will take from your quota. This setup can easily eat up all your free quota.

Also, if everyone sets up this cron job. Then a lot of quota is wasted in general, which might affect the service for other applications. This workaround tries to fix a problem that should be fixed in the inner workings of AppEngine it self (not excluding changes to Grails initialization).

There is however a query that is ongoing. Google is considering the possibility to allow people to pay for keeping their application in memory. If you would consider such an option then star the this issue. I myself starred it, I think it’s quite acceptable.

I turned the cron loading task off in my application, instead seeing it as an experiment with Cron on Google AppEngine. The payoff for this solution wasn’t that great and for me it comes with the expense of some of my code quality.

While it is too early to say anything about actual performance gains. After upgrading to the 1.3.0 AppEngine SDK, I’m still suffering from loading requests timing out.

I did some experiments with lazy-initialization, causing my grails deployment to crash in many wonderful ways.

I’m going to let pingdom measure the application for a while to see if there are any CPU savings.

In the 1.2.8 version of the AppEngine SDK there is an option to precompile your solution as you deploy it to the AppEngine. Apparently precompilation can save up to 30% runtime on loading requests.

I added the option to turn on the precompilation-enabled setting in appengine-web.xml to the AppEngine plug-in this morning. Hopefully I can submit a patch with change, I’ll get back to that in a bit.

As I’m writing this I’m deploying to AppEngine for the first time. What happens is that during the deployment to Google AppEngine, the appcfg.cmd compiles the solution locally.

Showing the following lines in the input:

37% Cloned 600 files.
40% Uploading 4 files.
52% Uploaded 1 files.
61% Uploaded 2 files.
68% Uploaded 3 files.
73% Uploaded 4 files.
77% Precompiling.
80% Precompiling. 174 files left.
82% Precompiling. 94 files left.
84% Precompiling. 22 files left.
90% Deploying new version.

Now it’s deployed, I’m going to let it run for a while while I fix something else.

I have Pingdom set up for checking the AppEngine project every minute. That way I can see the changes in response time, and from the dashboard I can see if the CPU use goes down.

As far as I can tell, the precompilation doesn’t help with the timeout issues when starting a grails application hosted on AppEngine, it might help, but it doesn’t affect the loading time noticeably. However I for one am going to have my application precompiled.

If you use the app-engine plug-in deploy command using AppEngine SDK version 1.2.8 it will fail with the following error:

[java] Bad argument: email requires an argument, for example, "email=FOO"
[java] usage: AppCfg [options] <action> <app-dir> [<output-file>]
[java] Action must be one of:
[java]   help: Print help for a specific action.
[java]   request_logs: Write request logs in Apache common log format.
…
[java]                         request logs are returned.
[java]   -n NUM_RUNS, --num_runs=NUM_RUNS
[java]                         Number of scheduled execution times to compute
[java] Java Result: 1

Google changed so deployment is targeting a directory instead of a war file.

What you need to do to deploy is instead to use the AppEngine built in command to deploy

%APPENGINE_HOME%/bin/appcfg.cmd update c:\users\morten\.grails\1.2.0.RC1\projects\<projectname>\stage

After this all I had to do was to update my deployment scripts, and the stage is set for some serious Grailsing.

I should point out that all previous workarounds for working on Windows still need to be in place.

Google just announced their attack plan for handling the long startup cost for, among others, Grails applications.

Apparently what I can do as a developer is to enable pre-compilation of my application. I’ll experiment with this at the end of the week I hope.

Read the full post on the AppEngine blog

Regarding the logging issue with Grails and Google AppEngine, I’ve just installed the latest build of Grails1.2 and deployed to AppEngine, and it works.

No change to the loading time of grails however, I’m still getting timeouts on my initial requests.

The issue with the AppEngine enhancer still exists with Windows 7. Let’s hope for the next version of Windows then…