Posts Tagged Testing
Book Review: Growing Object-Oriented software guided by tests
Posted by Morten in Uncategorized on August 1, 2011
Finally I finished reading Growing Object-Oriented software guided by tests, by Steve Freeman and Nat Pryce. We ran it as a reading circle at RemoteX but I think I’m the only one that will finish it, and I did so last week. So the circle quickly lost it’s members.
I would like to mention that my expectations on this book was very high, which could make them hard to live up to.
The book starts by describing testing, and the takes us through a fictive scenario where we gradually build up software guided by tests all the way. Then at the end it covers advanced topics.
I had trouble reading the scenario part of the book. Part because I do not like the use of mock-frameworks. But also due to not really buying in to the scenario. Some paragraphs were written to add realism to the scenario, such as sudden changes of UI. But these paragraphs seems taken out of thin air, and adds no value to the chapters. This annoys me. I also disliked how the book kept defending the technology choices of the example, it was quite clear that this was an example after all.
Now the advanced chapters at the end did indeed pick up the book quite a bit, some aspects of it I already knew, while others provided fruitful suggestions on how I could improve certain aspects of one of RemoteX’s test-suites.
The beginning of the book structures out testing and the terms used quite well, and I did like the last parts of the books discussing advanced topics.
I’m not sure if I would recommend this book. The approach in the book isn’t bad. It’s intended for beginners, however I’ve seen what a junior team can do to a test-suite armed with a mocking framework. I would recommend that junior developer would start by using OO-design to do their testing simply because they will learn much more and their tests will be much clearer. An intermediate level of developer could extract great value from this book however.
The power of nested describes in Jasmine
Posted by Morten in Uncategorized on June 6, 2011
I’m experimenting with the Jasmine JavaScript testing framework to see if I can create a cucumber style testing framework using JavaScript. I want to go full out TDD on it so I started with a feature file, now I’m working on a spec to get that file running.
However as I work I get stuck on the following:
The description is too high level!
What happens when I load the feature file? Obviously a feature is loaded, but how? Something needs to happen between loading the feature and running the steps. My test needs to be more detailed.
Why not describe the load function then?
At once I realized that the load function needs to be asynchronous.
What if I worked with a story to being with?
Posted by Morten in Uncategorized on May 16, 2011
I’ve been working on a pet project, just to see if my idea was possible at all. Central to the idea was sharing of content between users, and I wanted to see if I could build a model to make it work on Google AppEngine.
Now I started out with some spec-style unit tests to make sure that it was feasible. Then when I started to see some progress started to bone out a walking skeleton for the application, a thin red line.
However somewhere along they way I drifted off testing the primary entity of the application and instead the focus was around one of its neighboring entities. The reason was that the other entity is central to controlling the sharing but its not the entity shared. Once my walking skeleton was complete I noticed that I had indeed created a model that solved the wrong problem.
Now all was not lost, I changed the tests that had drifted off in the wrong direction and a few hours of intense thought later I had conceived a model that solved to correct problem. I had my spec-style unit tests in python covering my refactoring and most of the skeleton still worked after I refactored.
To rethink my model I went back and wrote down what I wanted to achieve in a story format. The tests I rewrote became more like Given When Then than the spec style that was a result of the initial unit-testing.
So now I can’t help but wonder, what if I had had a cucumber acceptance test to begin with?
I’d be forced to consider what I wanted to solve on the correct level of abstraction, and I might not have drifted away from the original idea.
Running tests as files change
Posted by Morten in Uncategorized on May 9, 2011
I mentioned earlier a solution where Eclipse would run all my Python tests when I save a file. This is very convenient when using TDD in languages that can run tests fast. However that solution is tied to Eclipse and as I like Vim more and more I stand without an option to automatically run the tests as I code.
This feature is quite similar to AutoSpec in Ruby. So I figured there would be some solutions available for Python already. Looking around I found this to be the best possible solution right now. However that solution doesn’t handle nose tests, and the project I’m working on is using nose-gae to do the testing. So I needed a solution that worked on Windows and could run nose tests with nose-gae.
This started to annoy me, especially since it would be quite easy to implement it in .Net and have it work with the command-line to add support for any type of action that needs to be run on a file change.
So I wrote a little tool, which I call: OnChange
It’s a command-line application that watches the current directory and any directory under it for file changes. When a change occurs it will file a specified program, in my case, run my tests.
It also has support for parsing the output of the command to run, so with regex patterns you can add “reactions” to how the executed program is run. This allows me to signal Growl for Windows to bubble up “tests fail” or “tests pass”.
I also added a filter option so it will only react to changes in certain files.
This is how I set up my onchange program to run tests and signal pass or test automatically.
onchange -f *.py -r "^FAILED.*$:fail.bat" -r ^OK.*:pass.bat runtests.bat
This command line will cause OnChange to watch for changes to *.py files (-f flag). It has two reactions set up on that matches on FAILED test output and in that case runs the fail.bat command, and on that matches the OK test output and in that case runs the pass.bat command. The last argument to the call is my runtests.bat file. The runtests.bat file will execute the nose tests, and in an extension also execute my Jasmine tests in the same project.
The source for OnChange is available on GitHub. Feel free to help add to it if you’d like.
Working with Jasmine
Posted by Morten in Uncategorized on March 28, 2011
I’ve been working extensively with JavaScript these past weeks. As I mentioned earlier I wanted to try out Jasmine and I definitively got the chance. I’ve been using it for all my testing needs these last weeks, and I love it.
What do I like about Jasmine?
- The ability to nest describe statements
- the splitting of specifications into several files
- the output structure
What I do not like
- At times failing tests can give cryptic error messages in Chrome, since Jasmine will trap the errors inside it’s own execution. This can cause the stack trace and the error messages to mess up.
Let’s see some examples then:
describe("A view holder", function() { describe("when adding a view", function() { it("it should add the view's element to the holding div", function() { loadFixtures("/specifications/spec/holder.html"); var holderDiv = document.getElementById("holder"); var docHolder = new ViewHolder(holderDiv); var template = "<div>test</div>"; var doc = new View({template:template}); docHolder.showView(doc); expect(holderDiv.innerHTML).toContain(template); }); describe("when removing a view", function() { it("the previous view is displayed", function() { var docHolder = new ViewHolder(); loadFixtures("/specifications/spec/holder.html"); var holderDiv = document.getElementById("holder"); var docHolder = new ViewHolder(holderDiv); var template = "<div>test</div>"; var doc = new View({template:template}); var doc2 = new View({template:template}); docHolder.showView(doc2); docHolder.showView(doc); docHolder.popView(); expect(doc2.element.style.display).toNotEqual('none'); }); }); });
This is an example of a specification I wrote for a view system I’ve developed, the output looks like this:
I added the following snippets to SnipMate so now I have a really nice flow when writing my specifications:
# Describe snippet desc describe("${1:description}", function() { ${2} }); # it snippet it it("${1:does}", function() { ${2} });
Jasmine is now my testing framework of choice for JavaScript testing.
Unit-testing writing styles in QUnit (JavaScript) vs xUnit/MSTest
Posted by Morten in Uncategorized on March 14, 2011
Every now and then there are people doing experiments with different ways to name tests. The first time I was inspired to adopt a new way to name my tests was after Corey Haines performance kata (no I wasn’t there I saw the video, thanks Gregg). I really enjoyed the executable specification style.
Last year my naming conventions changed again after a few days with Andreas Brink, discussing BDD and the value of naming the tests as specifications, but according to the BDD format: Given, when, then.
Lately this also came up on Twitter for naming MSTest tests, also as a specification style.
A while back I did a little experimenting with Chrome Extensions. I created an Extension that interacts with uTorrent web-gui through a context menu item. While not perhaps the best example, I noticed something interesting as I was writing the tests for it. I use the QUnit framework since it’s the testing framework I first got testing to work for me in JavaScript.
It felt more natural to write Specification style tests in QUnit than the BDD style I’ve been using for a while. Which was quite a surprise.
To the right is an example of the output from the specification style QUnit tests for ChromeTorrent. You can check the code at Github.
Back to the tests. I try to keep my tests in a AAA style. However as I was writing the tests I noticed that QUnit doesn’t really lend it self well to this. Part of it I think is the way it fits to well with the specification style tests.
So let’s look at a QUnit test:
test("When a ChromeTorrent is created ", function() { $.ajax = function(settings) { settings.success("<html><div id='token' Style='display:none;'>fakeToken</div></html>", 200); }; var torrent = new ChromeTorrent("host", "user", "password"); ok(torrent != null, "it is initialized without error"); equals(torrent.getUser(), "user", "it has a user"); equals(torrent.getHost(), "host", "it has a host"); equals(torrent.getPassword(), "password", "it has a password"); equals(torrent.getToken(), "fakeToken", "it fetches a token from the host"); });
This test is a specification of what you can expect from a new ChromeTorrent object. The null check is a bit excessive but it’s my way to wind up my mind up for writing tests. However as you can see the way a QUnit test is structured, it lends it self quite well to the specification style of testing. This test fits well with AAA but lets look at the other test:
test("A ChromeTorrent can", function() { $.ajax = function(settings) { settings.success("<html><div id='token' style='display:none;'>fakeToken</div></html>", 200); }; var torrent = new ChromeTorrent("host", "user", "password"); equals(torrent.createDownloadUrlForTorrent("torrent"), "host?token=fakeToken&action=add-url&s=torrent", "construct the url for starting a download for a torrent" ); var getUrl = ""; var password = ""; var username = ""; $.ajax = function(settings) { getUrl = settings.url; password = settings.password; username = settings.username; }; torrent.addTorrent("myTorrent"); equals(getUrl, "host?token=fakeToken&action=add-url&s=myTorrent", "add a torrent to the downloads"); equals(password, "password", "on a password protected host"); equals(username, "user", "with the given user"); });
We can see here that AAA is broken. To be honest I think of this more as a AAA cycle where I arrange, act and assert in cycles. In essence creating a “test” between each assert.
So why am I breaking AAA here?
Well its because of the output of QUnit, so each test isn’t isolated if you want to have the pretty output when using QUnit. I could improve the situation somewhat by wrapping ajax calls through a client class that I pass to ChromeTorrent, but the broken cycle will still remain.
I could use Modules to make it a little bit better but I am testing the module ChromeTorrent so it wont fit perfectly either.
There is hope! Recently I stumbled upon Jasmine, a JavaScript testing framework that has a similar output. It has better support for specification style testing, and it can be run using JSTestDriver, that was recently recommended to me. I hope to dig more into it as soon as I find some time to exercise my JavaScript Testing.
Talk on Python Testing on Google AppEngine
Posted by Morten in Uncategorized on December 10, 2010
Last night I held a talk at GTUG Stockholm about testing on Google AppEngine. Here are the slides that I used during the presentation.
Bananas!
Posted by Morten in Uncategorized on November 15, 2010
During Android Only Erik Hellman from Sony Ericsson explained how they create their Android Phones.
During his talk he mentioned that they put all phones through a monkey test. Where a phone has to sit for days with random input on the phone, without crashing.
Naturally the thought is, I wondered how the applications I’m working on would stand against such a test. A monkey test should find all places where the input validation fails, and thus crash.
So I started searching around for tools that did this for Windows Applications. However All I could find was the Hopper Windows Mobile stress test tool, and nothing for desktop Windows Applications.
The thought is if I should write my own, focusing on .Net desktop applications. A tool that sends keystroke and mouse move messages to an application generating random user input. The tool could constrain only to use certain keys or prevent certain patterns from occurring. It could restart the application when it crashes, and log all crashes to a log-file.
It could be quite easy to create a command-line interface for it and execute it in a build system. Naturally a good name for a monkey testing tool would be “Bananas!”
So far it’s just an idea.
Stress and TDD
Posted by Morten in Uncategorized on September 20, 2010
In Blink the author describes that there is an optimum peak of performance in regards to stress. What happens after that peak is reduced performance, inability to act on details and tunnel-vision so on.
In programming this means we will see the end goal, often the finished functionality according to a specific happy path scenario (that is developer specific). This means we will skip extra checking, and testing. We are not considering the alternative flows of the application. The end result when too much pressure is applied to software development is that things loose quality as the developers sight narrows.
TDD from a process perspective is designed to help developers maintain quality as the temperature rises. If the developer safely can write test for what he is doing, then there is an extra automatic quality gate for the production code to pass. Each test is a statement of the assumptions made by the developer as he or she writes the production code.
As a programmer I make decisions and assumptions all the time. I can see how stress cause a lot of the bugs that I’ve seen. In essence I think that this is what TDD is meant to resolve, to give you a framework that even though your stressed beyond cognitive functioning your decisions are recorded, and if you do something that breaks a previous decision, especially during a stressful period, you will be informed accordingly and forced to think.
However if you push hard enough something will break. What can break down instead? The obvious answer is Discipline.
Removal of tests, badly written tests, simply not writing tests etc. All you do is push the problem into the test suite. With a badly written test suite your production code will suffer.
So the question is: Can you correct an organizational problem with a methodology?
Specifying tests
Posted by Morten in Uncategorized on August 6, 2010
The latest post on the Google Testing blog is quite interesting. Philip Zembrod discusses the readability of code developed by TDD and found that sometimes reading the test first didn’t always help to understand the written code.
He makes a valid point. I started objecting to using mocks when it was pointed out to me that most of the tests produced with mocks weren’t readable. They typically don’t follow AAA (Arrange Act Assert), and thus work poorly as specifications. But it’s not the mocks, it’s how understandable the test is as a specification. Is it easy to understand what the test does?
If I look at a test 5 months from now, and it is unclear what its testing and how its testing it. Then the ROI of developing that test is lower than it could be.