Posts Tagged Design
Making it easy to work with performance
Posted by Morten in Uncategorized on June 13, 2011
We’ve been working with performance in our REST API a while back. We’ve been considering different options for measuring our API in production, but we had an idea to use the design of the API itself to handle the measuring.
Part of the inspiration came from the Changlelog’s coverage of CodeConf and Coda Hale’s Metrics talk. (Slides).
Using our pipeline approach to constructing a REST-API with conventions, we added a pipe that starts a stopwatch and stops it when the request is finished. Then we can log statistics for each individual part of the API, closely coupled with the code that is executed for that particular request. I pitched the idea to my good colleague Johan Andersson, and he did a fantastic job at implementing the solution!
The output looks something like this:
This allows us to see how our API is performing in production, and where we need to focus our optimization efforts to make the API go faster. We can even add some flags to the request to get more detailed output.
Bananas!
Posted by Morten in Uncategorized on November 15, 2010
During Android Only Erik Hellman from Sony Ericsson explained how they create their Android Phones.
During his talk he mentioned that they put all phones through a monkey test. Where a phone has to sit for days with random input on the phone, without crashing.
Naturally the thought is, I wondered how the applications I’m working on would stand against such a test. A monkey test should find all places where the input validation fails, and thus crash.
So I started searching around for tools that did this for Windows Applications. However All I could find was the Hopper Windows Mobile stress test tool, and nothing for desktop Windows Applications.
The thought is if I should write my own, focusing on .Net desktop applications. A tool that sends keystroke and mouse move messages to an application generating random user input. The tool could constrain only to use certain keys or prevent certain patterns from occurring. It could restart the application when it crashes, and log all crashes to a log-file.
It could be quite easy to create a command-line interface for it and execute it in a build system. Naturally a good name for a monkey testing tool would be “Bananas!”
So far it’s just an idea.
Refactoring of logging rules
Posted by Morten in Uncategorized on November 8, 2010
I’ve been working on a logging system for RemoteX Applications, the system is going Live any day now. (In fact since this post will be scheduled it might already have).
So I started out doing the code as straight forward as possible, but it quickly started to nest if-statements as the different logging conditions grew. I got something similar to this:
public void HandleNotification(NotificationEventArgs notificationEventArgs) { if(notificationEventArgs.Type == typeof(Case) ) { var c = getCase(notificationEventArgs.Reference) AddForLogging("Case status changed to "+c.Status); } if(notificationEventArgs.Type == typeof(Resource)) { AddForLogging("Resource changed"); } if(notificationEventArgs.Type == typeof(Cost)) { var p = getProduct(notificationEventArgs.Reference); AddForLogging("Cost updated for product" + p ); } }
Now I wanted the if statements to execute after one anther since they would eventually build up to a single entry in the Log. However as the information that should be logged increased. This method became increasingly hard to maintain.
Also this point is a natural extension point for adding new rules for what should be logged and when. Thus I wanted it to be easy and simple to add things for logging.
By extracting an interface using a variation of the Specification pattern, I get the ILoggingRule.
Each if-statements body is moved to and implementation of the Execute method, and the Boolean expression is expressed in the AppliesFor methods execution. With this and a simple foreach-loop. The above code becomes the following:
public void HandleNotification(NotificationEventArgs notificationEventArgs) { foreach( var loggingRule in _rules.Where( loggingRule => loggingRule.AppliesTo( notificationEventArgs ) ) ) { loggingRule.Execute( notificationEventArgs, AddForLogging); } }
Now this is simple enough to extend using Castle, and it moves the complexity away from the generic log handling. Thus it allows for more complex logging-rules as well.
Here is an example of a logging rule:
public class CaseStatusLoggingRule : ILoggingRule { const string CaseStatusChangeTemplate = "$L{{Log.Case}} $L{{Log.StatusChange}} $L{{Log.Case.Status.{0}}} $L{{Log.To}} $L{{Log.Case.Status.{1}}}."; /// <summary> /// returns true if this rule applies to the notification /// </summary><param name="notificationEventArgs"></param><returns></returns> public virtual bool AppliesTo( NotificationEventArgs notificationEventArgs ) { return notificationEventArgs.EntityType == typeof( Case ) && notificationEventArgs.NotificationType == NotificationType.PropertyChanged; } /// <summary> /// Executes the logging rule /// </summary><param name="notificationEventArgs"></param><param name="logCallback">A callback that will execute the actual logging, first parameter is href, second is message</param> public virtual void Execute( NotificationEventArgs notificationEventArgs, Action<string, string> logCallback ) { var caseHref = "cases/" + notificationEventArgs.EntityRestId; string message = String.Format( CultureInfo.InvariantCulture, CaseStatusChangeTemplate, notificationEventArgs.Changes[0].OldValue, notificationEventArgs.Changes[0].NewValue); logCallback(caseHref, message); } }
As you can see the if this would have been placed in the first code the if-statements and their bodies would wreak havoc on the readability of the code. Now instead its moved to it’s own class, and the logging can naturally be extended without adding complexity other than what’s required for the specific logging criteria.
What’s more funny is that about a month ago I did something similar to how Shuffle handles its serialization of Track entities. It uses the XmlPullParser since I’ve read that it’s the highest performing parser on the Android Platform (please let me know if I’m wrong).
Now parsing a single entity from the Tracks-service follows this procedure:
public ParseResult<E> parseSingle(XmlPullParser parser) { EntityBuilder<E> builder = createBuilder(); E entity = null; boolean success = true; try { int eventType = parser.getEventType(); while (eventType != XmlPullParser.END_DOCUMENT && entity == null) { String name = parser.getName(); switch (eventType) { case XmlPullParser.START_DOCUMENT: break; case XmlPullParser.START_TAG: Applier applier = appliers.get(name); if(applier != null) { success &= applier.apply(parser.nextText()); } break; case XmlPullParser.END_TAG: if (name.equalsIgnoreCase(mEntityName)) { entity = builder.build(); } break; } eventType = parser.next(); } } catch (IOException e) { Log.d("Exception", "IO EXception", e); return new ParseResult<E>(); } catch (XmlPullParserException e) { Log.d("Exception", "pullparser exception", e); return new ParseResult<E>(); } return new ParseResult<E>(entity, success); }
As you can see there is nothing entity specific about this code, it’s a template method. The parsing is instead controlled through the dictionary containing something I badly called Appliers. An Applier takes a xml tag and applies it to the Shuffle entity builder.
All parsing is done using this method and the implementing template method classes instead create a list of appliers that can react to an expected xml tag.
An applier looks something like this:
appliers.put("name", new Applier(){ @Override public boolean apply(String value) { specificBuilder.setName(value); return true; } });
I’m using Java’s direct implementation of an interface to allow create the different appliers. Thus the parser just specifies which tag-names it can handle and how to handle them. The parsing is naturally extensible and quite easy to test as well.
I’d appreciate feedback, I’m quite certain there’s a pattern lurking here and I don’t know if the Specification pattern is the correct one.
Working at the right level of abstraction
Posted by Morten in Uncategorized on May 25, 2010
At RemoteX we have been a bit unhappy with the code produced to create our REST-service using WCF.
The problem is that you have these contracts that you specify. These contracts then build up a web-service end-point using the WebGet and similar attributes, to create a REST-style endpoint. The contracts however often look like a SOAP contract, or an interface wrapping behavior. A typical contract often includes the following operations for a specific type of document:
- Get Single (web get on href entity-type/id)
- Get List (web get on href: entity-type)
- Get List with modified since (entity-type with query-string)
- …
- and so on
The trouble is the URI’s. A single contract often specifies several endpoint’s and several behaviors. There is no Single-responsibility principal at work, or rather the single responsibility might be that a contract handles a specific entity. End result, a lot of code for simple things and a REST-service that is unnecessary hard to extend.
This setup also makes it tricky to see the structure of the REST-service, the URI’s are spread out in the code contracts, forcing you to look at each method to figure out the different URIs.
After working with Grails and Django, and finally when I read the documentation for the Kayak web server. I realized what we were doing wrong. A reference in the Kayak documentation was the wakeup call:
The server component is an intuitive, unobtrusive, no-gimmicks implementation of HTTP, and the framework simply maps HTTP transactions to C# method invocations with minimal syntax and configuration.
We’re not working on entities as an abstraction, we’re building a REST-based service. Our level of abstraction needs to be HTTP.
So I discussed this with a colleague Johan, the day after. After a while Johan set to work on a piece of the REST-server that needed rewriting and after a week of TDDing. He came back with this wonderful piece of infrastructure, that worked along side the “old style” infrastructure.
The idea is that we have one contract. This contract has 4 methods:
- Get
- Put
- Post
- Delete
Essentially all the methods we want to implement. We then have a route configuration, similar to that of Grails or Django. That routes requests to Controllers that will take care of them.
Johan set things up so there is a single server, that looks at the route-configuration when a request arrives and executes a command on the controller object. The controller object doesn’t have to add methods for the operations doesn’t support. In fact, the controller object is a plain POCO with no inheritance or interface on it. As long as it has a method that matches the operation it will be called.
Serialization is handled in the framework and by following conventions when writing the controller you can take different input automatically processed by the infrastructure.
Lets see an example:
public class LogController
{
readonly DataContextFactory<RMXAppsDataContext> _contextFactory;
public LogController(DataContextFactory<RMXAppsDataContext> contextFactory) {
_contextFactory = contextFactory;
}
public void Post(LogEntry entry, IIdentity user)
{
PostMessage( user, entry.Message );
}
public void PostMessage( IIdentity user, string href, string entry )
{ using (var context = _contextFactory.CreateContext())
{ var dbUser = context.GetTable<User>().Where(u => u.UserName == user.Name).FirstOrDefault();
var dbEntry = new Applications.Data.LogEntry {
Message = entry, Date = DateTime.UtcNow,
User1 = dbUser };
context.GetTable<Applications.Data.LogEntry>().InsertOnSubmit(dbEntry); context.SubmitChanges();
} } [return: Xml]
public Log Get() {
Collection<LogEntry> entries;
using (var context = _contextFactory.CreateContext())
{ var e = (from l in context.GetTable<Applications.Data.LogEntry>()
orderby l.Date descending
select Mapper.Map<Applications.Data.LogEntry, LogEntry>( l )).Take( 50 );
entries = new Collection<LogEntry>(new List<LogEntry>(e));
} return new Log { Href = "log/", LogEntries = entries };
}}
It’s not the prettiest of examples (and I apologize if the code doesn’t look that great).
What your looking at is a simple log controller. It handles the log/ endpoint. The endpoint has a Get and a Post operation. Post is used to add LogEntries, and Get gives the latest 50 entries.
You will notice that the Get operation is marked with [return: Xml] this means that the operation will serialize the return value using XML.
The Post operation takes two inputs. LogEntry and a IIdentity. Both these parameters are set by conventions.
IIdentity is the user, as authenticated by the login-module, the infrastructue simply gives the user to any IIdentity parameter in a web-method.
The LogEntry is the deserialized body of the request. The infrastructure looks at any parameter that doesn’t fit a convention, checks if its serializeable and if the body of the request can be deserialized into such an object.
Now a route configuration is another object, that specifies which urls which controllers are registered too. These structures are set up using Castle, and loaded from any assembly in the web directory. Allowing us to extend the REST-service by dropping in assemblies that implement our structure.
Since this was added the development of the REST-service has really taken off. We add roughly one-two endpoints per week, designed to solve specific pains and give views of the domain specific to the task at hand.
By working at the HTTP level of abstraction instead of the contract abstraction, we have reduced the cost for extending the rest-service tremendously. Even better is that the controllers are easy to test, the code displays its intent clearly, and you can see what each specific Uri in the REST-service does.
Special thanks to Johan, great work!
Inheritance of features
Posted by Morten in Uncategorized on January 14, 2010
Given my recent combining of Tracks and Shuffle. An interesting thing happened. Something that happens when you combine two products so that they inherit features from each other.
On Shuffle’s issue list there are a few items currently regarding repeating tasks, and the ability to hide a task for a period of time. Tracks has these features, since Shuffle can synchronize with Tracks, Shuffle now also have these features. That is if you choose to synchronize with Tracks.
I’ve done a few integration projects thus far. And the ones that are truly successful are when they take the aspects and features of both applications into consideration, to create a win-win scenario between the two applications.
Here is another recent example. At RemoteX, we recently integrated a system that handles the ordering of materials, goods, equipment etc. In essence this integration feature means that you can go to a retailer, order goods, specify the order they should tag the invoice with. End result, RemoteX automatically adds the goods to the application and if possible attaches a copy of the invoice to the case. Ready to be billed to the customer. Since the other system has integrated with almost all resellers of service equipment in Sweden, the daring customers running this set-up can, at least in theory, eliminates the need for keeping stocks of equipment or goods in house.
If you understand where these synergy effects can show up you can also align the integration so that both applications maximizes the gain of integrating with the other system. This is important to think of when designing an integration, how do we achieve the maximum win-win?
The guessing user interface
Posted by Morten in Uncategorized on January 13, 2010
My girlfriend has downloaded a solitaire application to her HTC Tattoo. While solitaire isn’t anything revolutionary it has an interface feature which got me thinking.
Se what the interface does is move those cards that it is sure that the next step is to move. So if I for example get an Ace, it moves it to it’s correct place at the top four card stacks.
This feature got me thinking about mobile phone interfaces. In applications in general, but even more so in the mobile context, you want to reduce the amount of interaction done with the user interface.
What if we can “guess” the input of the users input in the interface even in business applications?
We have hints in the domain, such as if you select this type of project you cannot register this type of time on it. You don’t register vacation time on a customer’s bill if your a consultant. Well, at least not that often.
These hints are common constraints, you might not want to enforce them by disabling the alternatives available. But you can check them and see if for a given input if there are any “assumptions” you can make about the next input to be made.
To take this one level further. We have the application data. We can order things in the UI according to a statistical analysis. Given this value in a combo box, this other value has more than 80% chance to be set in the other combo box. In such a case we can “guess” that this is the input the user would want to enter, and worst case is that he has to change it.
The cost of changing a value in a combo box is equal to setting one in the first place.
Now statistical analysis of data can often be costly, and typically not something you would want to do in a UI scenario. But what you can do instead is to calculate the statistical values into a decision graph (I think of it as similar to a Markov process for those whom have studied them). We can calculate this graph every evening and have a graph available for all input for the next day. Traversing the graph is now a simple algorithmic exercise.
Rethinking design desicions
Posted by Morten in Uncategorized on November 21, 2009
I read a blog post at Three Rivers Institute that lead me to the following presentation.
The presentation is about startups and search engine optimization. How to drive visitors to your website.
Dharmesh discusses how to adapt your webpage to generate visitors, also promoting his hubspot seo grader.
Now this got me thinking of a project I’ve been working on. It’s a highly dynamic JavaScript based page, hosted on Google AppEngine. (currently not in a state I’d like to show it off since its still lacking the basic features).
The page is being designed with a few ideas in mind:
- All dynamic data i served to the page through REST
- All static pages are served with long expiration, most content is static
- The main page never reloads, instead JavaScript does AJAX to update the content
- The JavaScript is designed using SOLID design principles
- The JavaScript is tested
- Back and forwards buttons still work
- Marketing Content is served using Google Sites, to reduce bandwidth but especially to shorten roundtrips when working with Marketing material
Now the idea here is to give a similar feeling as to what GMail feels like. A common trend, in web application development I think. However, after watching the presentation the following thought occurred to me. The GMail application needs to be found by search-engines.
Google has a signup page with content that is searchable, but the main JavaScript client is not designed to be found by Search Engines.
I want my page to be found by search engines, and this means that there are a few of the basic Ideas behind the page that I need to reconsider. Especially the following:
- The main page never reloads, instead JavaScript does AJAX to update the content
- Meaning that the page looks “empty” to search engine crawlers since all content is loaded using JavaScript
- Marketing Content is served using Google Sites, to reduce bandwidth but especially to shorten roundtrips when working with Marketing material
- Right now this is implemented using IFrames, which probably cause trouble for search-engines (yeah yeah, I know frames suck especially IFrames… but there were some cross site issues I didn’t want to fix right now),
I need some of my content pages to be searchable, but not the AJAX-driven application in it self. So now I have to rethink some of my decisions which were made from a pure coding view-point, to adapt to marketing viewpoints (and search engine algorithms so its not all bad).
The importance of teaching
Posted by Morten in Uncategorized on July 1, 2009
I have a good friend whom after university focused on first communication and then management. But now he has turned to development. We’ve been having a few discussion over the past few days about SOLID and REST and Grails. IoC containers and what not.
He hasn’t seen these kinds of concepts in his development so far. But he finds the concepts interesting and he is eager to learn about it. Which has given me a unique opportunity to learn how to communicate these concepts.
Now I’ve been an assistant teacher at the university. But I found that some of the benefits of these concepts are hard to explain, simply because to me their very abstract and I usually don’t discuss them at this level. I’d also like to mention that my friend is great at asking the right questions.
So this is a shout out to Joel. Thanks for teaching me how to communicate these concepts and their benefits.
Refactoring how to open views
Posted by Morten in Uncategorized on April 18, 2009
We had a problem in RemoteX applications, not a big one, but an old structure that had to be refactored. Our open method only accepted URIs as arguments. This forced most of our commands that create new entities, to first save an empty new version of the entity to make it accessible by using a URI for it.
This in turn caused unnecessary round trips to the data access layer, but also caused problems with potentially empty data.
So this Wednesday this code died. In the old implementation; the general OpenView command accepted a URI. Now it can both take a URI but also an actual object.
To allow this I had to change the targeted end-point urls to activate a class instead of a delegate, as it was before. This has two great benefits.
- By using classes I open up for taking advantage of Castle for setting up the different code for opening. Previously all the setup was made in one big wiring class, resulting in excessive class coupling CA warnings on that class.
- Each class that opens an entity view conforms to SRP. It has one purpose, to open a view for a specific entity.
As I implemented the ViewOpener I noticed that all the delegates for opening entities had one thing in common. The first thing they did was to open the entity from the data storage.
By breaking out this behaviour into an abstract base class I could reduce the amount of code, simply by having the base class fetch the entity and then let the more specific view opener setup the UI.
This resulted in the structure illustrated below.

ViewOpener structure
So what are the insights of this refactoring?
Having delegates for a specified behaviour is a code smell. By having simple short classes with a single behaviour instead, we make code easier to refactor. Its easier to simply add another simple class to the Castle wire up, than it is to implement the same behaviour in a large complex class that handles all UI setup.
Performance monitoring of a deployed application
Posted by Morten in Uncategorized on April 14, 2009
I had an idea today while working with a performance issue on the REST-service of RemoteX Applications.
In theory having a performance counter with the average execution time of SQL-queries over time. Will allow monitoring of system wide average cost for queries in the system.
That means that overtime its possible to track the changes of the execution times as they grow or shrink during normal production environment conditions. The cost for calculating the average could be relatively cheap to calculate compared to the round trip to the servers, and also the calculation could be threaded of to a different thread, so as not to disturb the worker thread that handles the request to the web server to begin with.
The idea is to examine the changes of the average query time over time, to see if performance of data access is changed between version upgrades, or hardware upgrades for that matter. There are some performance counters built into MSSql that can give data that is somewhat like this. However most of the require heavy load to see any performance upgrade or downgrade. This instead logs an average of the actual execution time for the application.
A benefit with this is that the performance is individual per application, but the actual code or counter could be reused between different applications.