I've just come back from
http://www.devoxx.com/display/DV09/Home" target="_new">Devoxx 09 in Antwerp, Belgium, a conference that will probably become known as the one in which closures sneaked back in to Java 7.
Whilst I didn't feel any particular urge to return for the day 3, there was more than enough interesting material, both technical and abstract, in the first two days for me to chew on. Here some thoughts and comments on my personal "slice" of Devoxx.

Andrew's agenda

What I saw (and, for the curious, what I missed)¹:

Wed

• Keynote: Java, the Platform for the Future
• Keynote: Java EE 6 and GlassFish V3: Evolution of a Platform
• Agile Mythbusters
• The Concurrency Revolution
• Traditional Programming Models: Stone Knives and Bearskins in the Google Age
• Keeping Your Options Open, Even if the Cloud is Not
• Distributed Programming the Google Way

Thu

• Keynote: Do we really know how to develop software?
• Keynote: Filling The Professionalism Gap
• The not so dark art of Performance Tuning
• Google Appengine Java - Groovy baby
• Traditional Programming Models: Stone Knives and Bearskins in the Google Age
• Using BTrace and DTrace to Instrument and Analyse Java Applications
• Performance for the performance-shy

That announcement

Yes, it's somewhat ironic that I missed the big announcement of the conference: that closures may sneak in to Java 7 after all. In fact, I had decided to forego Mark's talk in the moderately firm belief that there wasn't going to be anything in it he hadn't presented elsewhere or blogged about before. So much for instinct.
Needless to say, news filtered through pretty soon, though, and of course almost everybody has an opinion about this unexpected (to say the least) development.

Some samples² of what they might look like

Collections.sort(strings, #(String a, String b) a.length() - b.length());
Collections.sort(strings, #(String a, String b) { return a.length() - b.length(); });

@Shared int clickCount = 0;
button.addActionListener(#(ActionEvent e) { clickCount++; });

What's buzzing

With Sun and Adobe "tier 1" sponsors of the event, it was hardly surprising to come across quite a number of JavaFX and Flex talks. Oracle's entry into the fray was slightly more intriguing, focussing as they did on new features in the upcoming WebLogic releases.

Needless to say, that was slightly more enterprisey than one is used at developer conferences; I personally was intrigued to see them starting to get interested in the deployment space with an initial offering that makes it possible to create virtual machine images from configured systems.
From what I saw, there are still some important concepts missing there (which, naturally, we at XebiaLabs are trying to address with Deployit), but it's interesting to see the middleware vendors beginning to pay attention to this.

Sponsors aside, the conference's main threads, as far as I could see, were the "usual suspects" of concurrency, OSGi, Cloud and Scala. Of these, concurrency/distributed programming and OSGi seem to be gaining a little more enterprise traction (in terms of the number of people who actually appear to be using them for real projects); Scala and Cloud remain things to be "interested in" and "play with", but still retain a slightly experimental flavour.

I can't say, though, that I came across anything that hasn't been around the conference block for at least a year. No real surprises, in other words.

Highlights

Most Professionally Thought-Provoking

Joint award here to Ivar Jacobson and Uncle Bob, both of whom presented a rather critical view of the current state of the software development industry and made some thought-provoking remarks.

The main gist of Ivar's talk was that software engineering as a discipline lacks any kind of solid theoretical foundation, lurching from (methodological) fad to fad rather than developing a cumulative body of best practice knowledge.

His approach to avoiding what he considers as a recurring tendency to throw out the baby out with the bathwater is to break methodologies such as Agile down into sets of practices. In this framework, these constitute the "atomic" entities, and supposedly make it easier to identify common features of different methodologies and incrementally improve them.

Whether the Iterative practice in e.g. Scrum is really the same as Iterative in a different methodology, or whether processes are really just "bags" of practices to which you can add others without disturbing some inherent balance, are questions left unanswered.

In any case, Ivar has drummed up a bunch of Thought Leaders and is issuing the following call to action, to which all developers are apparently invited to contribute.

By contrast, Uncle Bob's talk focussed less on the process and more on the technical side of software engineering. His central tenet: developers need to learn to behave more like skilled professionals. Which to him means, amongst other things:

• Learning to say no (how often have you heard a doctor say: "Fff, it'll be tough, but we'll try and get your liver done by tomorrow?")
• Demanding a right to a technical career path that suitably recognizes
  accumulated skill and experience (rather than forcing developers to become managers to progress)
• Always aiming for a "professionally ethical" level of code quality (independent of business value)

It intrigued me to note that the attitudes that Bob was referring to are most commonly found in guild-like professions (such as law or medicine), whose practices are usually based more on experience and tradition than hard science. But let's keep the "software development: craft or science?" debate for some convivial pub session, shall we?

Most Technically Thought-Provoking

Looking back on it as a whole, Cameron Purdy's talk was no more or less than a well-delivered, knowledgable presentation on the benefits of event-driven architectures. The observation he used as an introduction, however, I found fascinating: simply learning to program is bad for scalability.

The argument goes a little bit like this: in almost all the languages you might learn as a beginning programmer, you can write something like:

  int total = 0;
  int i = 10;
  addToTotal(i);
  // now total = 10
  i = 20;
  addToTotal(i);
  // now total = 30

Trival. But hidden amongst our "natural" interpretation of what code will do are the following expectations:

• the statements will be executed
• the statements will be executed in the order written
• the statements will be executed exactly once

Cameron's point is that, if you consider distributed systems, these requirements - guaranteed execution, global ordering and exactly once processing - are just about the most stringent, and hence costly, expectations one can have.

Systems which are fault-tolerant, free of race conditions and idempotent have a much better chance of scaling and distributing well, and one of these reasons this is hard is because it goes so deeply against the grain of the intuitive picture of how programs run that are ingrained right from the first line of code we write.

Best Technical Content

Trust Google to put together the talk with the most varied range of "cool stuff". Gregor Hohpe gave a very engaging presentation on MapReduce, Bigtable, Sawzall and others, interspersed with mentions of things like Bloom filters and other interesting theoretical concepts.

The common theme here was definitely trading off functionality for scalability, with the conclusion that we can often get away with much less functionality than one might expect. Sawzall, for instance, does lightning-fast distributed aggregations, but, being history-less (you only see the current row), can't even do duplicate checks.

So what do you do if you discover that the engineers over at the Mountain View chocolate factory have sacrificed even the tiny little bit of functionality you need? Can you hand back a bit of that Bigtable scalability for multi-row locks, or are you straight back to Oracle 11g?

Well, it appears that Google does have a transactional engine on top of Bigtable, after all. But that's certainly not open source!

Best Presentation

The prize for sheer viewing pleasure goes to Doug Tidwell's talk on SimpleCloud. Humourous, slickly presented, with a good mix of examples and concepts served in manageable chunks, I certainly enjoyed this one. Including a recurring joke (in this case, Adam Koford's set of HTTP response code illustrations isn't a new idea, of course, but this one was actually topical and funny.

In terms of the content, SimpleCloud looks like a pretty decent cloud API. Quite how useful a PHP library can be for Java developers remains to be seen; if you're looking for a powerful, concurrent Java cloud library with support for lots of cloud providers try jclouds³.

Lowlights

Most Annoying Trait

To presenters from North America, a polite request: if at all possible, please remove jokes or other references to intrinsically North American popular culture.
Having to sit through a five-minute explanation of who a particular character in an American sitcom in the 70s was, or what the point of a certain gameshow was, isn't a great start for a presentation that was supposed to begin with a quick joke and then get on to the topic the audience is actually interested in.
Thank you.

Biggest Disappointment

Working for a company that is widely known for its Agile expertise, I was looking forward to Scott Ambler's promise of investigating some of the myths and rumours surrounding Agile to the scrutiny of cold, hard statistics.
What we got, however, was a lightning dash through the self-reported results of a number of surveys, without so much as a hint of analysis or further explanation.

Whilst it's certainly not uninteresting to learn that more than 50% of Agile teams do in fact do some up-front design, what I would very much like to know is whether up-front design actually leads to more successful projects! Suffice it to say that my colleague and I left the hall with many questions and few, if any, answers, and that's not taking into account the confusing presentation of many of the numbers.

A note on Performance Talks

As you might have guessed from my conference agenda, I'm quite interested in all this concurrency and performance stuff (it probably wouldn't surprise you either to learn that I'm also trying to help with Multiverse, a Java STM implementation). But after this round I think I'll be skipping performance tuning presentations for a while.

Not that Kirk's or Holly's presentations were bad or even boring, they were simply going over the same basic "performance tuning 101" stuff that is talked about every year, at every conference.

Now, these sessions are usually packed, so there certainly still seems to be plenty of demand for learning how to get a thread or heap dump or how to write a JMeter test plan.

Naturally, I understand that, because performance tuning is so specific to the application at hand, it's difficult to give more than general information and guidance. But at the end of one of these sessions I always feel I've hardly scratched the surface of these experts' knowledge of application performance.

What I would love to attend, for instance, is session during which I could simply watch these performance experts at work, attacking the problem of tuning a real application "from scratch".
The point would not be to see to what extent the application could be speeded up - in fact, that would be completely irrelevant. But I think there would be a lot to learn from seeing how a performance tuning professional goes about such a task in practice: where to start measuring, what tools to use, how initial hypotheses are formed etc.

But of course it's also not realistic to expect people to give away the "trade secrets" that allow them to earn a living just like that.

Oh, and to all the JVM vendors out there: please consider coming up with some standardized formats for profiling information! It's such a pity to watch demonstrationns of really useful-looking tools (such as IBM's Health Center) only to discover they can't work with the data produced by the JVM your client is using.

Quote of the conference

The rumour that Mark had come up with the new closure proposal syntax on the plane over to Belgium certainly caused a few chuckles. My favourite, though, was a remark attributed to Larry Ellison, who in response to possible demands from the EU to sell off mySQL is supposed to have said: "The 'my' in 'mySQL' means Larry's SQL.".

A final note

Hats off again to the Devoxx team. Content seems to be more or less on a par with JavaOne, and yet the attendance fee is still roughly €1000 less than almost all other conferences.

It's not perfect, of course: the wireless connection is just as continuously unavailable as elsewhere, and the food-loving Belgians are unlikely to consider the catering the pinnacle of Belgian culinary tradition. But after all, for €1000 you could eat and drink yourself pretty silly in Antwerp's many good restaurants and bars, and still have enough left over for some lightning-fast internet.

For me, still the European conference to beat.

Last time I blogged about performance tuning based on evidence, the tuning cycle and some best practices. This time I'll blog about the last step of the seven steps: sharing the responsibility for the whole system chain.

When an incident happens in production, this usually means stress. A performance problem in production often leads to finger pointing. The DBA says that he has looked and nothing is wrong with his database. The network operator concludes the same thing about his network. The app server operator about his app server, the software developer about his source code and the back end operator about his back end. It is never them, it is the other.

Figure 1. Finger pointing does not help to solve the problem.

The application often gets thrown over the wall to the operation department. Responsibilities then hold only at one side of that wall. If software development, maintenance, testing and/or operation is outsourced to external parties this can lead to tricky situations. Before you know it, contracts and legal procedures are at play and cooperation is far away. Both parties stick to their position, costs will raise and precious time gets lost.

Finding out which part of the chain is responsible for the slowness can partly be solved with proper tools that monitor the whole chain and tools which are used from early on in the development process. But there is more to it than just tooling. Experience with and knowledge of tooling and technology is inevitable just as priority for the proper utilization of the tools. It is most important to prevent formation of separate kingdoms and finger pointing between them; and rather to operate together as a multi-disciplined performance team and share the responsibility for the whole chain.

We are almost there with this series Next time I'll draw the final conclusions of these seven steps.

Besides organizing a Scala workshop at the J-Fall meeting we also presented five technical posters to serve as discussion points for anyone interested (or just walking by). Unlike traditional meeting sessions we could interact directly, somewhat similar to open space sessions.

The five posters we presented were:

• Scala by Arjan Blokzijl - a quick overview of the main features and characteristics of the Scala programming language
• @Composite by Andrew Phillips - making Java annotations more reusable
• FitNesse and VMWare by Robert van Loghem - combining FitNesse and VMware to decrease installation and configuration of the system under test
• Domain-Driven Design and Command-Query Separation by Erik Rozendaal - separating the domain model from query and reporting concerns
• GIT by Arjan Molenaar - an overview of git - a fast version control system created by Linus Torvalds

All-in-all these poster presentations were successful and a lot of fun to do. The discussions were much more interactive than regular meeting sessions. This allowed the participants to quickly gain an understanding of the presented topics as well as allowing them to go into specific details, pros, and cons. Another big plus is that we received immediate feedback on the applicability of the presented topics.

At XebiaLabs we know a thing or two about the automated deployment of Java EE applications . One thing that strikes me as odd is the fact that the people you would expect to know most about application deployment, the application server vendors, don't seem to understand the problem at all.

On one of our previous blogs, you can read up on what we see as the full scope of a Java EE application deployment. The short version is this:

• It's more than just deploying an EAR file or a WAR file. For starters, it can involve more than one of these.
• Most applications also need other kinds of artifacts to be deployed, such as static content for the webserver or static configuration files to be read by the Java code on startup.
• You'll also need to configure Java EE resources such as data sources, JMS providers, etc.
• In most cases you need to configure the middleware itself too. Think about creating and configuring application server clusters, or setting up Apache virtual host configurations.
• All this needs to happen in the right order to minimize (or prevent) downtime of the application during deployment and improve speed of the deployment.

So what do the application server vendors make of this?

Oracle WebLogic Server

Well, Oracle WebLogic Server has a concept called a Deployment; it is the thing you create when you deploy an EAR, WAR or EJB JAR. You can customize its behaviour by supplying or editing a Deployment Plan. But there is no way to bundle multiple Java EE artifacts in one deployment or include the creation of Java EE resources such as data sources.

In WebLogic there is also something called a Subdeployment but, strangely enough, that concept is unrelated to that of a deployment. It is a mechanism that is used to target parts of a JMS System Module to specific JMS servers or WebLogic server instances. The JMS System Module abstraction itself seems to meant to group together JMS resources that belong together. Maybe one of the developers had originally called this thing the Deployment and was then reminded of the other Deployment already in WebLogic. This is all speculation of course, but it would nicely explain why a Subdeployment is called that.

WebLogic offers a number of different deployment APIs. There is the weblogic.Deployer command line tool, the wldeploy Ant task and the Python-based WebLogic Scripting Tool (WLST for short). Of these I think the WLST is the most feature complete. The file system abstraction layered on top of the MBean hierarchy works pretty well so usually it doesn't take too long to figure out how to get something done.

JBoss Application Server

Deployment in JBoss is handled by the Virtual Deployment Framework. This is a very flexible MBean based framework that allows many different kinds of artifacts (EARs, WARs, EJB JARs, RARs, SARs, etc.) and resources (data sources, JMS settings, etc.) to be deployed onto JBoss. All these are handled by their own standard deployer.

However, most people will not manually invoke these deployers, or even twiddle them. Instead most people will drop their artifacts in the deploy directory of the JBoss application server and wait for it to be picked up by the Deployment Scanner. And then they tail the logfiles to see when the deployment has finished.

The big problem with this is that deployment is not an atomic action. When you automate your deployment you only want to restart your web server until after the deployment of your new WAR file has succeeded. But writing code to tail log files and wait for a certain string to appear is just, well, ugly. An alternative would be to set the property ScanEnabled of the DeploymentScanner to false and invoke the specific deployer manually. Not only will this make your code more complex, you will still need to copy your files into the deploy directory for JBoss to pick them up after a restart.

So it turns out that JBoss Application Server makes it hard to integrate deployments to its servers into a bigger deployment scenario. This is something they definitely need to fix to be really ready for the enterprise.

IBM WebSphere Application Server

So how about that big blue's big Java EE beast? It offers a very versatile, transactional way to deploy applications and resources. Basically, the wsadmin scripting interface allows you to control all aspects of WebSphere and it is very clear when certain things have been activated. You can explicitly synchronize nodes and the Jython (or JACL if you are so inclined) method you invoke to deploy an application only returns when it is done. On the downside it is a lot slower than WebLogic's similar WLST tool (it seems to be copying a lot of XML files back and forth over SOAP. Argh!) and the API offered is pretty complex (who here knows the difference between a containment path, a configuration id and an object name?).

While WAS does not make it possible to group together the artifacts and resources that are related to one deployment, its transactional nature allow you to commit related changes to the configuration in one go. It can even manage some of the configuration of installed IBM HTTP Server or Apache HTTP Server instances.

So even though it is down on deployment speed, nothing beats WebSphere as far as deployment reliability is concerned.

Summary

The application servers each provide a different level of support for the complexity of real-life enterprise-level deploymens. Of course they can all deploy the basic Java EE artifacts, but none can group multiple artifacts into one deployment or link the Java EE resources that go with those artifacts. WebLogic is the only application server that tries tries to group together related resources (the JMS System Module and the Subdeployment), these abstractions cover so little of the full scope of Java EE deployment that they are actually in the way when implementing automation deployment.

Repeatability and predictability are very important for a successful deployment strategy, which might explain WebSphere's prevalence in the enterprise market. And finally, good API support is necessary to allow you to automate your deployment process and both WebSphere and WebLogic offer that. JBoss is seriourly lacking this area, twiddle just doesn't cut it!

My colleagues at XebiaLabs and myself have, in our previous jobs, written countless scripts to automate deployments and had to work around these issues again and again. We have now developed Deployit, a Java EE deployment automation product, because we believe deployment automation is of the utmost importance to keep on top of the growing number of deployments in today's enterprise environments.

One of the biggest strengths of Scrum is that it is a framework instead of a detailed methodology such as RUP. In Scrum, concepts are described that make essential aspects of projects fall into place in a very powerful way. One does not need a Process Engineer to tailor Scrum before it can be applied successfully. However, because there are many things that Scrum does not describe in detail, there is plenty of room left to mess things up

In a series of blogs, I want to share some best practices that I have found useful for ScumMasters. In this first blog, I discuss on how to facilitate the estimation of Product Backlog items so that the Product Owner can do release planning.

As Mike Cohn has pointed out in his excellent book Agile Estimating and Planning, planning is done at three levels in Agile projects:

• Daily planning;
• Iteration planning;
• Release planning.

In most Scrum projects, the first two of these levels get the attention they deserve: Daily planning is done at the Daily Scrum, Iteration planning is done at the Sprint Planning Meeting. The third level of planning is where things often get messier. In most projects, there is a need for a planning horizon that is further away than one Sprint. For example, a marketing campaign is planned three months from now and the Product Owner wants to know what features can be expected to be done by then. If the project uses user stories with story point estimates, the following information is needed:

• The velocity of the team(s): how many story points are done each Sprint?
• Estimates in story points for at least all user stories that have a chance of being done before the release date.

I have seen that people struggle with this because Scrum has traditionally not been explicit on when and how to establish estimates for user stories. In many Scrum projects, the only time when estimating is done is the Sprint Planning Meeting. Sometimes user stories are estimated in story points, but these estimates are given only to the same set of user stories for which tasks are identified and estimated subsequently. This really defies the purpose of story point estimates. The team might as well omit story point estimates altogether because the task estimates should be enough to check if it is viable to do the planned work in one Sprint.

My advice is to estimate user stories in Backlog Refinement Meetings separate form Planning Meetings. In these meetings, larger sets of user stories can be estimated than in a Planning Meeting because only just enough detail need to be discussed to estimate user stories relative to each other. At least the product owner and team representatives of all the disciplines that work on the Product Backlog must be present. Especially if multiple teams work from one product backlog, it is not always feasible to include everyone in every session.

Backlog Refinement Meetings are also a good place to discuss how to split up larger user stories into smaller ones and identify user stories that need further clarification. This is the reason why I prefer the term Backlog Refinement Meeting instead of Estimation Meeting. In Sprint Planning Meetings, only estimated user stories are discussed that have been identified to be READY for the team to start to work on.

The best and most widely used technique to estimate Product Backlog items is to estimate in story points using planning poker. If many items need to be estimated in a short time you can also consider an alternative approach that I have lined out in an earlier blog: let the team order them all at once on a horizontal scale and then assign story points.

Now that we have discussed the Backlog Refinement Meeting, note that there are strong analogies between Iteration planning and Release planning, detailed out in the following table:

With this blog I hope that I have helped some ScrumMasters a little further in applying Scrum to do projects successfully. I will now start to think about by next Tip for ScrumMasters ...

In the careers of most Spring/Hibernate developers I know, there sooner or later comes a point of no escape...they have to write a Hibernate user type. The first one is usually of the copy'n'paste variety, and by and large works perfectly well.
But when things are no longer going quite as expected - Hibernate is ignoring changes to items managed by the user type, for instance - it often becomes apparent that one doesn't sufficiently understand how these user type thingies are supposed to work. At least, that's what happened to me.
In this post, we'll be dissecting the Hibernate UserType interface, explaining the relationships between the various methods, and developing a set of base user types that capture common use cases.

The naked truth

Hibernate's UserType interface, in all it's glory (we'll be leaving the more complex EnhancedUserType to one side for the moment), looks like this:

public interface UserType {
    public int[] sqlTypes();
    public Class returnedClass();
    public boolean equals(Object x, Object y) throws HibernateException;
    public int hashCode(Object x) throws HibernateException;
    public Object nullSafeGet(ResultSet rs, String[] names, Object owner) throws HibernateException, SQLException;
     public void nullSafeSet(PreparedStatement st, Object value, int index) throws HibernateException, SQLException;
    public Object deepCopy(Object value) throws HibernateException;
    public boolean isMutable();
    public Serializable disassemble(Object value) throws HibernateException;
    public Object assemble(Serializable cached, Object owner) throws HibernateException;
    public Object replace(Object original, Object target, Object owner) throws HibernateException;
}

Of the methods, returnedClass, nullSafeSet and nullSafeGet are probably the most self-explanatory. They are likely also the ones you tweaked for your first user type. Indeed, they are perhaps the only ones one would really want to have to deal with as a user type implementor, and making this (almost) possible is one of the main purposes of the base classes we will develop.

What about the other methods, assemble, replace and the like? These used to be the ones whose implementation one copied from the example in semi-blind faith, although now that the Javadoc for them is much more detailed they will hopefully be more straightforward to implement. Still, the following "default" implementation is still common:

@Override
public boolean isMutable() {
    return false;
}

@Override
public boolean equals(Object x, Object y) throws HibernateException {
    return ObjectUtils.equals(x, y);
}

@Override
public int hashCode(Object x) throws HibernateException {
    assert (x != null);
    return x.hashCode();
}

@Override
public Object deepCopy(Object value) throws HibernateException {
    return value;
}

@Override
public Object replace(Object original, Object target, Object owner)
        throws HibernateException {
    return original;
}

@Override
public Serializable disassemble(Object value) throws HibernateException {
    return (Serializable) value;
}

@Override
public Object assemble(Serializable cached, Object owner)
        throws HibernateException {
    return cached;
}

What's wrong with this? Apart from the clutter, nothing, really...as long as you intend your objects to be treated as immutable.

Building up a case

Right, time to leave these heady abstract hights and cook up the usual artificial example. Imagine, thus, that you want to persist an entity with a StringBuilder property for, oh, I don't know, storing a frequently-updated history, for instance.
Now, StringBuilder is, of course, serializable, so can be handled by JPA (and thus Hibernate) out of the box. But what you get in the database isn't incredibly practical: a value of "Agent 007 entered the secret hideout" might end up as

aced0005737200176a6176612e6c616e672e537472696e674275696c64
65723cd5fb145a4c6acb0300007870770400000024757200025b43b0266
6b0e25d84ac020000787000000024004100670065006e0074002000300
030003700200065006e007400650072006500640020007400680065002
000730065006300720065007400200068006900640065006f0075007478

Useful, eh?

Instead, we'll do the obvious thing and persist the builder's string representation. Following the trusty copy'n'paste approach outlined above, we end up with:

public class ReadableStringBuilderUserType implements UserType {

    public Class<StringBuilder> returnedClass() {
        return StringBuilder.class;
    }

    public int[] sqlTypes() {
        return new int[] { Types.VARCHAR };
    }

    public Object nullSafeGet(ResultSet resultSet, String[] names, Object owner)
            throws HibernateException, SQLException {
        String value = (String) Hibernate.STRING.nullSafeGet(resultSet, names[0]);
        return ((value != null) ? new StringBuilder(value) : null);
    }

    public void nullSafeSet(PreparedStatement preparedStatement, Object value, int index)
            throws HibernateException, SQLException {
        Hibernate.STRING.nullSafeSet(preparedStatement,
                (value != null) ? value.toString() : null, index);
    }

    /* "default" implementations */

    @Override
    public boolean isMutable() {
        return false;
    }

    @Override
    public boolean equals(Object x, Object y) throws HibernateException {
        return ObjectUtils.equals(x, y);
    }

    @Override
    public int hashCode(Object x) throws HibernateException {
        assert (x != null);
        return x.hashCode();
    }

    @Override
    public Object deepCopy(Object value) throws HibernateException {
        return value;
    }

    @Override
    public Object replace(Object original, Object target, Object owner)
            throws HibernateException {
        return original;
    }

    @Override
    public Serializable disassemble(Object value) throws HibernateException {
        return (Serializable) value;
    }

    @Override
    public Object assemble(Serializable cached, Object owner)
            throws HibernateException {
        return cached;
    }

}

Apart from the null checks that one has to watch out for, very straightforward stuff. And it works, too! We can load a StringBuilder from the DB, and if we set a new StringBuilder it is correctly persisted. Bingo!

Not so fast

...until we try the following¹, that is:

@Test
public void dehydrateModified() {
    EntityWithStringBuilderProperty holder = loadEntity(EntityWithStringBuilderProperty.class, id);
    StringBuilder builder = holder.getBuilder();

    String addition = " Bond";
    builder.append(addition);
    session.flush();
    session.evict(holder);

    StringBuilder persistedBuilder =
        ((EntityWithStringBuilderProperty) loadEntity(EntityWithStringBuilderProperty.class, id))
        .getBuilder();
    assertNotSame(builder, persistedBuilder);
    assertEquals(builder.toString(), persistedBuilder.toString());
}

Bang! The value loaded is still the original value, and the final assertion fails. From the SQL it looks like Hibernate hasn't picked up the modification, for some reason:

Hibernate: select entitywith0_.id as id0_0_, entitywith0_.builder as builder0_0_ from EntityWithStringBuilderProperty entitywith0_ where entitywith0_.id=?
Hibernate: select entitywith0_.id as id0_0_, entitywith0_.builder as builder0_0_ from EntityWithStringBuilderProperty entitywith0_ where entitywith0_.id=?


Read before you paste

Looking again at our copy'n'paste implementation, the problem seems clear: no wonder Hibernate isn't picking up changes if we declare the type to be immutable²! Well, that's easily fixed:

/* "Maybe if we make it mutable..?" */

@Override
public boolean isMutable() {
    return true;
}

...

@Override
public Object deepCopy(Object value) throws HibernateException {
    return value;
}

/* should return copies for mutables, according to the documentation */

@Override
public Object replace(Object original, Object target, Object owner)
        throws HibernateException {
    return deepCopy(original);
}

@Override
public Serializable disassemble(Object value) throws HibernateException {
    return (Serializable) deepCopy(value);
}

@Override
public Object assemble(Serializable cached, Object owner)
        throws HibernateException {
    return deepCopy(cached);
}

me != me?

Nice try, but unfortunately no cigar yet: Hibernate still fails to realise that the property has been modified. But why? Hibernate should call the user type to determine if the object being persisted is equal to the one originally loaded, and our implementation of equals(Object x, Object y) should surely correctly determine that the object has been modified.

Indeed, setting a breakpoint on equals confirms that it is called, so what is going on? The breakpoint helps identify the problem: equals is being called with both arguments being references to the same, modified, StringBuilder object! So of course equals doesn't detect the change - it would be a strange equals indeed that obeyed equals(x, x) == false.

It appears, therefore, that Hibernate is "losing track" of the originally loaded state³ of the object, which makes it impossible for the user type to detect the change. A bug in Hibernate?

No. Digging a little deeper⁴ into the Hibernate code, it turns out that Hibernate's snapshot of the loaded state of an object is generated by a call to the user type's deepCopy method...from which our user type is simply returning a reference to the loaded object!

Without a clean copy of the loaded state (Hibernate's "clean" copy is being updated whenever we update the loaded object), it's no surprise that Hibernate can't detect that the object has been modified.

Now with copying

For a StringBuilder, deepCopy is trivially implemented:

private static StringBuilder nullSafeToStringBuilder(Object value) {
    return ((value != null) ? new StringBuilder(value.toString()) : null);
}

/* "Maybe if we make actually create a copy..?" */

@Override
public Object deepCopy(Object value) throws HibernateException {
    return nullSafeToStringBuilder(value);
}

And lo and behold, this finally does work as expected:

Hibernate: select entitywith0_.id as id0_0_, entitywith0_.builder as builder0_0_ from EntityWithStringBuilderProperty entitywith0_ where entitywith0_.id=?
Hibernate: update EntityWithStringBuilderProperty set builder=? where id=?
Hibernate: select entitywith0_.id as id0_0_, entitywith0_.builder as builder0_0_ from EntityWithStringBuilderProperty entitywith0_ where entitywith0_.id=?


Be gone, boilerplate!

So far, so good: we have a working user type. But we also have a tremendous amount of boilerplate: for the five methods (returnedClass, sqlTypes, nullSafeGet and -Set and deepCopy) containing six lines of code that actually deal with StringBuilders there are six methods dealing with "generic" functionality.

The first step in our quest to simplify our user type is, therefore, to extract the generic stuff into a base class:

public class ReadableStringBuilderUserType extends MutableUserType {

    public Class<StringBuilder> returnedClass() {
        return StringBuilder.class;
    }

    public int[] sqlTypes() {
        return new int[] { Types.VARCHAR };
    }

    public Object nullSafeGet(ResultSet resultSet, String[] names, Object owner)
            throws HibernateException, SQLException {
        return nullSafeToStringBuilder(Hibernate.STRING.nullSafeGet(resultSet, names[0]));
    }

    private static StringBuilder nullSafeToStringBuilder(Object value) {
        return ((value != null) ? new StringBuilder(value.toString()) : null);
    }

    public void nullSafeSet(PreparedStatement preparedStatement, Object value, int index)
            throws HibernateException, SQLException {
        Hibernate.STRING.nullSafeSet(preparedStatement,
                (value != null) ? value.toString() : null, index);
    }

    @Override
    public Object deepCopy(Object value) throws HibernateException {
        return nullSafeToStringBuilder(value);
    }

}

Extending from MutableUserType removes the boilerplate from our user type, leaving only StringBuilder-related code. Nice!

commons-hibernate-usertype⁵ also contains, amongst others, ImmutableUserType and XStreamableUserType, which provides convenient (and readable!) Object-to-XML serialization.

When equals equals trouble

Performing an equals check on a StringBuilder of many thousands of characters could be an expensive operation that we'd rather avoid. It might be much cheaper, for instance, to set a "dirty" flag whenever the StringBuilder is modified and then simply check this when deciding if an update is necessary. The DirtyCheckableUserType base class is intended for this kind of use case.

Here's a (not very efficiently implemented) DirtyCheckingStringBuilder we'll use as an example:

public class DirtyCheckingStringBuilder {
    private StringBuilder value;
    private boolean valueModified;
    ...

    public DirtyCheckingStringBuilder(String value) {
        assert (value != null);
        this.value = new StringBuilder(value);
        valueModified = false;
    }

    public DirtyCheckingStringBuilder append(String addition) {
        value.append(addition);
        valueModified = true;
        return this;
    }

    ...
}

Using the DirtyCheckableUserType base class, we can then create a user type that will avoid calling equals on "dirty" builders⁶.

public class DirtyCheckingStringBuilderUserType extends DirtyCheckableUserType {
    ...

    @Override
    protected boolean isDirty(Object object) {
        return ((DirtyCheckingStringBuilder) object).wasModified();
    }

    @Override
    public Object deepCopy(Object value) throws HibernateException {
        return nullSafeToStringBuilder(value);
    }

}

Usually, of course, it is likely that, if equals is expensive, then so will the deepCopy be that produced the copy to be compared against in the first place. With dirty checking, it is no longer necessary for the loaded state recorded by Hibernate to actually be a genuine copy, so you may consider simply returning the input value in deepCopy.

@Override
public Object deepCopy(Object value) throws HibernateException {
    return value;
}

However, since deepCopy is also called by the default implementations of assemble and other methods, this is likely to cause problems unless you're certain that your object will not be serialized to cache and you will not be performing merges. One possible option would be to override the implementation of these other methods to return a "real" copy⁷:

@Override
public Object deepCopy(Object value) throws HibernateException {
    return value;
}

@Override
protected Object realDeepCopy(Object value) throws HibernateException {
    // return an actual deep copy here
}

public Object assemble(Serializable cached, Object owner)
        throws HibernateException {
    // also safe for mutable objects
    return realDeepCopy(cached);
}

public Serializable disassemble(Object value) throws HibernateException {
    // also safe for mutable objects
    Object deepCopy = realDeepCopy(value);
    assert (deepCopy instanceof Serializable);
    return (Serializable) deepCopy;
}

public Object replace(Object original, Object target, Object owner)
        throws HibernateException {
    // also safe for mutable objects
    return realDeepCopy(original);
}

Concurrent testing is hard, but not as hard as you think. If you use the right tricks it can be done. This blog shows you one particular trick that uses a latch and a mock to ensure a test scenario is completed before running the verifications.

While working on Spring Integration in Action, I experimented with a neat solution for concurrent tests. When I showed it to some colleagues I was pleasantly surprised by the reaction that I got. Judge for yourself if it's worth the blog.

The main idea is to use a latch and let your mock count it down. It sounds trivial (and to be honest it is).

First let me show you how it would be done without mocking:

@ContextConfiguration
@RunWith(SpringJUnit4ClassRunner.class)
public class ConcurrentTest {

  @Autowired @Qualifier("in") MessageChannel in;

  @Autowired @Qualifier("out") PollableChannel out;

  @Autowired Service service;

  @Test(timeout = 5000)
  public void shouldGoThroughPipeline() throws Exception {
    in.send(MessageBuilder.withPayload("test").build());

    service.invoked.await();
    out.receive();
  }

  public static class Service {
    private CountDownLatch invoked = new CountDownLatch(1);

    public String serve(String input) {
      invoked.countDown();
      return input;
    }
  }
}

A message is sent on channel "in", which will result in an asynchronous invocation of the service. I'm using Spring Integration to make the test asynchronous, but I could just as easily have used @Async, Camel, JMS or what not. The basic idea remains the same.

This is already mostly painless, but we clean up the test a bit more by using a mock. You can wire a mock into your context using a factory-method:

<bean id="service"  class="org.mockito.Mockito" factory-method="mock">
  <constructor-arg value="iwein.samples.test.concurrent.ConcurrentTest$Service"/>
</bean>

Now this mocked service will be autowired into the test and you can make it do tricks:

  @Test(timeout = 5000)
  public void shouldGoThroughPipeline() throws Exception {
    final CountDownLatch serviceInvoked = new CountDownLatch(1);
    given(service.serve("test")).willAnswer(new Answer(){
      public Object answer(InvocationOnMock invocationOnMock) throws Throwable {
        serviceInvoked.countDown();
        return "test";
      }
    });

    in.send(MessageBuilder.withPayload("test").build());

    serviceInvoked.await();
    out.receive();
  }

The answer passed into willAnswer() method will first count down the latch and then return "test". This works, but it's not the most readable code. Let's tidy it up a bit:

@ContextConfiguration
@RunWith(SpringJUnit4ClassRunner.class)
public class ConcurrentTest {

  @Autowired @Qualifier("in") MessageChannel in;

  @Autowired @Qualifier("out") PollableChannel out;

  @Autowired Service service;

  @Test(timeout = 5000)
  public void shouldGoThroughPipeline() throws Exception {
    //given
    final CountDownLatch serviceInvoked = new CountDownLatch(1);
    given(service.serve("test")).willAnswer(latchedAnswer("test", serviceInvoked));

    //when
    in.send(MessageBuilder.withPayload("test").build());
    serviceInvoked.await();

    //verify
    Message<?> message = out.receive();
    assertThat((String) message.getPayload(), is("test"));
  }

  private <T> Answer<T> latchedAnswer(final T returning, final CountDownLatch latch) {
    return new Answer(){
      public Object answer(InvocationOnMock invocationOnMock) throws Throwable {
        latch.countDown();
        return returning;
      }
    };
  }

  //You'd have a class in main to mock, but an interface here will serve the example
  public static interface Service {
    public String serve(String input);
  }
}

That's really polite code for a concurrent test don't you think?

I'm thinking of adding the latchedAnswer method into a test utility that comes with Spring Integration, but as you can see it fits quite well in your copy buffer too. The point is that even without using a concurrent test framework (I know they're out there) simple concurrent tests can be as simple as they should be.

Caveats:

• If you pass an instance of the wrong type into latchedAnswer you'll not get a compiler warning, because of a generics issue with Mockito
• When you replace a service with a mock, frameworks using reflection around it might get a bit confused. For this example I had to add a method attribute to the service activator element that wouldn't have been needed without the mock.
• Allways set a timeout. You don't want the test to run eternally on your build server if you are unfortunate enough to break it.

Source code for this example can be found on github. Enjoy!

Last time I blogged about the relevance of monitoring and diagnostics in production to solve incidents quickly and prevent future problems. This time I'll talk about tuning based on evidence.

If an application turns out to be too slow, tuning can provide a solution. Tuning can take place on multiple levels. Adding hardware can be a cheap solution. However, when you add hardware at a place where the bottleneck is not located, this has little use.

Important steps of tuning are therefore identifying which pages or services do not meet stated requirements and isolating the problem: where is it located, in which layer, in which component. This can be made clear with testing and monitoring on application parts. The next step is diagnosing. In fact, this comes down to making up a hypothesis why this component is so slow.

Figure 1. Performance tune cycle.

This can for instance be a missing or wrong index on a database table or the invocation of too many small queries. Next, the component is improved based on this hypothesis. Finally, one needs to verify whether the improvement actually brings the expected speedup. If so, then the proposed hypothesis is true and the speedup is the result. If not, then there is something wrong with the hypothesis and we need an alternative hypothesis. As soon as the performance of the system meets its requirements, tuning is finished.

Figure 2. Finding evidence.

The right tools are indispensable: performance test tool, enterprise profiler, heap monitor, etc. I have seen developers work on assumed performance improvements which turned out not to help at all, or even slowed down the application and also deteriorate the maintainability and flexibility. This is caused by the fact that developers are used to mould functionality from source code and therefore work from source code to improve performance. What is missing here is measure, don't guess. This is something developers learn in my performance training. Experience also has taught me to judge every proposed improvement separately and to only implement the improvement when we have proven that it really helps. Without this systematic approach, a handful of steps together may take you backwards at the end of the day instead of forwards.

Next time I'll talk about Sharing responsibility for the whole chain.

In many Scrum projects, user stories that are Done at the end of a sprint have not yet been put into production. In other words, production is often not part of the Definition of Done. There can be several reasons for this. Examples are:

• additional acceptance or integration testing is needed that can not (yet) be done within the sprint;
• user stories have dependencies on hardware of software from an external party that is not yet available;
• minimal marketable features are bigger than what can be produced in one sprint.

I have seen several Scrum projects where the work that was needed to bring Done user stories to production was far less transparent than the work that was performed in a sprint. Have you ever seen acceptance testers being frustrated because software was not integrated and deployed correctly? A good first step to improve such things is to introduce some transparancy: which steps are needed, what work is in progress, is there a bottleneck? Besides the Ready Kanban described by Serge Beaumont in an earlier blog, we introduced the Done Kanban.

If you combine this Kanban with the Scrum Taskboard and a Ready Kanban, the whole flow of user stories form idea to production is visualized. This enhances transparency of work that is done to user stories outside of a sprint. Below is a picture of a Scrum Taskboard that is combined with two Kanbans. The Ready Kanban shows user stories that are made Ready to be picked up in a sprint. The Done Kanban shows the flow of user stories from Done to production.

Below the Done Kanban is shown at a later point in time. By this time, it had really proven its value. In the project, user stories were integrated one-by-one with code that was made by non-Scrum teams with a longer release cycle. After introduction of the Done Kanban, this integration became a joined effort instead of a single person's job. It also became more clear to everyone which user stories had been put into production.

To conclude, user stories often exist before they are Ready to be picked by a team in a Sprint. After they are Done they are often not yet in production. In combination with the Scrum Taskboard, Ready and Done Kanbans are a great way to visualize the complete flow of user stories form idea to production.

It took me a while before I really understood what this @Rule business in JUnit 4.7 is really about, but I like it!

Creating a Web Server

interface MethodRule {
Statement apply(final Statement base, FrameworkMethod method, Object target)
}

@Rule
public WebServer server =
new WebServer("WEBSERVER_PORT", 9191);

Environment Variables?

Which Resource?

Annotations to the rescue

@Test
@WebResource(content="classpath:whatever.xml", contentType="text/xml")
public void shouldBeAbleToDealWithWhatever {
String url = server.getURL();
// Download something from that URL using your client
}

Conclusions

Attending the FOSS4G conference in Sydney I have been attending a lot of presentations and involved in discussions about the OpenLayers JavaScript framework. see also my previous blog post on My Opensource GIS experiences.

Especially the Birds Of a Feather session yesterday made me really enthusiastic about the next level of OpenLayers.

Changes to come?

The discussion in the Birds of a Feather session were mostly about the following subjects:

Documentation

Documentation is always a big issue within open source projects. Its always difficult to write good documentation without any clear objective or guideline from the people using the documentation. In an opensource project you not only have to deal with the users (who you don’t know) but also a lot of developers with mixed experience.

Examples

There are a lot of examples on how to use OpenLayers. The only problem is that these are not all written with the same style. Some of them are still using some old school way of dealing with the OpenLayers code. It would be great if there could be a set of examples using the latest and greatest way of handling different use cases, written in the same style of coding. This prevents new time users to build of a deprecated example and asking all kinds of questions on the mailing list.

Separation of UI look and feel

The current control elements are containing both functionality and representation stuff so it is considered a good thing to separate those two concerns. The control object would be modified to only contain the specific functionality and launch events on which the UI layer could base its presentation. This way it would be a lot easier to substitute the current basic looks of OpenLayers whit something fancier with the help of Ext, GeoExt or JQueryUI.

HTML5

With the new html5 functionality around the corner it would be great if we can start using the geolocation api to for example center the map on your current position, use a canvas for the map and its layers to minimize the dom manipulation which causes big memory problems (at least in IE).

This will be a big change from the current API since i imagine the Map to be a 3d (or 2d) canvas where you can add your image tiles and features from different layers. Would be awesome to be able to use 3d and create a globe with your tiles on it!!

Summary

I personally want to work on the HTML5 stuff (2d at first) creating a new Map object and adding tiles and features to it would be the first go i think.

I know some people are afraid this would break the current way OpenLayers is working. So one thing i want to know from whoever is reading this: Is this really breaking the current API or is it just adding new and cool stuff?

Last week me and some of my colleagues had the pleasure of being on the receiving end of an excellent training given by Jonas Bonér. The topic was his new pet project: the Akka framework. Perhaps you've played around with Scala lately, and also have taken the first steps in using its Actor library. Simply stated, an Actor is a unit of execution that (usually asynchronously) processes messages and encapsulate their state. An actor does not expose its state, and messages are processed sequentially. The Actor model has been around for quite some time, but today the best-known Actor implementation is Erlang.
The Actor model has been implemented in the standard Scala library by Philipp Haller (for the interested reader, a solid reference is for instance this article explaining how actors in Scala work). In most Actor examples written in Scala, it is not uncommon to find only EchoActors, PingPongActors, and FibonacciSolvingActors. Nice examples, but perhaps you might wonder if they are of any use in enterprise Scala at all. Next to this, if you're interested in concurrent, message passing processing models, STM's, NoSQL data stores, and occasionally wonder what the future in enterprise computing might bring, than Akka might be just the framework you're looking for. This blog is intended to provide a brief introduction into one feature of this framework: Akka's supervisor Actors. It is mostly based on the knowledge extracted from Jonas during the training, and I hope to whet your appetite for it.

First of all, Akka is likely to support what Ray Racine has coined to be a 'System of Service'. I'm quoting him here literally from the scala mailing list:

Its not too far of stretch to say commercial vendors, and most frameworks are very System Of Record oriented. Need to build a new app to maintain X.
Install RDB, app server, select O/R mapping and GUI form framework, place warm bodies in front of drag and drop IDEs and go for it.

A System Of Service must service 1,000s of requests per second in millisecs, 24/7/365 with 99.99 % reliability e.g. a pricing service. The business
logic has to run in microseconds. Your favorite O/R mapping framework hasn't even initiated a JDBC call, heck hasn't even allocated a connection from the pool and its already exhausted its 1 ms in allotted time.

An item may undergo a few 10s of price changes a year on the System Of Record, yet that item's price may be served 100,000 times for each change on the System Of Service'.

In other words a System of Service is a system designed to have extreme throughput, and almost zero downtime. It may not be your everyday CRUD application, but if you need to build them, it is nice to know of some framework that supports it. In response to Ray Racine's post Jonas Bonér posted his vision for the Akka framework. These posts provide plenty of highly interesting reading material, and can be found here.

Akka is therefore meant to be a kind of framework that supports a System Of Service application. It has many features, too much to cover all at once, and therefore we will start of with having a look at Akka's actors supervisor capabilities.
Akka has an Actor model with supervision trees, based on Erlang's OTP design principles. Supervisors are processes that monitor a set of workers, that do the actual processing. The basic idea of a supervisor is that it should keep its child processes alive by restarting them when necessary. A worker is an Actor, that receives and processes a message. When a worker dies abnormally (by throwing an Exception), it is up to the supervisor to determine what to do. It may decide to restart the worker, and try to process the message again, with a maximum number of retries set. The approach is to expect that failures can and eventually will happen. Instead of trying to prevent this, the idea is to Let it crash (TM Jonas Bonér), reset the worker to a stable state, restart it. This idea has served Ericsson well in building highly fault tolerant, distributed message passing software. We can now profit from this idea in Scala as well via Akka.

In order to achieve this, Akka has its own version of the Actors library, that have more 'system of service' features than the standard Actors library found in the Scala distribution. To demonstrate a simple example of a supervisor class, here's some sample code to demonstrate the idea. First, we start of with a simple Actor:

import se.scalablesolutions.akka.config.ScalaConfig._
import se.scalablesolutions.akka.actor.{OneForOneStrategy, Actor}

sealed trait Message

case class Supervise(worker: Worker) extends Message

case class DoWork(work: Work) extends Message

case object Die extends Message

class Worker(workerName:String) extends Actor {
lifeCycleConfig = Some(LifeCycle(Permanent, 100))

def receive: PartialFunction[Any, Unit] = {

case DoWork(work:String) =>
log.debug("start working... at: " + work)

case Reset =>
log.info("%s has been reset", toString)

case Die =>
log.debug("Dying...")
throw new RuntimeException("I'm dead: " + this.toString)

case other =>
log.error("Unknown event: %s", other)
}

override def preRestart(reason: AnyRef, config: Option[AnyRef]) {
log.debug("pre-restarting " + this)
}

override def postRestart(reason: AnyRef, config: Option[AnyRef]) {
log.debug("post-restarting " + this)
}

override def toString = "[" + workerName + "]"

As regards to sending message passing, an Akka actor follows the same API as the 'normal' Scala ones: you can send a message to an actor using the ! (pronounced bang) method. This is a 'fire and forget' message, which is then processed asynchronously by the actor. There are some more options to send messages, but we'll skip these for now. In Akka, Actors have to be started explicitly by calling actor.start for it to be able to process any messages. It can be stopped by calling actor.stop.
Compared to the standard Scala actors, the API for processing messages is more limited: there is exactly one method that you must override. This is the receive method, which must return a PartialFunction. The receive method uses a pattern match to distinguish between various messages it can process. A pattern match is compiled down to a PartialFunction, which is the return value of the pattern match.
To send messages to actors, we've defined a Message trait, which various case classes that represent the actual messages extend. In principle, you can send any message to an Actor, however, in practice however, the only messages you should send should be immutable. In Scala, case classes and case objects are immutable by default, and thus are a natural fit for this. In this case, we've defined a 'DoWork' message (all messages extends from the same Message trait), which should set the Worker actor to do something that hopefully will be be of some use. When the worker receives a Die message, it will throw an exception. This is a somewhat silly example of course, but hopefully it demonstrates the idea.

So far, nothing special, apart perhaps the pre- and postRestart methods, and the field lifeCycleConfig = Some(LifeCycle(Permanent, 100)). This is configuration for Akka's actors supervisor hierarchy capabilities. To see further how this works, let's code up a supervisor:

import se.scalablesolutions.akka.actor.OneForOneStrategy

object WorkerSupervisor extends Actor {
trapExit = true
faultHandler = Some(OneForOneStrategy(3, 100))

def receive: PartialFunction[Any, Unit] = {
case DoSupervise(worker:Worker) =>
log.info("Supervising worker: " + worker)
startLink(worker)

case unknown =>
log.error("Unknown event: %s", unknown)
}
}

Our supervisor is also an Akka actor (in this case, it is an object, i.e. a singleton in Scala), that processes 'DoSupervise(worker)' messages. It handles this message by starting to supervise the worker by the invoking the 'startLink(worker)' method, which is defined in the Actor class. This will add our worker to the list of its linked actors, and also handles starts the linked Actor at the same time. Of cours the nice thing is that the Actor class will handle thread safety for us, so we don't need to worry about any multithreading issues here.
The linking achieves exactly what it promises: if a linked Actor (the worker in our case) throws an exception, the supervising actor (our WorkerSupervisor) will be notified of this. For this, it is necessary to override the variable trapExit, and set it to true. It may then decide what to do based on the fault handler strategy Strategy that is defined by defining the faultHandler variable. There are currently two strategies: OneForOne, meaning that only the actor that has crashed will be restarted, and AllForOne, meaning all linked actors (including the crashed one) will be restarted. This is also where the lifeCycleConfig defined on our worker Actor comes in. LifeCycle(Permanent, 100) means our actor has a permanent LifeCycle and will always be restarted in case of an exception. The preRestart and postRestart are callback methods that may be overridden by the supervised actor to do some initialization work when it is restarted. Note that our supervised actors will probably be some kind of application services, instead of our useless Worker actor, and therefore may really need to do some usefull initialization work like setting up resources, connections, etc, in order to operate properly.

Instead of manually coding a Supervisor like we've done, there's an alternative in setting up a set of linked services in a more declarative way by extending Akka's SupervisorFactory class, as follows:

import se.scalablesolutions.akka.actor.SupervisorFactory
import se.scalablesolutions.akka.config.ScalaConfig.{LifeCycle, SupervisorConfig, RestartStrategy, Supervise, OneForOne, Permanent}

class MySupervisorFactory extends SupervisorFactory {

val worker = new Worker("worker-1")

override protected def getSupervisorConfig: SupervisorConfig = {
SupervisorConfig(
RestartStrategy(OneForOne, 3, 10),
Supervise(
worker,
LifeCycle(Permanent, 1000)) :: Nil)
}
}

object factory extends MySupervisorFactory

This defines a list of supervised actors (in our case, a one element list containing our worker) and then use the following code:

val supervisor = factory.newSupervisor
supervisor.startSupervisor

which will link all workers to one supervisor (defined within the framework) and (re)start them based on the configured strategy. You can therefore pick your preferred way of configuring.

All that remains is to see whether it actually works. For this we need a small simulation class, defined as follows:

class Simulation {

var worker:Worker = _

@Before
def setUp = {

worker = new Worker("worker-1")
WorkerSupervisor ! DoSupervise(worker)
}

@After
def tearDown = {
WorkerSupervisor.stop
}

@Test
def testSuperviseWorker = {
Thread.sleep(500)
println("\n===> start working")

worker ! DoWork("Some work")
Thread.sleep(500)

worker ! Die
Thread.sleep(1000)

worker ! DoWork("Some more work")

println("\n===> finished")

}
}

Running this shows output like the following:

INF [20091020-19:28:25.242] akka: Supervising worker: [worker-1]
DEB [20091020-19:28:25.242] akka: Linking actor [[worker-1]] to actor
[Actor[1256143591312:class akka.supervision.WorkerSupervisor$]]

===> start working
DEB [20091020-19:28:25.750] akka: start working... at: Some work
DEB [20091020-19:28:26.249] akka: Dying...
java.lang.RuntimeException: I'm dead: [worker-1]
	at akka.supervision.Worker$$anonfun$receive$1.apply(Worker.scala:48)
	at akka.supervision.Worker$$anonfun$receive$1.apply(Worker.scala:38)
	at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:38)
	at se.scalablesolutions.akka.actor.Actor$class.transactionalDispatch(Actor.scala:496)
	at se.scalablesolutions.akka.actor.Actor$class.invoke(Actor.scala:461)
	at akka.supervision.Worker.invoke(Worker.scala:33)
	at se.scalablesolutions.akka.actor.ActorMessageInvoker.invoke(Actor.scala:39)
	at se.scalablesolutions.akka.reactor.EventBasedThreadPoolDispatcher$$anon$1$$anon$2.
            run(EventBasedThreadPoolDispatcher.scala:99)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
DEB [20091020-19:28:26.260] labs: pre-restarting [worker-1]
INF [20091020-19:28:26.262] labs: Restarting actor [class akka.supervision.Worker]
                                                         configured as PERMANENT.
DEB [20091020-19:28:26.265] labs: post-restarting [worker-1]

===> finished
DEB [20091020-19:28:27.251] labs: start working... at: Some more work

Process finished with exit code 0

Which indeed shows that our rudimentary supervisors work.
This concludes the introduction into Akka's supervising actors. There is much more that Akka's actors are capable of than shown here (remoting, transactions, STM, cassandra and mongo backend, to name just a few), but those are materials for future blogs. It may be perhaps a bit much to take all at once, as it requires to learn an API that is likely to be unknown to most of us. I won't deny that the Akka framework certainly has some learning curve, but the reward is great once you get the hang of it.
Akka is relatively young and still evolving. If you're interested in its future directions, check out the roadmap for the 0.6 release.

In my efforts to teach myself Scala, I tried solving a problem I've tackled in various languages, 6510 assembly code (didn't get far...), pl/sql, Java (with and without Drools) and Groovy among them. Usually I get bogged down in some detail of the language so I never get to reap any actual benefits of my efforts in daily life. The plus side of this never ending task is that by now I don't have to spend effort on defining a problem but instead can start coding right away.
So this story is about how to parse text in Scala and is part of THE software package that will automagically generate a menu for a week and the shopping list for that menu together with whatever else my family will need that week and send it to www.albert.nl and have the groceries delivered to my door.

Thinking big and starting small, I set about (again) parsing a set of recipes so I can extract the list of ingredients necessary for the shopping list. Since all of my practical knowledge of Scala came from reading the book by Martin Odersky, there was some evolution in my code. As we go, I'll try and amuse you with some of the clunky contraptions I've created along the way while trying to shake of old habits.

The aim is to parse a list of recipes like the one below:

name ZucchiniWithMinceMeat
season summer
category easy
cookingtime 45 minutes
preheat oven at 180 degrees
fry 500 gram mincemeat
fry 1 piece Zucchini
add 1 can mashedTomatoes
add 1 bag groundCheese
bake for 30 minutes

and turning it into a list of groceries like this:

mincemeat 500 gram
Zucchini 1 piece
mashedTomatoes 1 can
groundCheese 1 bag

Scala offers a parser for text named StandardTokenParsers in the package scala.util.parsing.combinator.syntactical. The idea is that a specific parser extends StandardTokenParsers. The parser should provide

  lexical.delimiters ++= List("(", ")", ",", " ", "\n")
  lexical.reserved ++= List("name", "season", "fry", "pack", "can", "bag", "piece", "at", "for", "bottle", "gram", "oven", "preheat", "bake", "serve", "with","add","degrees", "category", "cookingtime")

  def parseCookBook(cookBook:String):CookBook = {
		cookbook(new lexical.Scanner(cookBook)) match {
		  case Success(recipeList, _) => new CookBook(recipeList)
 		  case Failure(msg, _) => throw new ParseErrorException(msg)
		  case Error(msg, _) => throw new ParseErrorException(msg)
		}
  }

  def recipeName: Parser[String] =
    "name" ~ ident ^^ { case "name" ~ name => name }

  def ingredient: Parser[Ingredient] =
	opt(amount) ~ ident ^^
  	{ case Some(amount) ~ ingredient => (new Ingredient(ingredient,amount))
  	  case  _ ~ ingredient => {new Ingredient(ingredient)}
  	}

  def step: Parser[Step] =
	(fryStep | preHeatStep | serveWithStep | addStep | bakeStep) ^^ { case step => step }

  def listOfSeasons: Parser[List[Seasons]] =
    repsep(season ,",") ^^ { listOfSeasons: List[Seasons] => listOfSeasons }
}

  override def toString = {
  	val result:StringBuilder = new StringBuilder("Steps: ") 

   	list.foreach(step => result.append(step).append("\n"))
  	result.toString()
  }

	override def toString = "Steps: " + steps.mkString("\n")

  def findRecipesBySeason(season:Seasons):Seq[Recipe] = {
    recipes.filter(_.seasons.contains(season))
  }

public class Amount {
	private int amount;
	private String unitName;
	public Amount(int amount, String unitName) {
		this.amount=amount;
		this.unitName=unitName;
	}
	// getters and setters omitted for brevity...
}

class Amount (val amount:Int, val unitName:String) {
	override def toString = amount+":"+unitName
}

If you happen to be in the business of writing software serving XML documents or consuming XML documents - and if you read this post, then there is a fair chance you are - then there is always one big challenge: how do you make sure your service or client is capable of dealing with all of the XML documents you could possibly expect to be passed around?

And if you happen to come from the test-driven world, the answer is obviously: by testing it. However, if you try to do that, things might be harder than you expect at first.

What about schemas?

I clearly remember having to integrate with Google's Local Search Service. We managed to get them send us their schema, but the schema was merely illustrative, rather than normative. In fact, it didn't even 'parse' correctly. It was supposed to be a DTD, but in reality, it wasn't. In that case, you are basically lost. The only thing that you can really do is 'test by poking around', trying to see what the web service is going to reply, and then work into your test harness.

If you do however manage to get a schema, then you are still not done yet. Sure, if it's about SOAP based web services, then you might be able to generate stubs and skeletons, and those stubs and skeletons would give you some guarantee that you are covering most cases. But then there is still a chance that you would not cover for all cases, since - inside your XML document - there might be alternatives for content models, and you might - when you would implement your service - only be dealing with one of them.

If the schema is small, then you can probably figure it out by careful examination. However, if the schema is huge, then the range and variety of XML document instances that you might get will make that impossible. And even if you created the schema yourself, it might sometimes cover for a wider range of options than you expected. (I'm sure, I am not the only one who experienced this. )

XML Instance Generator to the rescue

So, back to test-driven. The good news is, there are tools that take a schema, and generate random instances, basically walking all of the different options. Xmlgen is one of those tools. It's a little bit hard to find these days. If you follow the 'XML Instance Generator' link on Kohsuke's homepage, you will end up in no-mans land. I dug a little further, and found out it's currently hosted at Sun's dev.java.net.

Xmlgen is extremely simple. It takes a schema (any schema language), and will generate any number of sample documents from that. It's exactly what you want, except… It doesn't support all datatypes defined by the XML Schema Datatypes specification. And that's something I ran into more often before.

In fact, I tried to use xmlgen before on a couple of occasions, and each time it broke on missing support for xs:dateTime or xs:pattern restrictions. And there doesn't seem to be an aweful lot of work going into xmlgen to fix that.

Fixing XML Instance Generator

So I figured I'd fix this myself. It turned out adding support for dateTime wasn't all that hard, even though xmlgen does not really have extensions points to implement, so you're basically left with a) hacking the source code big time, or b) hacking it just a little, in order to add plugpoints and then have something else implementing that plugpoint - which is what I did.

Whoops, xs:pattern

Adding support for xs:pattern turned out to be a little tricky. If you are new to this type of restriction, then you should know that it is about restricting content to fit a certain regular expression, as illustrated below.

<simpleType name='better-us-zipcode'>
<restriction base='string'>
<pattern value='[0-9]{5}(-[0-9]{4})?'/>
</restriction>
</simpleType>

Now, if you would have the desire to generate valid data for this restriction, then you should be able to generate text from that regular expression. It turns out there are quite a few Java libraries out there capable of matching text, but there nothing at all for generating text. So I implemented my own. I blogged about it here, and it is hosted here.

Once that was done, extending xmlgen to have support for xs:pattern restrictions was easy. That means that - with just a few changes - I am now able to generate a test set for a fairly complicated schema. And I'm pretty sure that it will cover all cases, as long as I make the number of instance documents big enough.

So, now for a restriction like this:

<xsd:simpleType name = "TimeValue">
<xsd:restriction base = "xsd:string">
<xsd:pattern value = "[0-2][0-9]\:[0-5][0-9](\:[0-5][0-9])?"/>
</xsd:restriction>
</xsd:simpleType>

… it will generate instances like this:

• 07:36
• 10:16:26
• etc.

You can download the modified version of xmlgen here.

Maven archetypes are an excellent way of allowing people to create instances of a particular type of project without having them to know or worry about all of the peculiarities and details. (And the latest incarnations of the Maven Archtetype Plugin are actually way better than I realized: somewhere down the line, they introduced the ability to use Velocity templates for parameterizing the output. Nice!)

Since some of the dependencies of Preon are not in the central Maven repository yet, I decided to create an archetype of a Maven project, to make sure all of the external repositories are set correctly and preventing you from worrying about it. But then the question is: how do you keep all your archetypes organized? Generating a project from an archetype is pretty easy nowadays, but either the archetypes should be present on your user's system, or they should specify the location of a catalog file. However, that catalog file will have references to the version of your archetypes. How do you make sure these versions are correct? And how do you make sure your archetype catalog is available online, without having to manually deploy it over and over again?

I couldn't find a real policy for managing all of this, so here is what I did:

Module for Archetypes

First of all, I decided to have a module for all of my archetypes, as outlined in the picture above. Having a dedicated module that organizes all of my archetypes allows me to set specific policies for my archetypes, and it also allows me to keep my catalog file in a sensible location. (The catalog has references to all of the individual archetypes.)

Parameterized archetype-catalog.xml file

Now, I want my catalog of archetypes to have references to relevant versions of the archetypes. One way to get there is to carefully update the file by hand once a new version of the archetype is available. But that's also quite boring, and error-prone. So, what I did instead is keep the archetype-catalog.xml file in src/main/resources, and make sure the Maven Resources Plugin is kicking in to substitute variable references with relevant values.

<archetype-catalog>
  <archetypes>
    <archetype>
      <groupId>nl.flotsam.preon.archetypes</groupId>
      <artifactId>preon-simple-archetype</artifactId>
      <version>${project.version}</version>
      <repository>http://preon.flotsam.nl/archetypes</repository>
      <description>Simple Preon project, with dependencies to everything required.</description>
    </archetype>
  </archetypes>
</archetype-catalog>

Attaching catalog files

So, now we have catalog files that have correct version references, but it's still impossible to download them from a certain location. I started wondering about a solution, and first considered having them included as site resources. But then I realized I could also attach these files as new artifacts. That has a couple of interesting side effects, one of them being that they will also get uploaded to the Maven repository with every new deployment of my project. Just what I wanted. So, by adding a little configuration, I made sure the files were getting attached.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <version>1.3</version>
  <inherited>false</inherited>
  <executions>
    <execution>
      <id>attach-artifacts</id>
      <phase>package</phase>
      <goals>
        <goal>attach-artifact</goal>
      </goals>
      <configuration>
        <artifacts>
          <artifact>
            <file>target/classes/archetype-catalog.xml</file>
            <type>xml</type>
            <classifier>catalog</classifier>
          </artifact>
        </artifacts>
      </configuration>
    </execution>
  </executions>
</plugin>

Panic: SNAPSHOT versions!

So far, everything works pretty smooth. But then a problem turns up: if you attach files, and you deploy them to a snapshot repository, then the version numbers will get a suffix making the version number unique for that particular build. That works fine with the usual artifacts, but not with the Maven Archetype Plugin. Remember that - although the archetype-catalog.xml file now has versions correctly generated by the Maven Resources Plugin - it will still have version references without the unique build suffix. However, the Maven Deploy Plugin will upload versions that include the unique build suffix.

In order to solve that, I replaced the distributionManagement section in the pom.xml file of the preon-archetypes module, and made sure that - for snapshotRepository settings - the uniqueVersion is set to false. This will drop the unique suffix. However, it will also mess up your snapshot repository. In order to keep it tidy, the archetypes are deployed to a dedicated repository: the archetypes repository.

Done?

With all of this work done, creating a project from one of the archetypes is pretty easy:

mvn -DarchetypeCatalog=http://preon.flotsam.nl/archetypes/nl/flotsam/preon/archetypes/preon-archetypes/1.1-SNAPSHOT/preon-archetypes-1.1-SNAPSHOT-catalog.xml archetype:generate

That's it. After that, you will able to select the specific archetype you want to instantiate, and Maven takes care of the rest.

Sometimes a service bus is not sufficient for the job at hand. You can use EJB's and JMS queues, but that might be overkill. That's where a java callout might come to the rescue. This article will show you how to do a callout with 'complex' objects. On the bus you can pass around java objects or use them on the bus (this requires a small transformation step). I used the AquaLogic service bus version 10.2, but I think it should work any version that supports java callouts. The only difference can be the version of xmlbeans (AL 10.2 uses version 1.0.3)

At our customer we did a little proof of concept with a java callout but ultimately decided to go with another solution. The java callout however can be very useful so this article will show you how to get a head start with it. I assume that you have at least a little experience with the alsb and Maven 2. Creating a proxy service and editing a message flow should be sufficient.

The prerequisites for a java callout are simple. Any static method in a jar-file will do. You can even return your home made objects (they don't even have to be serializable). One downside is that you cannot do anything with these objects on the bus, they're just an object with an id. You can however pass these objects to another java callout and use them as your regular java objects.

The static method you want to call should be in a jar file that is loaded on the alsb (so it is NOT a library in WebLogic). It is allowed to put dependencies in this jar file, so in theory everything is possible. But idea is to keep it small and simple, otherwise go with EJB's and JMS with a library or application on WebLogic.

It is very inconvenient when you just can pass the objects around. There are however some type of objects that can be handled on the bus. Those objects  are primitives, String, BigDecimal, BigInteger, byte[], String[] and XmlObject. XmlObject is an object provided by the Apache Xml Beans framework and is basically an xml-representation of a java object. I know that converting an object to xml should happen on the bus and it would be nice when a java object can be passed to the bus, but this just isn't the fact and this is the cleanest solution I came up with.

Now that you know the theory let's get our hands dirty:

Preparation

Create a class with a static method and pacakge it in a jar file. I added the following maven dependency to an empty pom.xml:

<dependencies>
 <dependency>
 <groupId>xmlbeans</groupId>
 <artifactId>xmlbeans</artifactId>
 <version>1.0.3</version>
 <scope>compile</scope>
 </dependency>
</dependencies>

This will give you the Apache Xml Beans library at compile time. This library is excluded in the packaged jar because it lives in WebLogic.

The java class :

public class CalloutDemo {
    public static String getString(String input) {
		System.out.println("input: " + input);
		return "Buy a bicycle from " + input + "?";
    }

    public static XmlObject getCode(String input) throws XmlException {
		System.out.println("input: " + input);

		String xmlString = String.format("<root><mainCode>%s</mainCode><subCode>%s</subCode></root>", "mc", input);
		XmlObject instance = XmlObject.Factory.parse(xmlString);

		return instance;
    }

    public static Code getCode() {
		return new Code("hoofdCode", "subCode");
    }

    public static void printCode(Code code) {
		System.out.println("code: " + code);
    }
}

This class returns a main code and a sub code for an input String or just a new String based on an input String.
Apache Xml Beans is ued to return 'complex objects'. In this example I'll return a String and an XmlObject.

When you return a normal java object (ie. a domain object) and print this on the bus you will see something like this:
<con:java-content ref="jcid:5333178f:11fa2a0451c:-7fda" xmlns:con="http://www.bea.com/wli/sb/context"/>
It's just a pointer to an object. When you pass this pointer to another java callout it becomes an object again. It's not possible to do transformations or even view the java object on the bus, you can only pass it around (and probably only on the same VM instance)

No rocket science, just for the example. When you understand the code displayed above it's time to put everything on the bus. But first run the command 'mvn package' to compile the project and create a jar.

On the bus

Open the AquaLogic service bus console (I'm using the web interface by the way).
Create a session with the create button. Then create a new project in the Project Explorer (I called mine callout-test)
Click on callout-test in the project tree and create a jar resource.
Give the resource a name (ie. callout-test-jar) and upload the jar maven delivered to you (from the target folder).
When the jar is uploaded it's possible to browse the jar. You will see your brand new class with al its methods.
Now create a new Proxy Service (callout-ps), just give it a name, click last and then save.
Edit the message flow and add a pipeline pair. Add a stage at the request pipeline. Edit that stage and add an action:
Message processing, Java callout.
When you click on <method> it's possible to pick your jar and choose a method. The first method I want to show is  public static java.lang.String getString(java.lang.String). For the input expression enter xs:string('Harrie').
In real life this is a value somewhere from the bus of course.
Give the result String the name result, we can use that value later on.
Click on the icon left of Java Callout and add an Alert action. At the expression enter $result and leave the summary empty.
Make sure you save everything and activate the session.

Putting it to the test

When everything is saved it's time to test the proxy service. Find the proxy service in your project explorer and click on the debug icon (the little bug icon in the action colum). Since we created an any xml-service it's okay enter anything in the payload field as long as it's xml. Click the execute button and you'll get the response you entered in the payload field. Of course we want to see what happened internally.
Click on home and then pipeline alerts. There probably is a new alert named 'AquaLogic Service Bus Alert' with a timestamp of a few seconds ago. Click on this alert and you'll see the new String in the Description field:

Since we used System.out.println (I know it's evil, but I wanted no dependencies in my little jar) the server console shows:
input: Harrie

The next step

Now of course it's time to use the 'advanced' xml.
Re-open the flow of the proxy service, click on Stage1, click Edit, click on the getString method and change it to getCode (the one that returns XmlObject).
Enter xs:string('Arie') at the expression. Since the method is returning xml we can print the part of the message we want. Enter $result/subCode at the Alert message (or $result/subCode/text() if you just want the text without xml tags, but alerting the text you just entered isn't cool )
Save and activate and repeat the steps from 'Putting it to the test'

It's easy to predict what happens now.

Unfortunately we didn't find use for the java callout at our project, but it's pretty cool nonetheless.

Disclaimer

I used the term alsb and AquaLogic Service Bus in this article, it's probably called Oracle Service Bus. But everyone I know still talks about the AquaLogic Service bus.
I started this article a while ago and since Oracle removed edocs.bea.com I can't show you the sources where I got my information from.

Many have assumption that IT / software architecture is a thing of logic. So, there are many discussions about ESB and that we should not have those. The fact is that in large companies, we do not often do architecture - we do politecture. The politics drive the architecture. It's not the way it should be, but that's the way it is.
Let's do one more shot...why do we need ESB? This time from politecture perspective:

Preventing a mess
It allows for a greater degree of control on delivered solutions. Before you even think about creating another quick and dirty integration solution; there is that annoying ESB compliancy making sure that integration mess created with each new technology is at least visible.

Disruptive Innovation
An alternative like REST is a "disruptive innovation" and does not fit in the line of old mindsets (CORBA, RMI, EJB, SOAP, WS-*,...). ESB fits in this line. Of course, it is a matter of time before something like REST wins from ESB's.

Legacy
Quite often the cleanest solution is to change or replace the legacy in case of integration problems. So you still don't need an ESB. But the real problem is who is going to let you change that legacy that has not been changed and (barely) working in production for a long time.

Nowadays paired programming concept is getting more and more common in the IT industry because of its benefits like design quality, overcoming difficult problems and better time management etc. Similarly a new concept of Paired Testing is getting famous now among the testers who are working in an agile environment. This concept of testing is similar to paired programming approach. In this approach the person doing the testing is called the driver while the other suggests ideas or tests, pays attention and takes notes, listens, asks questions, grabs reference material, observes or reviews etc.

Paired testing can be really helpful when a new team member has joined the team and the project is in the middle of the sprints. A team member who is mature enough and have a good knowledge about the application can then be taken as a driver making the new team member as a observer so that he can learn the functionality.

Paired testing can be started with defining of a charter which include what to test, what tools to use, what testing tactics to use, what risks are involved, what bugs to look for, what documents to examine, what outputs are desired, etc. and can be ended once the reports are generated which can have the defect numbers that were logged while doing the testing.

Benefits of Paired Testing

• Idea generation, as the testers are working in a pair so one can really come up with new ideas while the other is implementing the same.
• Switching of roles, if one of the tester is tired of doing the testing so they can switch the roles and then start doing the testing again.
• Helps the tester stay on task.
• Restricts the interruption, when 2 testers are working together it limits others to interrupt them which increases the productivity.
• Defect reproducibility, bugs can be easily reproducible as while doing the exploratory testing alone a tester tends to forget the steps due to which the defect occurred.
• Good learning platform, paired testing gives a good learning platform if a new tester has joined the team.

Drawbacks of Paired Testing

• Paired testing can be a big failure if the pair doesn't share the mutual understanding.
• If both the members are not mature enough.
• If both the members are new to the application or to the paired testing concept.

Recently for our project we started using Kettle for ETL purposes. Pentaho Kettle provides UI based tool. Initially it takes quite some time to get used to Kettle UI as it becomes difficult to visualize how to orchestrate available Kettle Steps to solve a business problem. As you know how to use it, it's all about drag and drop a step and configuring it with available UI. With our experience we observed that it's pretty easy to design 90% stuff easily but rest 10% involves a lot of research and at the end involved some hacks which we never liked.

As we created Kettle transformations and jobs, we were not very sure about its testability part. After some research we found that we can use BlackBoxTests class available in Kettle distribution for test purposes. The fundamentals of it are quite simple. You pass some inputs and define the expected file and in the output you get actual output file after executing Kettle transformation. BlackBoxTests asserts if expected file matches with actual file. So for instance if you have a Sample.ktr under test, BlackBoxTests will expect Sample.expected.<txt/xml/csv> as an expected file and Sample.actual.<txt/xml/csv> as actual file to make it work. It tests all available transformations under a folder and subfolders.

By definition Kettle uses kettle.properties (available under $HOME/.kettle folder) which creates complications from testing point of view. However you should be able to test a Kettle transformation in isolation. That's why instead of using kettle.properties, we planned to use application specific property file to pass it to TransMeta class with available injectVariables() method. We were kind of successful but later found out that Kettle still uses kettle.properties even if we use a different property file.

After a lot of debugging we found out the culprit. BlackBoxTests uses EnvUtil.environmentInit() and does all the magic. It loads the kettle.properties by default and to our horror loads into java.lang.System.

We quickly got rid of using EnvUtil but found again that it's not enough to pass the properties from outside. It works for the current transformation but somehow Kettle is not able to pass these properties to embedded sub-transformations. It worked earlier just because EnvUtil.environmentInit() loads properties into java.lang.System.

Overall, though we were finally able to do the testing with BlackBoxTests in isolation with some hacks, we concluded that the Kettle code is not designed to be testable and it can be termed as legacy code in Michael Feather's language.

Using Spring JMS in our application which needs to be running on WebSphere proved to be somewhat of a challenge. And since googling provided a lot of information but just a small ‘easy to miss’ piece of text to put the pieces together, i decided to write up this blog.

Part I

The need for using Spring JMS opposed to ‘normal’ MessageDrivenBeans came out of the following requirement:

On receiving messages from the Queue we need to send these messages to another system through a Socket connection.

This means that we cannot start processing the messages from the queue until the socket connection is available. This socket connection is created on application startup in  a spring bean so it isn’t possible to guarantee that the connection is available. And also the application will be deployed inside a WebSphere cluster so one of the two servers in the cluster will never get a connection to the socket.

So we need some kind of programmatic control over the starting of the message listener.

Luckily this is possible in Spring with the start (and stop) methods of the AbstractJmsListeningContainer or the GenericMessageEndpointManager. Since we already had an implementation with MessageDrivenbeans and ActivationSpecifications to distribute the messages we decided to go with the JmsMessageEndpointManager because we could reuse our ActivationSpecifications configured in WebSphere (well we thought we could do that…..)

Reusing the configured ActivationSpecification was not possible see this topic on the springsource forum so we configured the ActivationSpecification in the spring config files together with the message endpoint handler the resource adapter and the workmanager as described in the using Spring JMS on WebSphere article.

No messages were picked up and no error messages were found to guide us in the right direction. So without more information it became some kind of trial and error exercise (which i hate, and become to hate even more because of the long /try to fix/deploy/test cycle).

So after a few tries i decided to take a closer look at the sources i had used to develop the configuration and after reading over it more than once my eyes stopped at the following piece of text hidden in the using Spring JMS on WebSphere article:

…Be aware that no other Spring JMS MessageListenerContainer types are supported…

Did this mean…..? Yes it did. By making the decision to use the message endpoint handler i had used the wrong type. So after a few minutes of rewriting the configuration to use the DefaultMessageListenerContainer (together with a messageselector to replace the activationspecification), setting the autoStartup property to false and injecting it into my connection manager bean to programmatically start the listener after the connection is created, one more long /try to fix/deploy/test cycle………..

Part II

and then it almost worked. A ClassCastException in the onMessage method stating that class A could not be cast to class A. Why isn’t it possible to cast something to itself. Ok i admit it seems a bit strange to want to do this and i didn't do it on purpose really. The onMessage method was supposed to receive a Serialized message as input parameter so i thought i needed to cast it. But it turned out to be a class A already. So i changed the method signature to take a class A as input parameter.

So we got rid of the ClassCastException this way but now we got an NoSuchMethodError (i’m not exactly sure what i prefer). This is going to look like a ClassLoader issue and if it looks like a ClassLoader issue and behaves like a ClassLoader issue it is a ClassLoader issue!

But how can that be? I have an ear file with a war file which loads all the spring bean with the contextLoaderListener and nothing else. Well almost nothing else. On the same server a different ear is deployed which also makes use of class A. therefore class A is in the core module (its called core module for a reason!). Could it be that?

Yes it can be that! The Websphere server can decide to use a ApplicationClassloader for multiple applications so on server level we need to set the classloading policy to single to prevent the ClassCastException or the NoSuchMethodError.

Hope this helps anyone out there to avoid these pitfalls.