PoC: Doing It Right and Wrong

In the last decade I had a chance to participate in several large-scale projects as a developer and engineering manager. In large projects with significant amount of uncertainty in both business direction and engineering approach one clearly needs an effective means for uncertainty reduction. While people naturally hunch where the trouble is (sometimes it is obvious, for instance we lack experience in some technology) and realize that they need to somehow make the unclear things clear, they surprisingly often fail to eventually arrive at lower uncertainty levels than started. One particularly interesting approach is the arbitrarily designed attempt to run into uncertain terrain mistakenly called "proof of concept", or PoC for short.

The Anti-pattern

It goes usually like this. The team is confronted with some issue, for example they have to decide which NoSQL database to use, but they have never used any NoSQL database before. At the same time someone already decided that a NoSQL database shall be used to say hold business domain records. So what does the team do? They accept the decision and get themselves acquainted with the tech: Install it, configure it, learn the API etc. It builds confidence in being able to deploy and integrate that particular solution. This is nice and useful but has no concept nor proof in it.

At this moment you sure wonder how this has no concept and proof - they have tried to make the stuff run, learned how to do it an proved that it runs after all! Well, you see, proving that the established product actually works and that you have been able to run it does not reduce uncertainty you were confronted with. There is no information gain and the finding is obvious - a product which is supposed to work and targeted at development teams will work and can be learned by development teams. The actual question whether NoSQL database and a particular implementation of it is what you need and has characteristics actually required to solve the business problem at hand is not answered by demonstrating that you can run and deploy a database. So you proved an obvious thing, which is of no use, while leaving the actual question - whether the technology is the right one - unanswered.

Oftentimes people do approach the uncertainty in a more useful way but the corresponding experiments and their outcomes are not recorded. So months later nobody really knows what actually have been done, what have been learned and as consequence can not understand the decisions made. However, such decisions define constraints within the project, such as which technology has to be used. In this case the decision implies that a particular NoSQL database shall be integrated. If the choice of the technology is not appropriate, and this happens and can not be avoided, the discussions about alternatives have to go in the context of that choice. Typically, issues with the early decisions pop up later in the project. The problem is that without clear decision record, which explains what has been explored and why the choice has been made, we end up only with the fact that a particular technology shall be used. No further information. This makes discussions very difficult, especially if the responsible decision makers refer to the PoC which "was a success".

This way of dealing with uncertainty is not helpful and in my experience a factor for compounding friction and frustration in large projects.

PoC as Uncertainty Reduction Tool

The key point of PoC is to reduce uncertainty for the decisions at hand. In order to do so, you first need to understand which decisions you need to make. Proper decision analysis is the key here. In particular, you need to evaluate the information need and estimate the value of the respective information gain. Next, you define the PoC as a means to acquire information, necessary to reduce the uncertainty for the decision. You of course must keep costs of PoC lower than the value of information it provides. Otherwise it is of no use. The PoC therefore must support a particular decision. Sadly the methodical foundations for decision analysis are usually not present in the IT teams.