Monthly Archives: August 2015

Agile Waterfalls

In thinking about yesterday’s post, about using Planning Poker with traditional project estimation efforts, I didn’t expect to find a book written about this subject!

At work today I got an email from some research group peddling a white paper on agile adoption.  As it happens, I’m under a contract with Pluralsight to create a new course on agile adoptions – what you need to know beforehand.  So, I downloaded the white paper and gave it a quick scan.

The article didn’t really offer any new insight for me, but it did highlight a book written by another Davis (no relation) named Barbee Davis.  Her book is entitled, “Agile Practices for Waterfall Projects.”  How timely!

I ordered the book.  Sounds great, I hope it reads great, too!

Agile Estimating and Planning

I’m continuing to read Mike Cohn’s book, “Agile Estimating and Planning.”  I’d like to get past the halfway mark by week’s end.

In Chapter 6, I liked how he pointed out that three common techniques for estimating are:

  • expert opinion
  • analogy
  • disaggregation (fancy word meaning “to break down into smaller pieces”)

Mike points out that, “Each of these techniques may be used on its own, but the techniques should be combined for best results.”  I agree!

Certainly no one will complain about using expert opinion to make estimates — who wants a non-expert to size work efforts, agile user stories, expenses or revenues?  Expert opinion is bread-and-butter estimation.

Analogy is intriguing, though, because expert opinion might implicitly make analogies when the expert makes an estimate, but very often those analogies never see the light of day.  Agile estimation does make a point of comparing and contrasting one user story with other user stories, but in a traditional project, that may not be so overtly done.

Disaggregation (breaking down into smaller pieces) is what PMI calls “progressively elaborated” estimation.  As you learn more, and as you’re able to see the details more clearly, those smaller details can be sized up, then aggregated to create a larger estimate.

In Chapter 6, Mike introduces Planning Poker and explains why it works:

  • it brings multiple expert opinion together to do the estimating
  • it creates dialog between the experts, and experts have to justify their estimates among other experts
  • like in my Lamborghini Huracan estimation example from last Friday, averaging estimates leads to better results than just selecting single estimates

All this got me thinking:  Why can’t Planning Poker be used for non-agile projects?  Why can’t traditional/waterfall projects use the same kind of estimation approach?  Why can’t a group of developers examine specific coding modules — web pages and web functions, or back-end database procedures — and use Planning Poker to estimate?

The few reasons I can think of why NOT to use Planning Poker for any type of project is that it would cause a cultural shift that some people wouldn’t like, and it would take more time to do the estimation (but, on the upside, more accurate estimates would ensue, and those estimates would be peer-reviewed in a well-functioning team).  But unless the teaming is good, Planning Poker would be a near-fruitless exercise — yet that’s true irrespective of the kind of project methodology that the project team uses.

How much is a 2015 Lamborghini Huracan?

Earlier this year I held a meeting with a group of other project managers.  I wanted to demonstrate the value of collaborative work.  To do that, I showed a PowerPoint slide with a picture of a 2015 Lamborghini Huracan (yellow, if that matters).  My question to the group was,

How much does a 2015 Lamborghini Huracan cost?

No one could use any devices to lookup the answer on the Internet.  And, no one could blurt out answers, either.  Just like in a game of Planning Poker, I asked everyone to write down their estimate for the cost of the car, and hand it to me.  There were 14 project managers in the room.

The lowest estimate was $120,000.  Two PMs, in fact, thought that’s how much the car cost.

The highest estimate was $485,000.

The average estimate was $243,429.

The MSRP of a 2015 Lamborghini Huracan (in yellow) is $237,250.  The group’s average estimate was just about $6000 off from being correct.  (BTW, I considered the group’s estimate to be quite accurate, as it fit within my own, personal, tolerance for error; I knew based upon the group’s estimate that I could not afford a new Lamborghini!).

What was much more interesting to me was that out of 14 project managers, only one PM estimated more closely to the actual MSRP than the average estimate created by the entire group (one PM estimated that the MSRP was $240,000).  Thirteen out of fourteen PMs were less accurate individually than they were collectively.

That is the power of not estimating in a vacuum!

That is why the agile project management game of Planning Poker is so effective, because group estimation is — usually, but not always — more accurate than having a single person estimate.  While agile project management understands the importance of collaboration during the estimation process, traditional/waterfall project management often relies on creating estimates with little or no collaboration.  Even when a team estimates tasks for a traditional project, they often estimate their own, respective tasks singularly rather than collectively, so the project plan is an aggregation of many estimates for many tasks, but each task is estimated by only one person.

Not collaborating when creating estimates is a potentially very big error, indeed!

Estimating in a vacuum

This week I’ve mentioned several big errors in the estimation process:

  • Poorly choosing the three points in a three-point estimate (minimum, most likely, maximum)
  • Not updating estimates frequently throughout the project as new information is revealed
  • Not having the person who will do the work provide the estimate for that work

To the list above, let me add this problem with estimating that leads to big estimation errors:  estimating in a vacuum.  This is really the same coin as the last bullet point, only I’m looking at it from a different side of the coin.

I’ve been on many, many projects where it’s my responsibility to create the estimate for the IT-side of a project (in my world, there is a business partner who estimates the resources needed from the business-side).  Generally, I have carte blanche to create what I think are the resources I need to get the job done.  Whenever possible, though, I like to bring my development team into the effort of project estimation.  I like to carve up the project into pieces that each developer knows they’ll be responsible for, so they can estimate their respective work efforts, buy-into those estimates, and be held accountable for those estimates.

Another good reason to bring the whole team into the work of project estimation is that it helps to reveal aspects of the project that I, as a project manager, may not fully know about or understand.

Agile project teams get this.  Agile project teams estimate user stories together.  Agile project teams use Planning Poker to jointly estimate the user stories on the product owner’s product backlog — and they know that in the first round, they aren’t likely going to all agree on the number of story points that the user story will require to complete.

I had a mock session of Planning Poker with several IT project managers recently to expose them to the game.  In our mock session, we estimated one user story and the difference between one person and another was huge — like, 90+ story points difference (one person used a card marked ‘100’ and others held up a card with a ‘3’ or ‘5’).  As we talked about the discrepancy, we learned why the person who held up a ‘100’ did so.  Her perspective was enlightening and revealing.  So, even though she may not have been the one to code the function we were estimating for, her perspective would have been useful to the person who would be coding that function (and who would have been playing Planning Poker right along with us if this weren’t a mock session).

Bad things happen when one person thinks they know so much that they can create estimates without anybody else’s input.  It may not always be feasible or even desirable to create estimates in a team setting like an agile team.  But, that is the best way to create estimates.  Tomorrow I’ll share an example of this.

Who’s creating the estimate?

I’m reading a book by Mike Cohn on agile estimation (not estimation for agile projects — there is a difference!).  As I’ve been thinking about estimation errors this week, I noted in my reading of Mike’s book the several times where he points to specific research that underlines that the best estimates are estimates created by the persons who will do the work.  Project managers can create an entire project using their own expertise in estimating, but that won’t be as good as working with an entire team of people who will be doing the work that the team, together, is estimating.

When I think about the potential for a mathematical error in using Statistical PERT instead of some other, better estimation technique — Monte Carlo simulation, or any other estimation technique — I’m convinced that those aren’t the errors that ought to be of concern to those who create, review, and approve estimates.

If I were a project sponsor, and my project manager gave to me a project estimate for time and cost, I would want to know how those estimates were created?  Is this product of one person’s view of the project?  Or is it the few of several people?  Or of the entire project team?  Did they use 3-point estimates?  How?  How did they choose the minimum, most likely, and maximum point-estimates?  Were those three points subjectively chosen?

Project estimation is half-art, half-science.  It’s easy to focus on the science of estimation because mathematics involves right and wrong answers, with nothing much in-between.  But it’s the art of estimation where we gloss over the potential for error.

Perhaps you’ve had a manager look at your project estimates, and then shave them, believing that there were unnecessarily generous.  Maybe they were, maybe they weren’t, but that’s just another place where estimation errors arise.  I’ve never seen a manager trim my estimates and defend their action by presenting me with hard data that my estimates were wrong.

When I think about the errors that occur in the art-part of estimating, the potential for errors within Statistical PERT appear, to me, insanely small and wholly immaterial in circumstances where it’s a good fit for an estimation problem.

 

Estimation ignorance

Yesterday I wrote about the potential for making big estimation errors by poorly choosing the three points of a three-point estimate (minimum, most likely, maximum).  Today, I’d like to write about the problem of not updating estimates as new information becomes available.

Not updating estimates is a HUGE problem with traditional, waterfall-based project management.  In a traditional project, the project is initiated with a very high-level set of constraints (a high-cost estimate of the project cost and schedule is usually in the project charter), and then the work of project planning commences.  In the project planning phase, the business requirements are written out in detail, a detail design explains how the customer’s desired business requirements will be satisfied, and a project phase-gate occurs before development work begins.

At the project phase-gate, project approvers examine the detail design and the project plan that lists all the tasks for the remaining phases of the project, who will perform those tasks, how much those tasks will cost, and how long they will take to complete.  If you use project software correctly — like Microsoft Project — you should be inputting the resources, their respective hourly costs, and the work effort they’ll expend to complete the task, and then the scheduling engine will create the project schedule for you.  At this phase-gate, the project plan is baselined and the whole project team is on the hook to complete the project at the cost and schedule listed on the project plan file.

And then, those estimates are locked-in, written in stone, and everyone is held accountable to them.

Yes, traditional project plans have change control systems that allow changes to the project plan, but the whole change control system is designed to minimize or eliminate change, and often there are unpleasant political consequences for any team who uses this system to revise the project plan.

That’s a mistake.  That’s a HUGE mistake.

It is well known and well researched that no product owner can write business requirements so well that they never need to be revised or refined.  And no project team can create a detail design and fully identify every aspect of the solution.  In real life, new information arises, and in a traditional project approach, that new information is never utilized to refine/revise the project estimates that were made at the last phase-gate.

This is a worse problem than poorly choosing the three points of a three-point estimate, because if you don’t create a way of leveraging new information — without penalties, without political fall-out — you create a terrible organizational culture that hides its mistakes, makes bloated estimates to cover for possible mistakes (and when that happens, Parkinson’s Law ensure that the bloated estimates always appear right-on-the-money), and you don’t learn anything about how to make better estimates in the future.

Poorly choosing minimum, most likely, and maximum point-estimates is bad, and much worse than a 2% or 3% mathematical error in using a technique like Statistical PERT.  Willfully ignoring new information and failing to revise earlier estimates, however, is an order-of-magnitude worse problem.

The bigger problems with estimating anything

There are a couple of bigger problems to wrestle with than the fact that Statistical PERT relies upon the normal curve instead of any other, better-fitting probability distribution.  Let’s talk about a few of them.

The first one that comes to mind is the error in choosing the values for the minimum, most likely, and maximum outcomes for a bell-shaped uncertainty.

Anytime someone makes a three-point estimate, you have to find out — from where did they obtain their three points?  Are these obtained from historical records?  Are they industry benchmarks?  Are they by looking at other, equivalent projects  (or teams, or whatever)?  Was parametric estimation in-use?  Or — more likely — were the three points chosen from the estimator’s own knowledge and expertise in what was being estimated?

In my day-to-day life as an IT project manager, project schedules are almost exclusively built using the project manager’s own knowledge and expertise, along with the expertise of others, such as the development team, other managers, and perhaps someone from the Project Management Office.

Whenever I choose a three-point estimate, I’m at risk of poorly choosing my point-estimates.  If I look at a programming effort and I gauge it to take 24-40-60 hours, but I’ve greatly underestimated by wrongly presuming that I would get a seasoned developer to code that module — instead, I’m getting a new-at-the-job, junior developer — then SPERT’s 1% or 2% error is truly immaterial if I should have estimated 40-80-120 hours to begin with.  Statistical estimation models like SPERT are completely useless with bad input data.

Estimates are only as good as the estimator behind the estimate.  Whether we’re working with deterministic, single-point estimates, three-point estimates, or estimate ranges, the #1 concern ought to be in finding out how the estimates were chosen, and if a three-point estimate was used (in PERT, SPERT, a Monte Carlo simulation, or something else), the key question is, “How did you choose the basis for your estimate?”

If you’re really 88% sure (not 90%), does it matter?

I’m not a statistician and I don’t have a math degree; I have two business grad degrees, one in project management.  A statistician or math major will look at Statistical PERT and scowl.  “It uses a normal curve, not the beta distribution!  You’re using the wrong distribution for that uncertainty!”  But as a project manager, I’m intrigued with Statistical PERT’s simplicity and ease-of-use.

I’ve been running a lot of Monte Carlo simulations lately, comparing Statistical PERT with Monte Carlo simulation using the PERT (a type of beta) distribution, to find how much error exists with Statistical PERT.  This analysis makes several assumptions, among them, that the beta distribution is the better distribution to use for a given uncertainty.

In my analysis, I’ve come to realize that whatever errors Statistical PERT introduces to the estimation effort, they are, in most circumstances irrelevant.  The error is sometimes a difference of less than 1%, and often less than 2%.  I occasionally see differences arise in the 3% – 5% range, too, but that usually is with very skewed probability curves and looking at estimates at the high 90th percentiles.

I think Statistical PERT’s greatest value is how it can align expectations between project manager, project team, project sponsor, and other stakeholder groups.

If a team uses Statistical PERT and offers the project sponsor a comfortable estimate that has a SPERT-calculated 90% confidence level — but the Monte Carlo simulation of the same uncertainty shows that the estimate really has a 92% probability of being equal to or greater than the actual result — does it matter?  Maybe in a few situations it will, but I think in many or most situations, it’s an immaterial difference.  Everyone senses that the estimate has a high chance of working, of being successful.  If the project sponsor is willing to accept the high cost (in terms of money and/or time), he will be rewarded with a high confidence that the project team will finish on-time, on-budget.  Conversely, if a project team is forced into a low-ball estimate, and they calculate that the estimate has only a 28% chance of succeeding when the Monte Carlo simulation says that it’s really got a 30% chance, does that extra 2% make anyone feel any better about their chance of success?  No!

When a team tells a project sponsor that his top-down estimate has 28% chance of success, the question that the project sponsor ought to raise is obvious:  “Why do you have such low confidence in my estimate?”  Now, an authentic conversation can ensue, and now the project team can explain their three-point estimate, and the subjective opinion they used about how likely the most likely outcome really is.  Maybe the team will still be forced into accepting the sponsor’s own estimate, but Statistical PERT has shone light onto that decision and the high risk of failure that comes with using the project sponsor’s estimate.

Statistical PERT isn’t designed to be a mathematically bullet-proof estimation technique.  It’s designed to be a quick, easy, no-cost, reliable and reasonably accurate estimation technique that helps all project stakeholders communicate.  Statistical PERT helps everyone identify their chance of success — and failure.

How hot is hot tea?

I love drinking hot tea in the afternoons.  I’m sipping hot ginger tea right now as I type this blog post.  I enjoy hot tea that is hot enough I have to slowly sip it or else I’ll burn my mouth, but not so hot that I scald my lips as I draw the first tiny bit of tea into my mouth.  Guess how hot that is?

It’s about 180 degrees Fahrenheit.

I know that because I got a digital kitchen thermometer a little while ago, and instead of just boiling water at home, then waiting for the piping hot brew to cool enough to the point I can start sipping, I thought I’d take the temperature of perfectly hot water, then, next time, I’d insert the thermometer probe into the tea kettle and heat the kettle up to that perfect temperature, rather than waiting for the tea pot to whistle because it’s boiling inside.

My new kitchen thermometer is very precise; it measures in one-tenth of degree increments.  I know my kitchen thermometer is accurate, too, because when I inserted the probe into a teapot full of boiling water, the temperature read 212 degrees, plus or minus a few tenths of a degree.

Suppose my thermometer wasn’t quite so accurate as it is.  Suppose it read two degrees off, so when I put it into boiling water, it read either 210 degrees or 214 degrees, but never 212 degrees.

Would that matter?

It depends.  It depends on the context in which I’m using the thermometer.  In the context of bringing hot tea to a perfect temperature, being off +/- two degrees is immaterial.  I think I’d find hot tea heated to 182 or 178 degrees to be perfectly acceptable.  In fact, I’ve found that I can be happy with hot tea served between 175 and 185 degrees.  So, having a thermometer that reads one, two, three, four or even five degrees off isn’t a problem for me, and I wouldn’t seek to replace it.  It’s good enough for my intended purpose of heating hot water to a point where I don’t have to wait before I start sipping freshly brewed tea.

But is my thermometer accurate?

You might say it’s not — it reads two degrees off from what it should.  But in another sense, is IS accurate because its errant readings are within my tolerance for error.  My tolerance for error determines accuracy and inaccuracy.

If something fits my tolerance for error, it’s accurate.  If it doesn’t, it’s inaccurate.  My thermometer gives me great precision, but I don’t really care about the level of precision it offers me.  I want accuracy, and my tolerance for error says that as long as I don’t burn my mouth when I sip my tea, and I also don’t feel like I’m sipping tepid tea, then my thermometer is doing it’s job — it’s accurate for its intended purpose.

Palisade’s @Risk Excel Add-in

I love Monte Carlo simulation.  It’s fascinating to me, and I like that it’s half-art, half-science.  The science part involve statistics, probabilities and probability distributions.  The art part involves selecting the right distribution to characterize an uncertainty, then configuring that distribution in the model so it fairly represents the uncertain nature of the uncertainty itself.

The Monte Carlo simulation program I’m most familiar with is an Excel add-in program called @Risk, part of the DecisionTools Suite by Palisade.  It’s a pricey piece of software — nearly $1300 for a stand-alone, standard license that covers one year of support and upgrades, after which it will cost you a few hundred dollars to maintain ongoing support.

In future blog posts, I will be using Palisade’s @Risk program to compare the strengths and weaknesses of Statistical PERT.  As I continue estimating and experimenting with Statistical PERT, I’m pleased that SPERT creates probabilities that are close to a Monte Carlo simulation.

But is Statistical PERT good enough to use instead of other estimation methods, including Monte Carlo simulation?  Before answering that question, we have to wrestle with the difference between accuracy and precision.