To compare Statistical PERT’s use of the mode with a Monte Carlo simulation that characterizes high confidence in the most likely outcome, we have to depart from using the PERT distribution — a special form of the beta distribution — and use a different beta probability curve to better model high confidence in the mode.
There would be no point to even considering the differences between using the mean and mode if the bell-shaped curve was symmetrical; in that event, the mode and mean would be the same value, and choosing between them would be a moot point.
Instead, we have to consider a skewed, bell-shaped curve with high expectation in the most likely outcome, but a curve which is asymmetrical. Since in projects most outcomes that aren’t close to the most likely outcome exceed the mode, we’ll consider a bell-shaped curve that is skewed to the right. The three-point estimate we’ll consider is 50-100-200.
When I ran a Monte Carlo simulation model using a PERT distribution, it led to a mean of 108, a mode of almost 101, and a standard deviation of 27.64. That standard deviation is not showing the vertical rise that we want, where more of the area is closer to the mode (and mean).
Instead, let’s use a beta distribution where the alpha1 value is 4, and the alpha2 value is 7 (I’m using the RiskBetaGeneral function in Palisade’s @Risk Excel add-in). The resulting simulation creates a mode of 100, a mean of nearly 105, and a standard deviation of 21. We want a standard deviation that’s lower than the PERT distribution of 27.64 because there’s high confidence in the most likely outcome, so the standard deviation should be smaller.
Using the SPERT standard deviation formula and the SPERT-7 Rule, the formula is (Max – Min) * 14% for high confidence in the most likely outcome. Using my three-point estimate of 50-100-200, the SPERT standard deviation is (200 – 50) * 14% = 21. This matches the beta distribution using (4, 7) for the alpha values. Here’s what the curve looks like with high certainty around the mode, but which still extends to the left to 50 and to the right towards an improbable 200.
Assuming that the Monte Carlo simulation best represents the nature of the uncertainty, how well does SPERT estimation align with a Monte Carlo simulation just using the simplistic SPERT formulas that rely on the normal distribution?
The first column above are different points on the beta curve. The second column is the Monte Carlo result — how much area under the curve and to the left of the X point. The third and and fifth columns show SPERT-calculated values using the mode and PERT mean, respectively. Note that in SPERT estimation, the PERT mean is estimated using the PERT formula — not through simulation. So, in the SPERT analysis above, the estimated PERT mean was 108, not the nearly 105 it was during the simulation. The orange columns show the probability gap between the Monte Carlo simulation and the SPERT calculations.
Overall, SPERT calculations grew increasingly more accurate when dealing with probabilities of 90% and more. At its worst, SPERT calculations using the mode were 6.5% off from the Monte Carlo simulation (when X was 110). Using the estimated PERT mean, though, the error at that same point was 8.6%. The error using the PERT mean was, at its worst, 9.5% off, whereas using the mode, the worst error was 6.5%. It’s noteworthy to point out that SPERT estimation using the estimated PERT mean was more accurate than SPERT estimation using the mode at other points. If I take weighted averages of the errors for both SPERT using the mode and SPERT using the mean, SPERT using the mode is slightly more favorable over a sample of values across the entire curve than estimation using SPERT and the estimated PERT mean.
The take-away from this is that, under a condition of high confidence in the most likely outcome on a skewed, bell-shaped uncertainty, SPERT estimation using the mode will result in estimation errors that are less overall than SPERT estimation using the mean, and SPERT using the mode’s errors won’t ever be as bad as SPERT estimation using the mean.
Using the normal distribution curve to model a skewed probability isn’t ideal, but it can be quick and still reasonably accurate — accurate within an estimator’s tolerance for error. That won’t always be the case, though, and sometimes SPERT just isn’t a good technique to use to estimate uncertainties, even if they’re bell-shaped.
Recognizing this, I have an exciting announcement about Statistical PERT estimation on Monday, September 14!