Risk Evaluation Math

Introduction

Figure 1: Project Planning Complexity.

I am always looking for ways to evaluate the risk of the projects that I am undertaking. Project planning can be very complicated (see Figure 1) and things do not always go well. It is important to understand the potential for problems on a project. I was reading an article in the Journal of Light Construction (JLC, July-2014, "Four Common Delusions") that defined a business metric called "Disaster Potential" or DP that I thought was interesting and had some merit for discussion here.

Background

Program Management Metrics

There are an endless variety of metrics used in project management. I will list here a few of the standard ones that I regularly use:

While these metrics are good for tracking your performance while executing a project, they are not useful during the conceptual or design stage of a project. It is during the conceptual stage where most projects commit to cost and schedule risk. Figure 2 shows a commonly used chart that qualitatively depicts the relationship between the committed cost, incurred cost, and the cost of change at different points in the lifecycle of a project.

Figure 2: Qualitative Relationship Between Committed and Incurred Costs.

Unfortunately, most costs are committed to unknowingly because people fail to accurately estimate costs for two important items:

Knowable Unknowables
Every program incurs some level of unexpected delay and expense that are actually predictable and I budget for them. I have tracked my knowable unknowable for years and I have a very good track record of predicting these expenses over a long period of time (i.e. year).
Unknowable Unknowables
These are the tough ones because they are completely unpredictable. The most common sources of unknowable unknowables are:
- Ambiguous requirements
  I have worked on requirements my entire career. There is not such thing as a completely specified project. There is always something missing and people will fill the vacuum with what they think the requirement is. People often do not realize they are making unverified assumptions. This explains the old engineer's adage that "Assumption is the mother of all screwups".
- Incorrect understanding of the requirements
  Misunderstandings are usually associated with a definition disagreements. People often have different or even wrong understanding of definitions. I tell my staff that the most dangerous knowledge is information that we are certain of and is not true. People often delude themselves. Feynman used to say "The first principle is that you must not fool yourself and you are the easiest person to fool."
- Accidents and Acts of God
  The unexpected happens:
  - I had an excellent vendor whose plant was destroyed by a tornado.
  - A critical field issue may siphon away critical resources from your projects.
  - Sadly, I have had a critical co-worker killed in car accident.
  - Injuries on the job are more common than we'd like to think and these too can set projects and plans back some way. It's possible that workers who have been involved in an accident may want to see a lawyer about following it up with a workers' comp appeals process in order to get the compensation that they deserve.

What is the Disaster Potential?

The disaster potential provides a Figure of Merit (FOM) of the risk associated with a project assuming some level of unpredictable variation. The model can be interpreted as providing an estimate for how much additional cost a project may require for completion.

Disaster Potential Calculation

I do not know of a generally accepted, standard metric for program risk. I do see a number of non-standard metrics that individuals use to help them assess the risk of a project, one of which is the focus of this post.

The weak point of all risk assessment approaches is that you are required to estimate the unknowables of your project. Since the unknowables are probabilistic in nature, what you are trying to determine is the magnitude of a potential cost overrun. This is useful is determining how likely you are to make money on a project. I know of a number of businesses that went bankrupt because they took on a big project that went poorly. They ran out of cash before the projects were completed. This is a key reason why many of them use services from the likes of CreditRiskMonitor to help them assess business risks before committing to something and losing out in the long run.

Equation 1 shows the DP formula as presented in the JLC article.

Eq. 1

$\displaystyle DP=\frac{{{N}_{FTE}}\cdot {{N}_{Sub}}\cdot T}{{{K}_{Exp}}\cdot {{K}_{Sim}}}$

where

N_FTE number of full-time equivalent staff members working on the project.
N_Sub is the number of contractors/partners involved in the project. If only my group is involved in the project, then N_Sub = 1.
T is the project duration
K_Sim is an similarity measure to other work graded on a scale from 1 to 5, where 1 means that it has little similarity to current work and a 5 means it is very similar to your current work. This number association is arbitrary and can be changed to reflect the needs of a project.
K_Eff is an efficiency measure. Some projects involve communicating across timezones, working at night, or outdoors in winter. You become less efficient when you work in these situations than if you work in a controlled environment like a factory.

The minimum value DP value you can have is when:

Only one group is working on the project (N_Sub = 1).
Your team is working at maximum efficiency (K_Eff = 100%)
The work is identical to previous projects your team has performed (K_Sim = 5)

In this case, $DP_{minimum}=\frac{{{N}_{FTE}}\cdot T}{5}$ . I would interpret this to mean that your cost risk is 20% of the projects estimated overall cost. This number seems to conform to my own experience with home remodeling projects. All of these projects incur some sort of unpredictable variation. In the best cases, I would estimate that they typically run ~20% over budget. In the worst case, I have had remodeling projects cost twice what I expected.

This model makes a number of assumptions:

Your program costs are dominated by labor
In my department, labor constitutes 80% of my expenses. The model of Equations 1 can easily be extended to include material cost, but I will not pursue that enhancement here.
You can quantify your experience level.
This is where significant uncertainty enters the calculation. It is very difficult to estimate what you do not know.
No performance penalties are included.
Some contracts include performance penalties. There are also intangible performance penalties like lost goodwill. The model can be extended to include these penalties, but they are not in the base model.
The risk increases linearly with the number of contractors involved.
This may be a bit harsh, but adding contractors to a job definitely increases the risk of a project.

Example

Rather than use an example from my work, I will use a recent situation that a close relative encountered while having a geothermal heating system installed at her home. Evaluating the risk of this project provides a nice illustration of the role of risk analysis and its pitfalls.

Project Description

The subject of geothermal heating systems is a large one, but I will provide some basic background here - I am grossly oversimplifying what is going on here.

pipes are buried in the ground that carry water, which is the medium for carrying the warmth of the ground to a heat exchanger.
A heat exchanger is mounted on your home that transfers the heat from the water into your home.
There are pumps that move the water from the ground into your homes.

Project Characterization

My relative gave the project the following characterization:

N_FTE = 5: She was told by her contractor that five workers would be involved in the installation.
N_Sub = 1: She was told by her contractor that no subcontractors would be involved in the installation.
T = 1 week: She was told by her contractor that it would take 1 week to install the system.
K_Exp = 5: She did a great job of asking about their experience and they have been installing heating systems for 20 years. Unfortunately, the particular geothermal system she chose was new to them and they actually had no experience with her system. It turns out that this number should have been 1.
K_Eff = 1: She had them perform the task before winter arrived and they had excellent access to her site. The project could not have had better work conditions.

Given the information she had, I compute the disaster potential for this project as $DP=\frac{5\ \text{man}\cdot 1\cdot 1\ \text{week}}{5\cdot 1}= 1\ \text{man-week}$ - she had no reasons to suspect a major cost overrun. Note that there was significant material cost involved in this system and the DP model ignores it.

With the benefit of hindsight and knowing that the contractor had no experience with the geothermal system she chose, I would estimate their disaster potential to be $DP=\frac{5\ \text{man}\cdot 1\cdot 1\ \text{week}}{1\cdot 1}=5\ \text{man-week}$ - a doubling of the project cost. Unfortunately, this number is probably about right.

Project Outcome

The initial install went as predicted and she had to wait for winter to come to test the system out. When winter came, the heat was not adequate and sometimes there was no heat at all. She called the contractor, who tried to repair the system multiple times, made many changes, and the system never did work properly. The contractor refused to pull it out and re-install it. She is now taking court action against the contractor. Her total cost will end up being far more than the cost of the initial installation. The final numbers are not in.

Conclusion

This is an interesting metric that I will be experimenting with on my projects at home and at work.

M	T	W	T	F	S	S
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31