Previous Contents Next
RISK AND UNCERTAINTY

 

ANNEX

 Annex 4

 

INTRODUCTION
1
This annex provides further guidance in each of the following areas:
 
Risk management;
Transferring risk;
Optimism bias;
Monte Carlo analysis;
Irreversibility; and
The cost of variability in outcomes.

RISK MANAGEMENT
2
Risk management is a structured approach to identifying, assessing and controlling risks that emerge during the course of the policy, programme or project lifecycle. Its purpose is to support better decision-making through understanding the risks inherent in a proposal and their likely impact.
3
Effective risk management helps the achievement of wider aims, such as: effective change management; the efficient use of resources; better project management; minimising waste and fraud; and supporting innovation.

Organisation level risk management
4
Public sector organisations should foster a pragmatic approach to risk management at all levels.1 This involves:
 
Establishing a risk management framework, within which risks are identified and managed;
Senior management support, ownership and leadership of risk management policies;
Clear communication of organisational risk management policies to all staff; and
Fully embedding risk management into business processes and ensuring it applies consistently.
5
These actions should help establish an organisational culture that supports well thought out risk taking and innovation.

back to top

Policy, programme and project level risk management
6
At the level of individual policies, programmes and policies, risk management strategies should be adopted in a way that is appropriate to their scale.
7
A risk register or risk log is a useful tool to identify, quantify and value the extent of risk and uncertainty relating to a proposal. A risk register / log can be used to identify the bearer of each risk and uncertainty associated with the project being appraised, provide an assessment of the likelihood of each risk occurring, and estimate its impact on project outcomes. Box 4.1 provides more detail.
 
BOX 4.1: RISK REGISTER (RISK LOG)
 

PURPOSE

A risk register lists all the identified risks and the results of their analysis and evaluation. Information on the status of the risk is also included. The risk register should be continuously updated and reviewed throughout the course of a project.

CONTENT

A risk register is best presented as a table for ease of reference and should contain the following information:

Risk number (unique within register);
Risk type;
Author (who raised it);
Date identified;
Date last updated;
Description;
Likelihood;
Interdependencies with other sources of risk;
Expected impact;
Bearer of risk;
Countermeasures; and
Risk status and risk action status.

FURTHER INFORMATION

For an example of a risk log and further information on the identification of risks and successful project and risk management refer to the OGC.2
back to top

Risk Mitigation
8
There are a number of approaches appraisers might take to mitigate the impact of the identified risks. These are outlined in Box 4.2.
 
BOX 4.2 OPTIONS TO HELP MANAGE RISK
 
 
Active risk management – Effective management of risks involves:
  
   
identifying possible risks in advance and putting mechanisms in place to minimise the likelihood of their materialising with adverse effects;
  
   
having processes in place to monitor risks, and access to reliable, up-to-date information about risks;
  
   
the right balance of control in place to mitigate the adverse consequences of the risks, if they should materialise; and
  
   
decision-making processes supported by a framework of risk analysis and evaluation.
  
 
Early consultation – Experience suggests that costs tend to increase as more requirements are identified. Early consultation will help to identify what those needs are and how they may be addressed.
  
 
Avoidance of irreversible decisions – Where lead options involve irreversibility, a full assessment of costs should include the possibility of delay, allowing more time for investigation of alternative ways to achieve the objectives.
  
 
Pilot Studies – Acquiring more information about risks affecting a project through pilots allows steps to be taken to mitigate either the adverse consequences of bad outcomes, or increase the benefits of good outcomes.
  
 
Design Flexibility – Where future demand and relative prices are uncertain, it may be worth choosing a flexible design adaptable to future changes, rather than a design suited to only one particular outcome. For example, different types of fuel can be used to fire a dual fired boiler, depending on future relative prices of alternative fuels. Breaking a project into stages, with successive review points at which the project could be stopped or changed, can also increase flexibility.
  
 
Precautionary Principle – Precautionary action can be taken to mitigate a perceived risk. The precautionary principle states that because some outcomes are so bad, even though they may be very unlikely, precautionary action is justified. In cases where such risks have been identified, they should be drawn to the attention of senior management and expert advice sought.
  
 
Procurement/contractual – risk can be contractually transferred to other parties and maintained through good contractual relationships, both formal and informal. Insurance is the most obvious example of risk transfer. The main text of this annex provides further information about the types of risk that can, and often are, transferred.
  
 
Making less use of leading edge technology – If complex technology is involved, alternative, simpler methods should also be considered, especially if these reduce risk considerably whilst providing many of the benefits of the option involving leading edge technology.
  
 
Reinstate, or develop different options – Following the risk analysis, the appraiser may want to reinstate or options, or develop alternative ones that are either less inherently risky or deal with the risks more efficiently.
  
 
Abandon proposal – Finally, the proposal may be so risky that, whatever option is considered, it has to be abandoned.
  

9

By reducing risks and uncertainty in these ways, the expected costs of a proposal are lowered or the expected benefits increased.
10
Additional guidance on risk management can be obtained from Risk Analysis and Management for Projects (RAMP), the Office of Government Commerce (OGC), the National Audit Office (NAO), HM Treasury, and the Cabinet Office.3

back to top


TRANSFERRING RISK
11
Box 4.3 describes the general types of risk a project manager is likely to encounter.4
12
Risk assessment will inform an overall view of the viability of an option, i.e. whether its risk-adjusted benefits exceed its risk-adjusted costs, or whether (in the case of uncertainty) the costs of a possible adverse outcome are so great that precautionary action needs to be introduced to obtain a cost-effective solution.
 
BOX 4.3: GENERAL TYPES OF RISK
 
Availability risk
The risk that the quantum of the service provided is less than that required under a contract.
Business risk
The risk that an organisation cannot meet its business imperatives.
Construction risk
The risk that the construction of physical assets is not completed on time, to budget and to specification.
Decant risk
The risk arising in accommodation projects relating to the need to decant staff/ clients from one site to another.
Demand risk
The risk that demand for a service does not match the levels planned, projected or assumed. As the demand for a service may be partially controllable by the public body concerned, the risk to the public sector may be less than that perceived by the private sector.
Design risk
The risk that design cannot deliver the services at the required performance or quality standards.
Economic risk
Where the project outcomes are sensitive to economic influences. For example, where actual inflation differs from assumed inflation rates.
Environment risk
Where the nature of the project has a major impact on its adjacent area and there is a strong likelihood of objection from the general public.
Funding risk
Where project delays or changes in scope occur as a result of the availability of funding.
Legislative risk
The risk that changes in legislation increase costs. This can be sub-divided into general risks such as changes in corporate tax rates and specific ones which may affect a particular project.
Maintenance risk
The risk that the costs of keeping the assets in good condition vary from budget.
Occupancy risk
The risk that a property will remain untenanted – a form of demand risk.
Operational risk
The risk that operating costs vary from budget, that performance standards slip or that service cannot be provided.
Planning risk
The risk that the implementation of a project fails to adhere to the terms of planning permission or that detailed planning cannot be obtained, or if obtained, can only be implemented at costs greater than in the original budget.
Policy risk
The risk of changes of policy direction not involving legislation.
Procurement risk
Where a contractor is engaged, risk can arise from the contract between the two parties, the capabilities of the contractor, and when a dispute occurs.
Project intelligence risk
Where the quality of initial project intelligence (eg preliminary site investigation) is likely to impact on the likelihood of unforeseen problems occurring.
Reputational Risk.
The risk that there, will be an undermining of customer/ media perception of the organisations ability to fulfil its business requirements e.g. adverse publicity concerning an operational problem.
Residual Value risk
The risk relating to the uncertainty of the value of physical assets at the end of the contract.
Technology risk
The risk that changes in technology result in services being provided using non-optimal technology.
Volume risk
The risk that actual usage of the service varies from the level forecast.

13
When faced with significant risks, a public body should consider transferring part or all of it to the private sector. The governing principle is that risk should be allocated to whichever party from the public or private sector is best placed to manage it. The optimal allocation of risk, rather than maximising risk transfer, is the objective, and is vital to ensuring that the best solution is found. Accordingly, the degree to which risk is transferred depends upon the specific proposal being appraised.
14
Successful negotiation of risk transfer requires a clear understanding by the procuring authority of the risks presented by a proposal, the broad impact that these risks may have on the suppliers’ incentives and financing costs, and the limits to risk transfer which might still be considered for value for money.
15
Where the private sector has clear ownership, responsibility and control, it should be encouraged to take all of those risks it can manage more effectively than the procuring authority. If the public body seeks to reserve many of the responsibilities and controls that go hand-in-hand with service delivery and yet still seek to transfer significant risk, there is a danger that the private sector will increase its prices.
16
Appropriate transfer of risk generates incentives for the private sector to supply timely cost effective and more innovative solutions. As a general rule, PFI schemes should transfer risks to the private sector when the supplier is better able to influence the outcome than the procuring authority. Risks to be considered include:
 
Design and construction risk: to cost and/ or time;
Technology and obsolescence risks;
Commissioning and operating risks, including maintenance;
Regulation and similar risks (including taxation, planning permission);
Demand (or volume/ usage) risks;
Residual value risk; and
Project financing risk.

17
A risk allocation table can be a useful tool to identify the bearer of each risk relevant to a proposal. An example of this is set out in Box 4.4.
 
BOX 4.4: EXAMPLE OF RISK ALLOCATION TABLE
 
Risk Scale Bearer Key Issues
Purchaser
Provider
Obsolescence Low   Assets require low levels of technology
Demand Risk Med   …
Design risk High   …
Residual Value Low   …
3rd party revenues Low   …
Regulatory change High   …
etc. … … … …

back to top

OPTIMISM BIAS
18
Optimism bias is the demonstrated systematic tendency for appraisers to be over-optimistic about key project parameters. It must be accounted for explicitly in all appraisals, and can arise in relation to:
 
Capital costs;
Works duration;
Operating costs; and
Under delivery of benefits.

Capital costs

19
The two main causes of optimism bias in estimates of capital costs are:
 
poor definition of the scope and objectives of projects in the business case, due to poor identification of stakeholder requirements, resulting in the omission of costs during project costing; and
poor management of projects during implementation, so that schedules are not adhered to and risks are not mitigated.

20
Appraisers should adjust for optimism bias in the estimates of capital costs in the following way:
 
Estimate the capital costs of each option;
Apply adjustments to these estimates, based on the best available empirical evidence relevant to the stage of the appraisal; and
Subsequently, reduce these adjustments according to the extent of confidence in the capital costs' estimates, the extent of management of generic risks, and the extent of work undertaken to identify and mitigate project specific risks.

21
Departments or agencies may be able to provide the best empirical evidence to support adjustments for optimism. Alternatively, and if applicable, they may be taken from the Green Book homepage5, which provides the recommended adjustments to be made at the outline business case stage for buildings, civil engineering, equipment and development, and outsourcing projects.
22
If no obvious empirical evidence is available, this may indicate that the project is unique or unusual, in which case optimism bias is likely to be high. In these cases, adjustments should be based on the nearest equivalent project type, and adjusted up or down, depending on how inherently risky the project is compared to its nearest equivalent type.
23
If a department chooses to apply its own adjustments, these must be prudent. Where possible, the cost estimates, and the adjustments for optimism bias should be reviewed externally (using Gateway reviews for large projects, or internal audit reviews of smaller projects).

back to top


Works duration
24
The same approach should be taken with estimating the length of time it will take to complete the capital works. In summary:
 
Estimate the time taken to complete the capital works;
Apply adjustments to these estimates, based on the best available empirical evidence relevant to the stage of the appraisal;
Subsequently, reduce these adjustments according to the extent of confidence in the capital costs estimates, the extent of management of generic risks, and the extent of work undertaken to identify and mitigate project specific risks; and
The estimates of works’ duration, and the adjustments for optimism, should ideally be reviewed independently.

25
The application of optimism bias adjustments to works’ duration should be reflected in a delay in the receipt of benefits. This will be shown in the net present value calculations. The appraisal period may need to be extended to reflect the expected delay in benefits’ stream, but different periods should not usually be set for different options.

Operating costs and benefits
26
Analysis should also be undertaken on potential benefits’ shortfalls and increases in operating costs. If there is no evidence to support adjustments to operating costs or benefits’ shortfalls, appraisers should use sensitivity analysis. This should help to answer key questions such as:
 
By how much can we allow benefits to fall short of expectations, if the proposal is to remain worthwhile? How likely is this?
How much can operating costs increase, if the proposal is to remain worthwhile? How likely is this to happen?
What will be the impact on benefits if operating costs are constrained?

Preventing optimism bias
27
To minimise the level of optimism bias in appraisal, best practice6 suggests that the following actions should be taken:
 
Project managers, suitably competent and experienced for the role, should be identified;
Project sponsor roles should be clearly defined;
Recognised project management structures should be in place;
Performance management systems should be set up; and
For large or complex projects:
 
Simpler alternatives should be developed wherever possible;
 
Consideration should be given to breaking down large, ambitious projects into smaller ones with more easily defined and achievable goals; and
 
Knowledge transfer processes should be set up, so that changes in individual personnel do not disrupt the smooth implementation of a project.

back to top


MONTE CARLO ANALYSIS
28
Monte Carlo analysis allows an assessment of the consequences of simultaneous uncertainty about key inputs, and can take account of correlations between these inputs. It involves replacing single entries with probability distributions of possible values for key inputs. Typically, the choice of probabilistic inputs will be based on prior sensitivity testing. The calculation is then repeated a large number of times randomly (using a computer program) to combine different input values selected from the probability distributions specified. The results consist of a set of probability distributions showing how uncertainties in key inputs might impact on key outcomes.
29
Box 4.5 provides an example illustrating the use of Monte Carlo analysis.7
 
BOX 4.5: ALLOWING FOR UNCERTAINTY IN AN ANALYSIS OF COSTS
 
 
The table below gives the costs of various parts of a construction project, broken down into excavation (E), foundations (F), structure (S), roofing (R), and decorations (D). All costs are independent of each other. The model for total cost is as follows:
  
           Total cost = E + F + S + R + D  
 
Costs for construction project (£)    
 
Minimum
Best Guess
Maximum
Excavation (E)
30,500
33,200
37,800
Foundations (F)
23,500
27,200
31,100
Structure (S)
172,000
178,000
189,000
Roofing (R)
56,200
58,500
63,700
Decoration (D)
29,600
37,200
43,600
 
 
From this information we can produce a best guess of £334,100 for the total cost of the project. However, we can also conclude a possible range from £311,800 to £365,200. Suppose the project would not go ahead unless the total cost is unlikely to exceed £350,000; how much assurance can we take from these figures that the total cost will be less than £350,000?
  
 
By undertaking a Monte Carlo analysis, we can simulate many possible values of the input variables, weighted so that the ‘best guess’ value is more likely than the extreme values. The total cost is calculated for each simulation, giving a distribution of values for total cost. The precise weighting depends on the probability distributions specified for each variable.
  
 
Using triangular distributions, it can be concluded that the most likely total cost is £334,000; and that the chance of total cost exceeding £350,000 is less than 1%.
  
back to top

IRREVERSIBLE RISK

30
Irreversibility occurs where implementation of a proposal might rule out later investment opportunities or alternative uses of resources. Examples of irreversibility are destruction of natural environments or historic buildings. It is particularly important to make a full assessment of the costs of any irreversible damage that may arise from a proposal.
31
Irreversibility is often associated with facilities on which people place ‘option values’ (the value of knowing a facility is available to enjoy, if they wish to do so). This is also linked to ‘existence values’ (the value of knowing that something continues to exist, even if the respondent does not expect to make any practical use of it).
32
Where lead options involve irreversible damage, assessment should include the consideration of options which involve delay, allowing more time for investigation of alternative less damaging ways to achieve stated objectives. Appraisal of different proposals should not ignore the ‘option’ value of avoiding or delaying irreversible actions, and the benefits of ensuring flexibility to respond to future changed conditions.

THE COST OF VARIABILITY IN OUTCOMES
33
In estimating the future costs and benefits associated with particular proposals, there will inevitably be variation between these estimates and the actual costs and benefits realised. This will be over and above the impact of optimism bias, and will be as a result of random factors unforeseen at the time of appraisal.
34
For the public sector as a whole, such random factors will tend to cancel out, taking all proposals together. But in some cases, this would not be expected to happen. Some projects - for example transport use - will tend to have appraisal risks that are systematically related to the overall performance of the economy. Because the majority or all of such projects will be affected by this same factor, appraisal errors will not cancel out between projects.
35
A decision-maker who is risk averse cares about this potential variability in outcomes, and is willing to pay a sum in exchange for certainty (or willing to put up with variability on receipt of compensation). This compensation is the cost of variability, and should be included in appraisal when it is considered appropriate.
36

Generally, a variability adjustment may be required when:

Risks are large relative to the income of the section of the population that must bear them (including very large risks borne by the whole population); or
When risk is correlated systematically with income or GDP, and so cannot be diluted by spreading across the economy.


37

The fraction of income worth paying for certainty (C) is approximated by the expression:

C = - var(y) / 2y*

where y is the net additional income resulting from the proposal, and y* is the total expected income or benefits (including the project income) of those impacted by the proposal.
38

Given the size of national income relative to the scale of most individual projects, the cost of variability for projects that benefit the community as a whole is usually negligible.
 
1
On the 20 Novemeber 2002, the government (Strategy Unit) published new proposals to help improve risk management in the public sector. See the Cabinet Office website for further details (http://www.cabinet-office.gov.uk/)
2
3
Reference can be made to RAMP (http://www.ramprisk.com/), or the OGC (http://www.ogc.gov.uk/) for a range of materials including ‘Managing a Successful Programme’, HM Treasury: Management of Risk: A Strategic Overview (The ‘Orange Book’), NAO: Supporting Innovation: Managing Risk in Government Departments. Also available are: Management of Risk: A Practitioner’s Guide, published through the Stationery Office, and the Risk Portal found on the Cabinet Office website (http://www.cabinet-office.gov.uk/)
4
See OGC website: http://www.ogc.gov.uk/
5
See website: http://www.hm-treasury.gov.uk/greenbook for empirical adjustments for generic project categories outlined in Review of Large Public Procurement in the UK, published in July 2002
6
‘Review of Large Public Procurement in the UK’, Mott MacDonald (2002), available at www.hm-treasury.gov.uk/greenbook
7
This example was adapted from ‘Measuring costs and benefits – a guide on cost benefit and cost effectiveness analysis’, National Audit Office (NAO) and Vose, D (1996)
Previous Contents Next