Previous Contents Next
EVALUATION Seven

 

INTRODUCTION
7.1
When any policy, programme or project is completed or has advanced to a pre-determined degree, it should undergo a comprehensive evaluation. Major or on-going programmes, involving a series of smaller capital projects, must also be subject to ex post evaluations.
7.2
Evaluation examines the outturn of a policy, programme or project against what was expected, and is designed to ensure that the lessons learned are fed back into the decision-making process. This ensures government action is continually refined to reflect what best achieves objectives and promotes the public interest.
7.3
Evaluation comprises a robust analysis, conducted in the same manner as an economic appraisal, and to which almost identical procedures apply. It focuses on conducting a cost benefit analysis, in the knowledge of what actually occurred rather than what is forecast to happen.
7.4
In preparing for an evaluation, it is usually helpful to start with an outline plan, setting out the general boundaries of the proposed evaluation, including:
 
Questions which it seeks to answer;
Staff and other resources available;
Provisional timing and cost; and
Who should be consulted.

back to top

EVALUATION PROCESS
7.5
 The evaluation itself would normally follow this sequence:
 
1.
Establish exactly what is to be evaluated and how past outturns can be measured.
2.
Choose alternative states of the world and/or alternative management decisions as counterfactuals.
3.
Compare the outturn with the target outturn, and with the effects of the chosen alternative states of the world and/or management decisions.
4.
Present the results and recommendations.
5.

Disseminate and use the results and recommendations.

 
 
Evaluation requires management initiative (sometimes political commitment) and intensive monitoring. The thoroughness of an evaluation should depend upon the scale of the impact of a policy, programme or project, and to some extent on the level of public interest. There may be a high level of media interest around a project which has required a significant degree of expenditure, or one which is highly complex, novel, or represents a pilot for future large scale programmes. Evaluation reports should be widely disseminated and published, where appropriate, to contribute to the knowledge base upon which future decisions will be taken.


Establish what is to be evaluated
7.6
The activity to be evaluated needs to be clearly specified. The evaluation might be of a project, programme or policy, particular aspects of the activity, or of key common issues affecting a number of activities. It might also be a pilot designed especially for evaluation.

7.7
Objectives, outcomes and outputs should be defined and quantified as precisely as possible for use in step three below.1 It is important to distinguish between the objectives and outcomes, and the outputs and targets.
7.8
The availability of output and performance measures and targets, and other monitoring data, and how they relate to the objectives should be reviewed. If this information is inadequate, consideration should be given to the collection of additional data, although ideally, data needs would have been considered at the outset of the project.
Alternative States / Management Decisions
7.9
The definition of exactly what needs to be compared with what needs to be clearly stated. The outturn of any complex activity will never be exactly as projected in advance. However, the reasons for the outturn being better or worse than expected may be attributable to the ‘state of the world’, or to actions of the responsible body. These might include the management of the project, forecasting assumptions, or the inherent design of the policy.

back to top

Compare the Outturn with Targets

7.10
As discussed earlier, the technical methodologies used for appraisal and evaluation are similar. Each should identify and measure, where possible, both the direct and indirect benefits of the policy, programme or project. The main difference is that evaluation tends to be based on actual data, and appraisal on forecasts and projections.
7.11
The evaluation should include the following:
 
An assessment, quantified where possible, of what happened;
A comparison with the target outturn; and
A comparative assessment of one or more counterfactuals (i.e. alternative outturns given different states of the world, or different management decisions).
7.12
Where possible the comparative assessment should include a ‘control group’, to whom the activity was not applied.
7.13
It is usual to take as a benchmark for comparison, what would have happened if the activity under consideration had not been implemented. It is also useful to consider the consequences of implementing one or more of the alternatives considered during appraisal. Occasionally it may be appropriate to consider an option that was not originally appraised, as long as it was feasible at the time of implementation.

7.14

 
The evaluation should assess the success of the project, programme or policy in achieving its objectives, and also how this achievement has contributed to the wider outcomes. If the objectives were not achieved, the evaluation should establish why that was the case.
Presentation of results and recommendations
7.15
 The results of an evaluation should summarise:
 
Why the outturn differed from that foreseen in the appraisal;
How effective the activity was in achieving its objectives, and why;
The cost effectiveness of the activity; and
What the results imply for future management or policy decisions.

7.16

The results obtained should generally lead to recommendations for the future. These may include, for example, changes in procurement practice, delivery, or the continuation, modification, or replacement of a programme.

back to top

Disseminate the results and recommendations
7.17
The results and recommendations from evaluation should feed into future decision making. The methods used to achieve this will generally require senior management endorsement. Efforts should be made to disseminate the results widely, and, for this purpose, it may be helpful to use summaries of the main points, and reports which synthesise the results from a number of evaluations with common features.
7.18
Evaluation reports and the research that informs them should be placed in the public domain unless there are good reasons, in terms of security or commercial confidentiality, for not doing so.
Comparison of Appraisal and Evaluation
7.19
Box 25 sets out the differences between undertaking an assessment at the outset, in support of government intervention – appraisal – and undertaking an assessment to evaluate how successful such action has been – evaluation.
  BOX 25 COMPARISON OF APPRAISAL AND EVALUATION
 
  Appraisal Evaluation
Aim Ex ante assessment of whether action is worthwile and impacts Ex post assessment of whether action was worthwhile and impacts
Use of Output Project procurement, policy and programme design

Feedback for:
(a) future procurement, project management,
(b) wider policy debate, and
(c) future programme management.
Application Projects, policies and programmes Projects, policies and programmes
Timing Always prior to implementation
During implementation (‘formative’)
After implementation (‘summative’)
Data Forecasted Historic and current, estimated and actual. Estimates of counterfactuals
Method Comparison of options against ‘do nothing’ option Comparison of results against ‘do nothing’ option
    Comparison of actual outturns against target outturns/ alternative outturns
  Estimated assessment of risk Assessment of risks that did or did not materialise
Analytical Techniques Cost Benefit/ Effectiveness Analysis
Discounted cash flow analysis
Multi-criteria analysis
Other statistical analysis		 Cost Benefit/ Effectiveness Analysis
Discounted cash flow analysis
Multi-criteria analysis
Other statistical analysis – e.g.: analysis of
performance indicators
Decision Criteria Comparison of NPV, NPC for different options
Non quantifiable factors may be included if quantification impossible		 Consideration of whether correct criteria were used
Audit and Enforcement Public Accounts Committee (PAC), NAO, HMT, OGC Gateways 0, 1 Departmental arrangements PAC, NAO, HMT, OGC Gateway 5, Departmental arrangements
 

 

BOX 26: EXAMPLE ‘EXPANDING VOCATIONAL TRAINING’ - QUESTIONS FOR EVALUATION:
 
To what extent did the anticipated costs and benefits match the actual outcome (‘benefits realisation’)?
In the light of experience with the target group of trainees, would better results have been achieved if this group had been more tightly defined, e.g. the alternative option of focusing purely on low or unskilled workers?
Has any new information about the impact of vocational training come to light since the policy was implemented? (i.e. how effective is it in meeting objectives)?
Were the risks assumed for completion of the training course justified or did they understate/exaggerate the true risk?
Control group – how does the productivity of those individuals who undertook training compare to the productivity of workers of similar skill who were not offered training?

 
1

The objectives, outcomes and outputs of a policy, programme or project should have been identified and documented during appraisal. See Chapter 4 for more detail.

back to top

 
Previous Contents Next