Previous Contents Next
OVERVIEW OF APPRAISAL AND EVALUATION Two

 

INTRODUCTION
2.1
    

This chapter summarises the key stages of appraisal and evaluation. The remaining chapters discuss them in more detail.

THE APPRAISAL AND EVALUATION CYCLE
2.2
Appraisal and evaluation often form stages of a broad policy cycle that some departments and agencies formalise in the acronym ROAMEF (Rationale, Objectives, Appraisal, Monitoring, Evaluation and Feedback). This is shown below:
          BOX 2: ROAMEF CYCLE
 
BOX 2: ROAMEF CYCLE

back to top
THE ROLE OF APPRAISAL
2.3
Appraisals should provide an assessment of whether a proposal is worthwhile, and clearly communicate conclusions and recommendations. The essential technique is option appraisal, whereby government intervention is validated, objectives are set, and options are created and reviewed, by analysing their costs and benefits. Within this framework, cost-benefit analysis is recommended, as contrasted with cost-effectiveness analysis below, with supplementary techniques to be used for weighing up those costs and benefits that remain unvalued.
 

 

COST-BENEFIT ANALYSIS

Analysis which quantifies in monetary terms as many of the costs and benefits of a proposal as feasible, including items for which the market does not provide a satisfactory measure of economic value.

COST-EFFECTIVENESS ANALYSIS

Analysis that compares the costs of alternative ways of producing the same or similar outputs.


PROCESS FOR APPRAISAL AND EVALUATION

2.4 
Appraisals are often iterated a number of times before their proposals are implemented in full. Therefore the stages set out below may be repeated, and they may not always be followed sequentially. In particular, as options are developed, it will usually be important to review more than once the impact of risks, uncertainties and inherent biases. This helps to avoid spurious accuracy, and to provide a reasonable understanding of whether, in the light of changing circumstances, the proposal is likely to remain good value for money.
2.5

As the stages of an assessment progress, data must be refined to become more specific and accurate. The effort applied at each step should be proportionate to the funds involved, outcomes at stake, and the time available. Accordingly, in the early steps of identifying and appraising options, summary data only is normally required. Later on, before significant funds are committed, the confidence required must increase.

back to top
 Chapter 3 – Justifying Action
2.6
The first step is to carry out an overview to ensure that two pre-requisites are met: firstly, that there is a clearly identified need; and secondly, that any proposed intervention is likely to be worth the cost. This overview must include an analysis of the negative consequences of intervention, as well as the results of not intervening, both of which must be outweighed to justify action. In many cases, the preliminary step will involve research to set out the scope of the issue to be addressed, and the reasons for intervention.
Chapter 4 – Setting Objectives

2.7
The second step is to set out clearly the desired outcomes and objectives of an intervention in order to identify the full range of options that may be available to deliver them. Targets should be set to help progress towards meeting objectives.
 Chapter 5 – Option Appraisal

2.8
The third step is to carry out an option appraisal. This is often the most significant part of the analysis. Initially a wide range of options should be created and reviewed. This helps to set the parameters of an appropriate solution. A shortlist may then be created to keep the process manageable, by applying the techniques summarised below to high level estimates or summary data. The ‘do minimum’ option should always be carried forward in the shortlist, to act as a check against more interventionist action.
2.9
Each option is then appraised by establishing a Base Case.1 This is the best estimate of its costs and benefits. These estimates can then be adjusted by considering different scenarios, or the option’s sensitivity to changes can be modelled by changing key variables. More fully, the appraisal may develop as follows:
 
If required, adjust the valued costs and benefits for:
 
Distributional impacts (the effects of proposals on different sections of society);
Relative price movements.

back to top

 Chapter 6 – Developing and implementing a solution
2.10
Following option appraisal, decision criteria and judgment should be used to select the best option or options, which should then be refined into a solution. Consultation is important at this stage, regardless of whether it has taken place earlier. Procurement routes should also be considered, including the role of the private sector.
2.11
Issues that may have a material impact on the successful implementation of proposals must be considered during the appraisal stage, before significant funds are committed. This is to ensure that the outcome envisaged in the appraisal is close to what eventually happens.
Chapter 7 – Evaluation

2.12  
Evaluation is similar in technique to appraisal, although it obviously uses historic (actual or estimated) rather than forecast data, and takes place after the event. Its main purpose is to ensure that lessons are widely learned, communicated and applied when assessing new proposals.

 

PRESENTING THE RESULTS

2.13

The ultimate outcome of any appraisal is a decision whether or not to proceed with a proposal or a particular option. As these decisions will often have far reaching consequences, the presentation of the conclusions and recommendations to decision makers and key stakeholders can be as important as the analysis itself. In all cases, transparency is vital. Presentations and reports should be clear, logical, well founded, and geared towards helping the decision at hand. Summary reports in particular should be drafted in non-technical language wherever possible, but, if it is necessary to use technical terms, they should be explained.
2.14
Reports should provide sufficient evidence to support their conclusions and recommendations. They should provide an easy audit trail for the reader to check calculations, supporting evidence and assumptions. Major costs and benefits should be described, and the values attached to each clearly shown rather than netted off in the presentation of the analysis. This should help to ensure that decision makers understand the assumptions underlying the conclusions of the analysis, and the recommendations put forward. Appraisal reports should contain sufficient information to support the conduct of any later evaluation.
2.15
The results of sensitivity and scenario analyses should also generally be included in presentations and summary reports to decision makers, rather than just single point estimates of expected values. Decision makers need to understand that there are ranges of potential outcomes, and hence to judge the capacity of proposals to withstand future uncertainty.
  BOX 3: POSSIBLE OUTPUTS OF AN ECONOMIC APPRAISAL OR EVALUATION
 
Business cases (either Preliminary, Outline or Full) consisting of:
 
Strategic Case;
Economic Case (or Option Appraisal);
Financial Case (or Affordability);
Commercial Case;
Programme;
Project Management Case (or Achievability).
Regulatory Impact Assessment
Health Impact Assessment
Environmental Appraisal
Health and Safety Impact appraisal
Consumer Impact Assessment
Integrated Policy Appraisal (IPA)2
Evaluation and audit reports.

2.16


Departments and agencies often issue good practice templates for policy and project appraisals and evaluations.3

back to top


MANAGING APPRAISALS AND EVALUATIONS

2.17
Conducting an assessment can be resource-intensive. Appraisals and evaluations should therefore be carried out collaboratively wherever possible between stakeholders, but lead responsibilities need to be well defined, and accountability for accuracy and thoroughness clearly understood. Carrying out assessments should never be regarded as a specialist activity, and therefore sidelined.
2.18
Departments and agencies should consider how appraisals and evaluations are integrated with decision-making processes and governance structures. To ensure a coordinated approach to conducting assessments, departments and agencies are encouraged to consider:
 
Establishing formal evaluation or assessment units, or other centres of technical expertise;
Formalising access to internal and external auditors. In complex cases, it may be helpful to discuss appraisal methodology with sponsor departments, the Treasury or the National Audit Office;4
Providing incentives for conducting thorough and timely appraisals; and,
Maintaining an accessible archive.5
2.19
For individual assessments, consideration needs to be given at the outset to:
 
The availability and cost of financial and specialist resources that may be needed;
The possible need for quality assurance, for example, by academic experts and service providers;
How the findings are to be disseminated (e.g. publication of assessments; dissemination via web sites, etc);
The possibility of deferring a proposal pending further research; and,
Establishing a project plan for the assessment, setting out key milestones, resources and work streams.

 
  Advice is available on this guidance from:
 
Departmental analysts, Public Services Delivery Analysis (PSDA) team in HM Treasury, and a variety of referenced sources on specific issues.
Training on project and policy assessment is available from a range of sources, including the Civil Service College.

 

back to top

FRAMEWORKS

2.20

 The frameworks below are particularly relevant to appraisals and evaluations:
 
The OGC Gateway Review (mainly for programmes and projects);
The Regulatory Impact Assessment (mainly for policies involving regulatory impacts); and,
The Centre for Management and Policy Studies (CMPS) Policy Hub.
Office of Government Commerce Gateway Process
2.21
Gateway is a review process for civil procurement projects conducted by the Office of Government Commerce.6 It examines policies and projects at critical stages in their lifecycle to provide assurance that they can progress successfully to the next stage. Compliance with the Green Book is incorporated into the first and second gateways. Detailed information is available from the OGC website.7

Regulatory Impact Assessment

2.22
A regulatory impact assessment (RIA) is a policy tool that assesses the impact, in terms of costs, benefits and risks of any proposed regulation that could affect businesses, charities or the voluntary sector. It is Government policy that all government departments and agencies where they exercise statutory powers and make rules with general effect on others must produce an RIA. They should also produce an RIA for proposed European legislation that will have an effect on businesses, the public sector, charities or the voluntary sector in the UK.8
2.23
Although the trigger for producing an RIA is that the proposal could affect businesses, charities or the voluntary sector, the RIA itself should cover the full range of economic, social and environmental effects, in line with the Green Book methodology.
The CMPS Policy Hub
2.24
The CMPS Policy Hub9 aims to improve policy making and delivery, by providing:
 
Tailored access to resources and activities from the UK and abroad that help formulate, develop and evaluate policy more efficiently and effectively;
Innovative examples of improved policy making and delivery;
Tools to help break down organisational and geographical barriers, and improve collaborative working within and beyond government; and,

A platform for promoting the highest standards of research and evaluation.

 

back to top

ISSUES RELEVANT TO APPRAISAL AND EVALUATION


2.25
There is a wide range of generic issues that may need to be considered as part of any assessment. The following list should be checked for relevance to options under appraisal, and used for later evaluations:
 
Strategic impact – new proposals can be said to have strategic impacts on organisations if they significantly affect the whole or major part of an organisation over the medium to long term. Proposals should therefore be considered in terms of their potential scale of impact, and how they fit in with the strategy of the organisation(s) they affect.
Economic rationale – proposals need to be underpinned by sound economic analysis, which should be provided by a cost benefit analysis in an option appraisal. See Chapter 5 in particular.
Financial arrangements and affordability – proposals need to be affordable, and an affordable financial plan needs to be developed. See Chapter 6.
Achievability – all proposals should be assessed for their achievability, and recognised programme and project management arrangements set up as necessary. See Chapter 6.
Commercial and partnering arrangements – proposals need to take account of commercial, partnering and procurement arrangements; what can be delivered in the market; how costs and benefits can be guaranteed through commercial arrangements; how contracts will be managed through to completion. See Chapter 6.
Regulatory impact – as discussed previously, the impacts of new proposals on businesses, voluntary sector and charities should be assessed. See Chapter 2.
Legislation – consideration should be given to legislation specific to the case in hand, as well as statutes that affect many proposals, such as the Human Rights Act, or the Data Protection and Freedom of Information Acts.
Information management and control – The information requirements of proposals, including the data needed for later evaluation, and the supporting IT that may be required. Further guidance is available from the OGC.10
Environmental impacts – The effects on the environment should be considered, including air and water quality, land use, noise pollution, and waste production, recycling and disposal. Further guidance is available from ODPM, Defra and DfT.
Rural issues – The government is committed to ensuring that all its policies take account of specific rural circumstances. Appraisers should assess whether proposals are likely to have a different impact in rural areas from elsewhere. Further guidance is available from Defra.11
Equality – Impacts on various groups in society should be considered as part of an appraisal. Chapter 5 describes how distributional impacts should be brought into the appraisal process.
Health – the impacts of proposals on health should be considered, and evaluation made of the impact on health of poverty, deprivation and unemployment, as well as poor housing or workplace conditions. The Department of Health can provide further advice12, or can be accessed via the policy hub.13
Health and safety – the health and safety of people at work or arising from work activity may need to be safeguarded. Obviously this is of particular concern in construction. The Health and Safety Commission can provide further advice.14
Consumer focus – Assessments may need to involve consideration of the cost and quality of goods and services, as well as access to, choice of, and information about them.15
Regional perspectives – CMPS provides guidance on how regional perspectives are best incorporated into the policy making process.16
European Union – It will often be important to take account of proposals and activities in other European Union countries, as well as specific legislation and regulations. State aid rules are particularly important to consider, as these prescribe the extent to which government can intervene.17
Design quality – The design quality of facilities can be important in ensuring that objectives are successfully achieved.18
 
1
The term ‘Base Case’ is sometimes used to refer to the ‘do minimum’ option, but it is not used in this way in the Green Book.
2
The IPA is a policy tool that attempts to cover all aspects of an appraisal. It provides a checklist of questions on issues such as climate change, air quality, landscape, land use, waste, water, biodiversity and noise. Further guidance is available from ODPM, Defra and DfT.
3
For instance, the OGC provides business case templates on its website, which are recommended for use in project appraisals: http://www.ogc.gov.uk
4
See NAO website: http://www.nao.gov.uk/
5
See CMPS Knowledge Pools (http://policyhub.gov.uk), and ‘Adding It Up’ (http://www.addingitup.gov.uk/)
6
Similar processes exist in departments exempt from the Gateway Review Process, for example the Ministry of Defence’s ‘Smart Acquisition’ arrangements. Departments that are exempt should periodically review their monitoring procedures to ensure compliance with the Green Book methodology
7
8
Guidance is available in Better Policy Making and Regulatory Impact Assessment available from the RIU website: http://www.cabinet-office.gov.uk
9
10 See http://www.ogc.gov.uk
11
12
13
14
15
16
17
18
Previous Contents Next