Socio-economic analyses in perspective:
Uncertainties and bias in decision support
Associate Professor, PhD Kim Bang Salling
DTU Transport
Traffic days in Aalborg 2012 –
Special session: “Uncertainties in Transport Project Evaluation (UNITE)”
Project Plan of UNITE
Uncertainties in Transport Project Evaluation (UNITE): the five Work-Packages
(5) Evaluation methodology
WP5 project leader: Steen Leleur (DMG)
(4) Uncertainty calculation in transport models WP4 project leader: Otto Anker Nielsen (TMG)
(2) Organizational context of Modelling, an empirical study
WP2 project leader: Petter Næss (AAU) (3) Uncertainty calculation of cost
estimates
WP3 project leader: Bo Friis Nielsen (DTU Informatics)
(1) Systematic biases in transport models (recognized ignorance), an empirical study WP1 project leader: Petter Næss (AAU)
How do we evaluate transport projects?
• Various existing guideline report:
–Denmark, Sweden, UK, European Union, ....
• Socio-economic analysis by the use of Cost-Benefit Analysis (CBA)
• Produces single point estimates such as Net Present Values (NPV), Benefit Cost Ratios (BCR), etc
• However, no common rule have been set in order to acommodate the uncertainties in CBA!
–Recent conducted PhD dissertation proved this point
Background & Motivation
• The Manual for socio-economic analysis in the transport sector (2003)
–Unique guidelines for evaluating transport infrastructure projects
–Lack of uncertainty handling –Expected revision 2012-2013
How do we evaluate transport projects?
• However, no common rule have been set in order to acommodate the uncertainties in CBA!
–Recent conducted PhD dissertation proved this point
The Case Study: HH-Connection
• Connecting Denmark with Sweden: Scandinavian link –Currently, close to the capacity limit on Oresund
HH-Connection (alternatives*)
Description
(Alignment of connection)
Cost
(million DKK) Alternative 1 Tunnel for rail (2 tracks) person traffic only 7,700 Alternative 2 Tunnel for rail (1 track) goods traffic only 5,500 Alternative 3 Bridge for road and rail (2x2 lanes & 2 tracks) 11,500
Alternative 4 Bridge for road (2x2 lanes) 6,000
* Larsen, L.A. & Skougaard, B.Z. (2010). Vurdering af alternativer for en fast forbindelse Helsingør- Helsingborg, M.Sc. thesis, Department of Transport, Technical University of Denmark (in Danish)
The UNITE-DSS Modelling Framework
Todays Outline
Results: Cost-Benefit Analysis
• Construction costs – by far the largest contributor of costs
• User Benefits – by far the largest contributor of benefits – Consists of Ticket revenue and time savings
– Relies on the prognosis of future number of passengers i.e.
demand forecasts
HH-Connection (alternatives)
Cost
(million DKK)
BCR NPV
(million DKK)
Alternative 1 7,700 1.50 5,530
Alternative 2 5,500 0.16 -6,640
Alternative 3 11,500 2.71 28,240
Alternative 4 6,000 3.08 17,860
Are we telling the truth?!?!
Construction cost overruns
0%
200%
400%
600%
800%
1000%
1200%
1400%
1600%
1800%
2000%
Suez Canal Sydney Opera House Concorde Supersonic Aeroplane Boston's Artery/Tunnel Project, USA Humber Bridge, UK Boston- Washington- New York Great Belt Rail Tunnel, DK A6 Motorway Chapel-en-le- Frith/Whaley Shinkansen Joetsu Rail line, Japan Washington metro, USA Channel Tunnel, UK & France Karlsruhe- Bretten light rail, Germany Øresund Access links, DK & Sweden Mexico city metro line, Mexico Paris-Auber- Nanterre rail line, France
Cost Overruns (%)
Q: Have we learned anything from history?
”Chunnel” in 1987 £2,600 million (’85 prices) Completion 1994 £4,650 million (’85 prices) Total cost overrun of approx. 80%
”Øresund access link” in 1991 3.2 billion DKK (’90 prices) Completion 1998 5.4 billion DKK (’90 prices)
Total cost overrun of approx. 68%
Theoretical anchoring
The Transport Planning Phase: Adapted from the British Department for Transport (DfT) (2004)
Reference Class Forecasting: Optimism Bias
Inside View Outside View
”Uniqueness” of Project
”The Planning Fallacy”
Reference Class Forecasting
Forecasting of particular projects
Forecasting from a group of projects
(1) Identification of relevant reference
classes
(2) Establishing probability distribution
(3) Placing and comparing the
project
Optimism Bias Uplifts Current Situation
Optimism Bias and uplifts
• Deriving uplifts is highly dependet on large data-sets
–Flyvbjerg from (AAU) has since 2003 developed a large database
–Unfortunately, it looks upon mega-projects
• The basis is Reference Class Forecasting i.e. statistical measurements on various project pools
Source: Flyvbjerg and COWI (2004)
Results : Optimism Bias Uplifts
• The BCR are lower, however, still point estimates towards DM –Moreover an advanced form of sensitivity analysis
• Imply to introduce risk analysis and Monte Carlo simulation
HH-Connection (alternatives)
Cost (uplifted) (million DKK)
BCR (orig.) (from slide 8)
BCR (uplifts):
80% uplift
Alternative 1 12,090 1.50 0.97
Alternative 2 8,640 0.16 0.10
Alternative 3 15,180 2.71 1.75
Alternative 4 7,920 3.08 1.98
The UNITE Project Database (UPD)
• The convention used is as follows:
( ( ) )
forecasted forecasted actual
X X
U X − ×100
=
Over estimation of Demand
• Demand forecasts (user benefits) are derived:
– U is percent inaccuracy,
– Xa is the actual traffic after the project is opened – Xf is the forecasted traffic on the decision to build
• Combination of two database samples
0 5 10 15 20 25 30
(-120;-100) (-100;-80) (-80;-60) (-60;-40) (-40;-20) (-20;0) (0;20) (20;40) (40;60) (60;80) (80;100) (100;120) (120;140) (140;160) (160;180) (180;200) (200;220) (220;240)
Frequency of occurence (%)
Inaccuracies in demand forecasts (%)
Inaccuracies in demand forecasts (road projects)
Salling et al. (2012)
Flyvbjerg et al. (2003) Nicolaisen et al. (2012)
The UNITE Project Database (UPD)
• The convention used is as follows:
( ( ) )
forecasted forecasted actual
X X
U X − ×100
=
Under estimation of costs
• Construction costs bias derived similarly:
– U is percent inaccuracy,
– Xa is the actual traffic after the project is opened – Xf is the forecasted traffic on the decision to build
• Combination of two database samples
0 10 20 30 40 50
(-100;-80) (-80;-60) (-60;-40) (-40;-20) (-20;0) (0;20) (20;40) (40;60) (60;80) (80;100) (100;120) (120;140) (140;160) (160;180) (180;200) (200;220) (220;240)
Frequency of occurence (%)
Inaccuracies in construction costs (%)
Inaccuracies in construction cost (road projects)
Salling et al (2012) Flyvbjerg et al. (2003)
Nicolaisen et al. (2012)
Results (RCF): Monte Carlo simulation
Conclusions
• Feasibility risk assessment can be carried out by using historical experience stemming from RCF in order to obtain interval
results
• An important aspect in RCF and UNITE is to set and validate input parameters. Hence, empirical data enter the
assessment.
• The RCF approach has been illustrated on a case example concerning the construction of a new fixed link, the HH-
Connection, between Denmark and Sweden.
• Clearly vital to include uncertainties within socio-economic analyses in order to validate results
Perspectives
• Recovering of further data (UPD) with regard to both the
demand forecast uncertainty as well as the construction costs through large-scale research study
• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model combined with overconfidence theory allows for expert opinions (SIMSIGHT)
• More info on UNITE can be found: (www.transport.dtu.dk/unite)
• An international conference on the topic is scheduled in
September 2013 – a specific call will be posted in the upcoming month.
SIMSIGHT: Decision Conferencing (DC)
• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model
• Enables to include Stakeholders and Decision-makers in an early stage, i.e. to include experts opinion on MIN and
MAX values as entries to the Monte Carlo simulation
Results from DC and RSF
SIMSIGHT: Overconfidence
Perspectives
• Recovering of further data (UPD) with regard to both the
demand forecast uncertainty as well as the construction costs through large-scale research study
• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model combined with overconfidence theory allows for expert opinions (SIMSIGHT)
• More info on UNITE can be found: (www.transport.dtu.dk/unite)
• An international conference on the topic is scheduled in
September 2013 – a specific call will be posted in the upcoming month.
Thank you for your attention!
Affiliation:
Associate Professor, PhD Kim Bang Salling Department of Transport Technical University of Denmark kbs@transport.dtu.dk
