In several recent posts I’ve taken a look at people’s revealed preferences for roads (nobody’s willing to pay directly for them) and public transport, walking, and cycling (people are queuing up to get on the train). In those posts, I’ve argued that observing how people vote with their feet (or their wallets) can teach us a lot about demand for different travel modes.
Rail is now growing too fast to be un-fit for survival.
But as any economist knows, markets have two sides to them: demand and supply. As transport infrastructure has a lot of “public good” characteristics, it tends to be provided by government agencies such as Auckland Transport and the New Zealand Transport Agency. (These agencies wouldn’t say no if a private company turned up and offered to build a new motorway at no cost to them… but that’s not going to happen any time soon due to the fact that most recent private toll roads have failed financially.)
As a result, we have to consider how transport agencies make decisions about what to supply to the market. I’ve written a few posts on the basics of cost-benefit analysis, which is one of the tools that they use to decide which projects to build.
But is cost-benefit analysis robust, or are the results systematically biased in a certain direction? Thinking about this question led me to re-read one of my favourite papers on infrastructure costings (don’t laugh!): Bent Flyvbjerg’s “Survival of the Un-fittest: Why the Worst Infrastructure Gets Built – and What We Can do About It” (fulltext pdf). Flyvbjerg takes an empirical look at hundreds of major infrastructure projects around the world, finding that cost overruns are all-pervasive:
- 9 out of 10 projects have cost overrun.
- Overrun is found across the 20 nations and 5 continents covered by the study.
- Overrun is constant for the 70-year period covered by the study, cost estimates have not improved over time.
In addition, benefits are systematically overestimated in ex-ante evaluations. The result is that a number of bad projects get built on the back of over-optimistic business cases. Flyvbjerg attributes this to “cognitive and political biases such as optimism bias and strategic misrepresentation”. (This is a polite way of saying “lying about the project to ensure that it gets built.”)
So how do New Zealand’s transport agencies stack up against Flyvbjerg’s analysis? Fortunately, we’ve got some empirical data to investigate this question with. Between 2009 and 2012, NZTA conducted and published a number of post-implementation reviews of (mainly) road projects that it funded in part or fully. Matt did an excellent job summarising the data in a post last year.
While the projects aren’t necessarily representative of all road projects, they do run the gamut from small pavement upgrades to multimillion state highway expansions. NZTA provided data comparing ex-ante and ex-post evaluations of costs and benefits for 69 projects in total. I subjected the data to some basic statistical analysis, finding that:
- The average project had a cost overrun of 34% – a difference that was found to be highly statistically significant, meaning that there is a less than 1% probability that the observed difference happened by chance.
- The average project had actual benefits that were 28% lower than expected – although as this difference was not statistically significant we can’t determine whether it simply reflects random chance.
In other words, NZTA and regional transport agencies seem to have had some issues accurately costing road projects. And the errors they are making are not random – they have systematically underestimated costs. This can be seen really clearly if we graph the data in histogram format.
Here’s the data on construction cost overruns, in percentage terms. The size of the bars represents the number of projects. Bars to the right of the black line indicate projects where costs were higher than expected. As you can see, costs were higher than expected for the vast majority of projects – sometimes to a quite significant degree (i.e. over 100% more expensive than planned).
And here’s a similar chart for benefit overruns/underruns. This shows that although estimates of benefits have in some cases been wrong by a quite large amount, most of the errors are clustered closer to the zero line. This shows that while NZTA or transport agencies often miss the mark on their estimates of benefits, the errors are sometimes positive and sometimes negative. In other words, optimism bias seems to be less pervasive when estimating benefits than when estimating costs.
This data has (or should have) important implications for the way we plan and fund transport projects. It suggests that it’s necessary to be much more conservative when estimating the costs and benefits of road projects. This is especially important in light of the fact that NZTA’s funding is being devoted in large part to major motorway projects – the kind of “megaprojects” that Flybjerg identifies as posing the greatest risks for good project evaluation.
Unfortunately, NZTA stopped publishing post-implementation reviews in 2012, so it’s impossible to say whether agencies have used this data to refine their cost estimates. I hope they have, but there are indications that optimism bias is still running rampant. Take, for example, NZTA’s long-term forecasts of road traffic and public transport patronage, which blithely disregard the market realities. Or, more concretely, there’s the strange case of the Additional Waitemata Harbour Crossing traffic forecasts, which Matt picked up on a few years ago.
A 2010 business case for the AWHC, which would be New Zealand’s most expensive infrastructure project of all time, found that the project’s benefit-cost ratio was a mere 0.4 to 0.6. (Indicating that it costs about twice as much as it returns in benefits.) But, as it turns out, this figure was based on traffic modelling that overestimated actual traffic across the bridge in 2008 by almost 10% – in spite of the fact that the actual data was available at that point. That’s some serious optimism bias right there…
Auckland Harbour Bridge Traffic volumes (actual and forecast)
Finally, it’s also worth noting that Flyvbjerg finds that cost overruns (and benefit underruns) tend to be a more serious issue for rail projects than for road projects, especially in the United States. Unfortunately, we simply haven’t completed enough rail projects to robustly evaluate whether the same holds true in New Zealand. However, there are some signs that recent public transport infrastructure projects have outperformed their business cases – as seen in NZTA’s post-implementation review of the Northern Busway and booming ridership at Britomart.