Between $6 and $9 trillion dollars—about 8% of annual world-wide economic production—is currently being spent on projects that individually cost more than $1 billion. These mega-projects (including everything from buildings to transportation systems to digital infrastructure) represent the biggest investment boom in human history, and a lot of that money will be wasted.

some-recent-mega-projects-compared-with-us-debt-to-china-cost-of-the-project_chartbuilderJust seven recent mega-projects total up to almost the entire debt between the world’s two largest economies.

Humans have always wanted to build big, whatever the consequences. Even the earliest religious mythology refers to attempts to build a tower to the heavens, often ending in disaster. And early civilizations around the world undertook giant constructions such as the Taj Mahal in India or the pyramids in Egypt at great human cost. And yet we keep doing it, and the reason has to do with our willingness to fool ourselves.

In the late nineties, Bent Flyvjberg, now an economist at Oxford University’s Said School of Business, watched his home country of Denmark building what was the world’s longest suspension bridge and second-largest undersea tunnel. The project was a disaster of biblical proportions, Flyvjberg says, with problems ranging from cost overruns to floods and fires, and the taxpayers were taking a loss before it was even finished. But there had been no comprehensive study of whether these problems were were typical of such projects, so Flyvjberg set out to find out the answer.storebraen-040214

Over the course of the last fifteen years, he has looked at hundreds of mega-projects, and he found that projects costing more than $1 billion almost always face massive cost overruns. Nine out of ten projects faces a cost overrun, with costs 50% higher than expected in real terms not unusual. We’ve written about Flyvbjerg’s specific research on mega-dams because numerous emerging markets are taking them on as solutions to their electricity deficits, and likely throwing away hundreds of millions of dollars on them.

In fact, the number of mega-projects completed successfully—on time, on budget, and with the promised benefits—is actually too small for Flyvbjerg to determine why they succeeded with any statistical validity. He estimates that only one in a thousand mega-projects fit that criteria.

But why this pervasive failure to predict these overruns, despite their seeming inevitability? The answer lies in the very things that make us human, what Flyvbjerg calls the “four sublimes”: the excitement of engineers and technologists building the newest or largest item of its kind; the rapture politicians receive from building monumental works that increase their public profile; the delight of businesses and trade unions generating money and jobs; and the aesthetic pleasure generated by iconically large design.

All this optimism contributes to overestimates of benefits and underestimates of cost. The economist Albert O. Hirschman saw this and came up with an idea of “the hiding hand”—if people knew the actual costs of projects like these, they would never undertake any, so there must be a mechanism that hides this reality from people, engendering progress but also the occasional disaster. It’s not so different from the idea that financial complexity exists to engender investment that would never happen if people understood the real costs.

Flyvbjerg rejects the hiding hand hypothesis, however. It allows public officials a pass on using shoddy accounting to push a project through, and then say things like “if people knew the real cost from the start, nothing would ever be approved,” as former San Francisco Mayor Willie Brown did in 2013 to justify the cost overrun for building the San Francisco Transbay Terminal. That reasoning has created a situation where only the projects with the most misleading cost forecasts are approved.

Flyvbjerg argues instead for more study of why successful projects succeed and more independent analyses of existing proposals, as well as efforts to recognize the incentives for bad forecasting that come with the financial and political interests of those touting the projects.

One promising sign he sees: Mega-projects are more and more likely to damage people’s careers, especially as they move into the digital realm. Cost overruns with the Airbus A380 jumbo jet cost a chief executive his job; a failed IT upgrade at K-Mart lead to the company’s bankruptcy, and China’s rail minister lost his job over corruption and safety problems. Even US president Barack Obama’s reputation has been dented by the troubled rollout of digital marketplaces built as part of his signature healthcare reform law.

“IT projects are like ticking time bombs, and they can undo people’s careers, whole companies and now even presidencies,” Flyvbjerg told Quartz. “We find it a fascinating area to study.”

Photo Credit: The Dubai skyline with Burj Khalifa (C) is seen during the late afternoon from the Sheikh Zayed highway March 28, 2010. Jumana ElHeloueh / Reuters