Why Did We Abandon Classical Liberalism?

From the Encyclopedia Britannica (16th Ed. Vo. 27, p.425):

By the end of the 19th century, some unforeseen but serious consequences of the Industrial Revolution in Europe and North America had produced a deepening disenchantment with the principal economic basis of classical liberalism—the ideal of a market economy. The main problem was that the profit system had concentrated vast wealth in the hands of a relatively small number of industrialists and financiers, with several adverse consequences. First, great masses of people failed to benefit from the wealth flowing from factories and lived in poverty in vast slums. Second, because the greatly expanded system of production created many goods and services that people often could not afford to buy, markets became glutted and the system periodically came to a near halt in periods of stagnation that came to be called depressions. Finally, those who owned or managed the means of production had acquired enormous economic power that they used to influence and control government, to manipulate an inchoate electorate, to limit competition, and to obstruct substantive social reform. In short, some of the same forces that had once released the productive energies of Western society now restrained them; some of the very energies that had demolished the power of despots now nourished a new despotism.

So the problems with classical liberalism was that it concentrated wealth in the hands of an elite and this elite misused the power this gave them. Let me put this in own words.

1. Most people did not benefit. Poverty was widespread and many people lived in slums, with all the accompanying social problems such as diseases and substance abuse. There was child labour and many people died in the factories.
2. The system of free trade lead to cycles of boom and bust. Left to itself the market is subject to extreme fluctuations. Some decades later, John Maynard Keynes showed that government investment during these periods reduced the severity and length of these periods. 
3. Finally the government was captured by the wealthy interests, partly through lobbying, but mainly through rich people gaining party nominations and becoming representatives. Wealthy men also formed a series of exclusive networks based on such commonalities as where they went to school through which they promoted each others and excluded others. 

These problems helped to give birth to the new liberalism in which the state took a role in helping the poor to help themselves. Liberalism itself abandoned the free market ideology because it was destructive to society and created a form of government that was not concerned with liberty, but which rather tended towards a tyranny of the minority.

And when Milton Friedman and Friedrich von Hayek revived classical liberalism, creating neoclassical liberalism or neoliberalism, they ignored this history. Indeed, quite against the facts of history Hayek argued in 1944 that redistribution of wealth lead inevitably to totalitarianism. And subsequent history has shown that this is not true.

Neoliberalism became the dominant political ideology of the world. Poverty decreased in the third world but increased in the first. Economic recessions came thick and fast, with the global financial crisis being the worst since the Great Depression. And government has been captured by a wealthy elite.

But worse, this time around, is the existential threat from the destruction that large corporations and governments have wrecked on the environment. Pollution, extreme weather, sea-level rise, and mass extinction threaten our very survival. 

We abandoned classical liberalism because it didn't work. It still doesn't work in it's neoclassical form. And it's because Adam Smith and John Stuart Mill and others were fundamentally wrong about how human beings make decisions and about the fact of being a social primate. We need a new politics which takes into account what we now know about people.

"Those who cannot remember the past are condemned to repeat it."

– George Santayana