If you ever want to drive online commenters insane, all you need do is write an article headlined ‘Why it’s often right to drive in the middle lane of the motorway’.

Many people insist that there is a clear-cut rule. These motoring Prussians claim that, unless overtaking a slower vehicle, you must drive in the left-hand lane at all times. But if you think about it, this rule only works if some people break it.

The modern organisation is in thrall to a tight fitness function that leaves little room for inventiveness

In reality, if everyone followed this keep-left rule assiduously, then at busy times traffic in the left-most lane would become excessively dense, forcing everyone to slow down, causing frequent braking and uneven use of the road. The best ‘rule’ here isn’t a hard-and-fast rule at all – it is simply ‘don’t drive like a git’. We notice the few middle-lane drivers who are idiots, but many are doing us all a favour.

Sometimes the best rules are loose rather than strict. If you allow intelligent people to exercise independent judgment, most are perfectly capable of coordinating their activities without the need for their every action to be codified or standardised. As Barbossa says in Pirates of the Caribbean: ‘The Pirate’s Code is more what you’d call “guidelines” than actual rules.’

The comedian Max Miller supposedly liked to stand on the terraces at football games in the 1930s periodically shouting at players: ‘Use your own judgment!’ In all kinds of decision-making in our personal lives – whom to marry, where to go on holiday – it is rightly understood that a degree of subjectivity is essential. Yet in institutional decision-making, any exercise of discretionary decision-making is seen as dangerous. Consistency of reasoning and adherence to process are prized more highly than quality of outcome.

Why the difference? One explanation is ‘defensive decision-making’. Monotonous adherence to procedure is a great way to avoid possible challenges or to insulate yourself against the risk of blame. It also suits the managerial mindset to impose universal metrics, however crude, simply for ease of comparison.

Generally, press articles attacking this bureaucratic mindset focus on the direct costs – the £100 million cost of the bat tunnel for HS2 or (in an outraged Daily Mail article) listing what government departments spend on biscuits. This mostly misses the point. The real price of a bureaucracy is not the cost, but the ‘opportunity cost’. The valuable activities which never happen because they now cost far too much, take far too long or involve too many lawyers. Our love of procedure costs us invisible billions in homes that aren’t built, businesses that don’t grow and ideas that never take root. Britain thus is a relatively high-trust economy which incurs all the costs of a low-trust economy simply to avoid accusations of inconsistency.

The mathematician Stephen Wolfram recently said that the reason evolution in nature works is that it has ‘quite a loose fitness function’. Whether you are a bacterium, a patch of moss, a mushroom or a shark, provided you can find a niche in which you can survive long enough to reproduce, you stay in the game. This loose rule leads to extraordinary biodiversity and multiple ingenious solutions to the same loosely defined problem.

By contrast, the modern organisation is now in thrall to a tight fitness function that leaves little room for diversity or inventiveness. Consider the quarterly financial reporting to which all listed companies must submit. It never seems to occur to anyone that the reasons these businesses don’t grow is that they are all forced to act in pursuit of the same narrow goals. The shareholder-value movement is another of those ideas which makes sense only until everyone adheres to it. At which point you have simply created a Soviet-style control economy in capitalist clothing.