IT IS no exaggeration to say that civilisation has entrusted its continued existence to the smooth functioning of technological systems. This trust runs so deep as to be nearly invisible.
We鈥檙e now in the early days of a turning point in our dependence upon machines. Last century, we trusted them to do things for us; this century, we鈥檙e starting to trust them to decide things for us.
Advertisement
That鈥檚 not all bad. Humans have a notoriously patchy record when it comes to decision-making. But relying on technological systems to make decisions for us 鈥� especially when risks are involved and our safety is at stake 鈥� can have major consequences.
Developments in promise to give some of our technologies extraordinary decision-making skills. With self-driving cars, for example, the experiences of each individual vehicle can be combined, adding together every mile driven, every hazard identified and every accident avoided or not, such as a self-driving car鈥檚 recent self-inflicted run-in with a bus.
Within a year of self-driving autos being widely available, such an aggregated artificial intelligence could have orders of magnitude more experience than any human driver. These patterns of improved performance are likely in other systems that have many inputs along with a clear distinction between what鈥檚 correct and what鈥檚 not, such as in medical diagnostics or financial analysis.
But along with successes will come failures, where machine decisions lead to disaster.
Some will come from faulty programming 鈥� no software is perfect, and even the best can have bugs. Others will come from situations or conditions outside of programmed (or learned) recognition, or when multiple correctly functioning systems interact in an undesirable way.
Our dilemma is the result of being too willing to trust technologies to make the right choices 鈥� and being unable to recognise an incorrect choice until it鈥檚 too late.
It鈥檚 here that faith in decision-making technologies can be the most problematic. We may trust too much 鈥� one experiment found that people blindly followed a robot into danger in a mock emergency.
We may believe that a machine 鈥渒nows鈥� more than we do, or can access information we can鈥檛. We鈥檒l need to bring a healthy scepticism into interactions with them.
Civilisation鈥檚 adoption of decision-making machines will have its own learning curve so that we realise when trust is appropriate. And we鈥檒l definitely have to figure out how to identify and adapt to situations where our machines are in over their metaphorical heads鈥� and hit the brakes for them.
This article appeared in print under the headline 鈥淚n machines we trust鈥�
