(Image: David Lemm)
Software is deciding who gets a loan, who counts as a citizen and what prices you pay online. Who will step in when the machines get out of hand?
鈥淎MAZON is all kinds of broken.鈥� If you caught on 12 December last year, and were quick, you might have grabbed some exceptional bargains. For an hour only, Amazon was selling an odd mix of items 鈥� cellphones, video games, fancy-dress costumes, mattresses 鈥� for one penny.
Advertisement
. Goods usually marked 拢100 went for a 99.99 per cent discount. Hundreds of customers leapt at the chance, often buying in bulk. Even though Amazon reacted quickly and cancelled many orders, they were unable to recall those that their automated system had already dispatched from warehouses. Once set in motion, the process was hard to stop. Thanks to a software glitch, a handful of independent traders using Amazon鈥檚 Marketplace lost stock worth tens of thousands of dollars. Some faced bankruptcy.
. Most of the time they get on with business out of sight and out of mind. And business is booming. Automated processes are no longer simply tools at our disposal: they often make the decisions themselves. Much of the news we read, the music we listen to and the products we buy are served up automatically, based on statistical guesswork about what we want. Invisible chaperones shape our online experiences. Systems we can鈥檛 examine and don鈥檛 understand determine the route we take to work, the rates we get for mortgages, and the price we see for airfares.
鈥淎lgorithms are no longer simply tools: they often make the decisions鈥�
Many are proprietary and all are complex, pushing them beyond public scrutiny. How can we be sure they鈥檙e playing fair? A new wave of algorithm auditors are on the case, intent on pulling back the curtain on the hidden workings and hunting for undue bias or discrimination. But is this the fix? Do algorithms need better policing, or must we accept their nature as a price we pay for our automated world?
There鈥檚 nothing inherently mysterious about them: an algorithm is simply a set of instructions for getting something done. The trouble is that algorithms get nested inside or bolted on to others, interacting in ever more complex ways. It can also be hard to predict how algorithms will behave with real-world data once released into the wild.
The scope of their influence is often unclear. Some people swear blind that they鈥檝e seen , for example. Others think that鈥檚 bunk, an urban myth for our times. Such debates highlight the shadowy nature of today鈥檚 systems.
Not only are most algorithms secret recipes, sometimes even the developers who wrote them are in the dark. When at Northeastern University in Boston, Massachusetts, looked closely at how many of us have our search results skewed by factors like location and browsing history, she noted things even Google didn鈥檛 know: for example, that around 12 per cent of searches get personalised. Google engineers thanked her. They鈥檇 never made such measurements and hadn鈥檛 known the exact impact of their personalisation algorithms.
Exposing hidden algorithms can cause outrage. That鈥檚 what and his colleagues at the University of Michigan, Ann Arbor, found when they lifted the lid on Facebook鈥檚 newsfeed algorithms, which decide which posts from friends and family we actually see. The team compared filtered and unfiltered feeds and found that Facebook鈥檚 algorithms hid posts deemed uninteresting, according to unspecified criteria.
Around two-thirds of the participants in Sandvig鈥檚 study had no idea that algorithms were deciding what they saw. Many were shocked and upset when posts from close friends or family were excluded. Some had been blaming themselves or their friends for the algorithms鈥� work. 鈥淚f you post something and it doesn鈥檛 get any comments or likes, people assume that either their friends don鈥檛 like the topic, or their friends don鈥檛 like them,鈥� says Sandvig.
Even for news, it鈥檚 a popularity contest. During last year鈥檚 Ferguson riots in Missouri, for example, because these had hundreds of thousands of likes.
What Sandvig鈥檚 team did for Facebook, Hannak and her colleagues are doing for other online activity. Hannak is interested in how algorithms can tailor prices to different shoppers. In one , the researchers looked at how online retailers such as Walmart, Office Depot and Expedia varied prices according to factors including a user鈥檚 choice of browser, operating system and purchase history.
They found many instances of what they consider price discrimination though they are not sure of the rationale. Often the difference was small. Android users, for example, saw higher prices on about 6 per cent of items, though only by a few cents. In other cases, price quotes varied by up to $100. The greatest differences were typically seen between users who were logged in to a site and those who were not.
Crash damage
Hannak鈥檚 group now wants to understand exactly how location influences search results. They are simulating hundreds of Android phones and spreading them across Ohio using faked GPS coordinates. They鈥檒l also be looking to see whether people from rich and poor neighbourhoods get different search results when hunting for financial services.
Evidence of that may already have come to light. Some think hidden algorithms played a part in the 2008 sub-prime mortgage crash. Between 2000 and 2007, US lenders like Countrywide Home Loans and DeepGreen doled out home loans at an unprecedented rate via automated online applications. 鈥淓veryone was saying what a great innovation it was,鈥� says of Northern Iowa in Cedar Falls. 鈥淓veryone was very high on these fast web-based loans. No one anticipated the problem.鈥�
鈥淓veryone was high on fast web-based loans. no one anticipated the Problem鈥�
The problem was granting so many high-risk loans without human oversight. Americans from minority groups suffered most in the resulting crash. Automated processes crunched through vast amounts of data to identify high-risk borrowers 鈥� who are charged higher interest rates 鈥� and targeted them to sell mortgages. 鈥淭hose borrowers turned out to be disproportionately African American and Latino,鈥� says of the , a public policy think tank based in Washington DC. 鈥淎lgorithms played a role in that process.鈥�
The exact degree to which algorithms were to blame remains unclear. But settled with several cities, including Baltimore, Chicago, Los Angeles and Philadelphia, for hundreds of millions of dollars over claims that their sub-prime lending had disproportionately affected minorities. Although the decision-making process big banks used to target and sell sub-prime loans may not have been new in itself, the reach and speed of those decisions when algorithms were the driving force was new. 鈥淚t鈥檚 the scale factor,鈥� says Gangadharan. 鈥淭his was a problem that affected many people in the US and we have seen the effects fall along race and class lines in devastating ways.鈥�
Automated systems are replacing human discretion in ever more important decisions. In 2012, the US State Department started using an algorithm to randomly select the winners of the green card lottery. The system was buggy, however: it awarded visas only to people who applied on the first day, says , a Princeton University computer scientist who is investigating the event. Those visas were rescinded, but it鈥檚 a good example of how hidden algorithms can have a life-changing effect.
In a similar example, the documents leaked by Edward Snowden revealed that the National Security Agency uses algorithms to decide whether a person is a US citizen. According to US law, only non-citizens can have their communications monitored without a warrant. In the absence of information about an individual鈥檚 birthplace or parents鈥� citizenship, . Is this person in contact with foreigners? Do they appear to have accessed the internet from a foreign country? Depending on what you do online, your citizenship might change overnight. 鈥淥ne day you might be a citizen, another you might be a foreigner,鈥� says , at the University of Michigan in Ann Arbor. 鈥淚t鈥檚 a categorical assessment based on an interpretation of your data, not your passport or your birth certificate.鈥�
Algorithms are also used to police voter fraud. Several US states use software called Crosscheck to . But people have been deleted simply for having the same name. As with the sub-prime algorithms, minorities are again hit hardest. The names it scrubs are disproportionately those of black, Asian and Hispanic voters, who are more likely to share names 鈥� such as Jackson, Kim or Garcia.
The next scandal may be prison sentencing. Judges and lawyers in Missouri can use a website to make an . The system calculates incarceration costs for defendants, and weighs that against the likelihood the defendant will reoffend, based on prior criminal history and behavioural and demographic factors. Some think this will lead to minorities being given harsher sentences. Proxies like address, income and education level make it almost impossible to avoid racial bias. Similar systems are appearing across the US. 鈥淚 think it鈥檚 terrifying,鈥� says , a computer scientist at Haverford College in Pennsylvania.
The scales are falling from our eyes as the impact of algorithms is felt in almost every area of our lives. What should we do about it? In many of these examples, the problem is not the algorithms themselves, but the fact that they over-amplify an existing bias in the data.
Higher standards
鈥淧eople who work with algorithms are comfortable with the idea that they might produce these unintended results,鈥� says Sandvig. But for a growing number of people, that鈥檚 not good enough. , who works with Hannak at Northeastern University, thinks that large technology companies like Google and Facebook ought to be considered as public services that huge numbers of people rely on. 鈥淕iven that they have a billion eyeballs, I think they have a responsibility to hold themselves to a higher standard,鈥� he says.
Wilson thinks that automated systems might be made more trustworthy if users can control exactly how their results are personalised 鈥� such as leaving gender out of the equation or ignoring income bracket and address. It would also help us learn how these systems work, he says.
Others are calling for a new regulatory framework governing algorithms, much like we have for the financial industry, for example. A recent recommends that policy-makers pay more attention to what the algorithms do with the data they collect and analyse. To ensure accountability, however, there would need to be independent auditors who inspect algorithms and monitor their impact. We cannot leave it to governments or industry alone to respond to the problems, says Gangadharan.
鈥淭he big question now for me is who are the watchdogs,鈥� says Sandvig. For now, he suggests it should be the researchers who are beginning to reveal algorithms鈥� broader effects. Wilson, for example, is looking into setting up dummy credit profiles to better understand price-fixing systems. But independent auditors face tough obstacles. For a start, digging around inside many automated services violates their terms of use agreement, which prohibits attempts to analyse how they work. Under the US , such snooping may even be illegal. And while public scrutiny is important, the details of proprietary algorithms need to be kept safe from competitors or hackers, for example.
What鈥檚 more, most automated systems are too complex for humans to inspect by hand. So some researchers have developed algorithms that check other algorithms. Kroll is working on a system that would let an auditor verify that an algorithm did what it was supposed to with what it was given. In other words, it would provide a foolproof way of checking that the outcome of the green card lottery, for example, was in fact random. Or that a driverless car鈥檚 algorithm for avoiding pedestrians treats both people walking and people in wheelchairs with the same caution.
Friedler has a different approach. By understanding the biases inherent in the underlying data, she hopes to eliminate bias in the algorithm. Her system looks for correlations between arbitrary properties 鈥� like height or address 鈥� and demographic groupings like race or gender. If the correlation is expected to lead to unwanted bias, then it would make sense to normalise the data. It is essentially affirmative action for algorithms, she says.
That鈥檚 fine for cases where discrimination is clear, where a system is found to be unfair or illegal. But what if there is disagreement about how an algorithm ought to behave? Many would say Facebook鈥檚 filtering of its newsfeed keeps it readable. Some would argue that highly personalised price adjustment can benefit both customers and retailers. What鈥檚 acceptable to some won鈥檛 be for others.
As Sandvig notes, unlike for financial systems, there are no standards of practice governing algorithms. But how we want them to behave may turn out to be a harder question for society to answer than we think. Maybe we鈥檒l need an algorithm for that.
鈥淗ow we want algorithms to behave may be a harder question to answer than we think鈥�
Leader: 鈥�Living with the algorithms that run our lives鈥�
This article appeared in print under the headline 鈥淣o one in control鈥�
