Governments and businesses today look to computing to cut costs, scale up workflows, and even uncover hidden opportunities. Jobs are getting replaced with algorithms at an amazing pace. Ideas and implementations differ, but one thing remains constant: the blind trust that computers are always correct. As software rapidly takes over work, many business leaders will have to consider the famous question asked by the Roman poet Juvenal: who watches the watchmen?
Consider how Internet changed retail sales. One-person shops can now sell globally, around the clock, without even owning a single warehouse. Amazon provides everything from physical storage through to payments to delivery. Add automated pricing to the mix, and it almost becomes too easy. Why leave money on the table when relentless robots can monitor, day and night, all your competitors and make sure your offer is the best?
Assisted by cloud services, small businesses can now punch well above their weight. However, one small software glitch can bring everything crashing down. As an example, one of the most popular automated pricing services dropped everything to just 1p before Christmas 2014 and bargain-hunters were able to purchase phones and mattresses at a ridiculous discount.
Possibly the worst part of this whole situation is that the glitch happened after working hours, so most people weren’t even aware that their entire stock was on a fire sale. Because the whole pipeline was completely automated, by the time humans became aware of the issue, most orders had already shipped and could not be canceled.
The dirty little secret of the software industry is that automation doesn’t make things better, only faster. The two are not necessarily the same. Small operational mistakes, kept under the radar at human speed, get turbo-charged when computers take over. For example, when Dallas police installed a new case management system in 2014, thirty years of tiny record entry inconsistencies suddenly surfaced, overloading caseworkers for weeks. The officers did not have time to deal with incoming cases promptly, and as a result, many newly arrested people had to be released without posting any bail. Two lucky jailbirds committed a violent robbery just three days later, with serious casualties.
Another way to look at process automation is akin to putting a rocket engine on a car. It will zoom ahead, but a tiny error in steering can cause a chain-crash. The best example is the meltdown of Knight Capital, previously one of the key brokers on the New York Stock Exchange. In 2012, Knight automated small retail order trading using a system called SMARS. It broke down orders into chunks and predicted the best time to send them to the exchange to make the most money. After a software update, one of the eight SMARS systems went rogue. It kept enthusiastically approving trades long after a whole order was complete. They lost fractions of cents on each trade, but after buying and selling more than $6 billion shares in less than one hour, the costs added up to over $460 million. The blunder led to the collapse of Knight Capital stock price, and ultimately, the departure of their CEO.
Digitisation of almost everything is an inevitable future, and so are the unpredictable glitches. However, small mistakes do not have to turn into a digital earthquake.
Apart from having a crystal ball that can see into the future, the best way to stop bad automation is to establish good systems of oversight.
One concrete step business leaders can take to keep their hands on the wheel is establishing good baselines for key workflows, and building up alerting mechanisms for unexpected changes in trends. If your inventory starts moving out too quickly, get people to investigate automation before it’s too late.
Another key consideration is that speeding up a single part of a process might create problems downstream. For example, increasing the capacity to send out customer notifications can overload your call centre and create more problems than it solves. Investigate and optimise operational bottlenecks throughout the entire pipeline before letting the algorithms roam free.
Finally, if an automated system outside your control is managing your key business data, such as prices, put monitors in place to check that automated changes are inside a valid range. For example, alert a person if the price goes too low or too high. This will help you avoid an unexpected fire-sale.
For my tester colleagues all over the world – a quick mind map with test heuristics from Humans vs Computers https://t.co/SGoqmGsAtK
— Gojko Adzic (@gojkoadzic) September 6, 2017
If you would like to have your company featured in the Irish Tech News Business Showcase, get in contact with us at [email protected] or on Twitter: @SimonCocking