Ruled By Computers

BUILT BY HUMANS. RULED BY COMPUTERS by Gabe Cherry

In 2014, a computer system called MiDAS plucked his file out of the Michigan Unemployment Insurance Agency database and calculated, without any human review, that he had defrauded the unemployment system and owed the state of Michigan approximately $22,000 in restitution, penalties and interest – the result of a supposed $4,300 overpayment, plus Michigan’s customary 400 percent penalty and 12 percent interest. Then, still untouched by humans, MiDAS began to collect. It seized more than $10,000 from Russell by electronically intercepting his tax refunds in 2015 and 2016. He knew nothing about the fraud determination until his 2015 tax refund disappeared.

How do you beat something you can’t see? It’s like swinging in the dark. What are the laws that apply to a computer system? And what about us humans?
Brian Russell
Russell simply couldn’t afford the five-figure hit to his income. For the next two years, he made ends meet the best he knew how – he cancelled family trips, cut back on medical care for his diabetes, worked odd jobs. For a time, he lived in a friend’s basement.

While Russell struggled in the aftermath of the fraud determination, MiDAS kept rolling. An algorithm-based administration and fraud collection system implemented by the state of Michigan, it ran without human intervention for nearly two years between 2013 and 2015. During that time, it accused about 50,000 Michiganders of unemployment fraud. A 2017 review by the state found that more than 90 percent of those accusations were false.

Russell still doesn’t know why MiDAS accused him of fraud. He collected unemployment on and off a few years back when he was working as a journeyman electrician. Like generations of electricians before him, his union filed for unemployment on his behalf when he was between jobs. He can’t see the system, can’t touch it, can’t talk to it, can’t ask it why it has taken his money. The Michigan Unemployment Insurance Agency hasn’t shared any information with him.

“How do you beat something you can’t see?” Russell said. “It’s like swinging in the dark. What are the laws that apply to a computer system? And what about us humans?”

Generally, these algorithms keep a low profile. But occasionally, they collide spectacularly with humans. That’s what happened to Russell.

BUILT BY HUMANS. RULED BY COMPUTERS by Gabe Cherry

In 2014, a computer system called MiDAS plucked his file out of the Michigan Unemployment Insurance Agency database and calculated, without any human review, that he had defrauded the unemployment system and owed the state of Michigan approximately $22,000 in restitution, penalties and interest – the result of a supposed $4,300 overpayment, plus Michigan’s customary 400 percent penalty and 12 percent interest. Then, still untouched by humans, MiDAS began to collect. It seized more than $10,000 from Russell by electronically intercepting his tax refunds in 2015 and 2016. He knew nothing about the fraud determination until his 2015 tax refund disappeared.

How do you beat something you can’t see? It’s like swinging in the dark. What are the laws that apply to a computer system? And what about us humans?
Brian Russell
Russell simply couldn’t afford the five-figure hit to his income. For the next two years, he made ends meet the best he knew how – he cancelled family trips, cut back on medical care for his diabetes, worked odd jobs. For a time, he lived in a friend’s basement.

While Russell struggled in the aftermath of the fraud determination, MiDAS kept rolling. An algorithm-based administration and fraud collection system implemented by the state of Michigan, it ran without human intervention for nearly two years between 2013 and 2015. During that time, it accused about 50,000 Michiganders of unemployment fraud. A 2017 review by the state found that more than 90 percent of those accusations were false.

Russell still doesn’t know why MiDAS accused him of fraud. He collected unemployment on and off a few years back when he was working as a journeyman electrician. Like generations of electricians before him, his union filed for unemployment on his behalf when he was between jobs. He can’t see the system, can’t touch it, can’t talk to it, can’t ask it why it has taken his money. The Michigan Unemployment Insurance Agency hasn’t shared any information with him.

“How do you beat something you can’t see?” Russell said. “It’s like swinging in the dark. What are the laws that apply to a computer system? And what about us humans?”

We admit that humans are flawed and make mistakes. But we also know when mistakes have been made and we can correct. I wonder if it's rational to be optimistic that flawed humans can design automated systems to find and correct for flaws in the system?