In 2017, the New Zealand Accident Compensation Corporation (ACC) – which manages a scheme to cover citizens and visitors who are injured in an accident – came under fire for using a prediction modelling tool to highlight which of its clients may be on the scheme for long periods.
The ACC was accused of possibly using the tool to decline applicants with long predicted treatment times, or following cues to “harass and harangue” someone to get off the scheme if the tool decided they had been on it for a long period.
Researchers called on the ACC to come clean about how the tool was being used. Critics called the lack of transparency “unacceptable”.
The tool used client data from 364,000 claims lodged between April 2007 and May 2013. At the heart of the criticism was the fact people weren’t aware their data was being used in this way.
“People have a right to know what is being done with their data by the Government," said specialist ACC lawyer Warren Forster.
The ACC responded by saying clients gave up their information when they agreed to its privacy notice.
Cases such as these serve as a warning to companies wanting to use their customers data to train AI systems and improve services. Just because organisations have sought the legally required consent to use data doesn’t make them immune to a backlash from customers.
Avoiding such a situation was a central talking point at a discussion at a SAS event in Sydney last week, by a panel featuring Telstra data, reporting and analytics executive Liz Moore; Bank of New Zealand head of data strategy and transformation, Sonya Crosby; and chief risk officer of NSW workers compensation scheme iCare, Gavin Pearce.
“They collected data for one purpose and were using it to essentially triage claims. The public probably got the wrong end of the stick. There was an absolute media storm,” said Pearce
“[The ACC response] was almost ‘nothing to see here’ instead of going down that transparency route of saying this is what we're doing and why,” he added.
“The way it needs to be managed is all about transparency. This is the customer’s data not our data. So whenever we’re wanting to support them using that data in any way we need to tell them. We need to explain the purpose for which it’s being used and make sure you have explicit consent,” said Crosby.
“That visibility is really important, if people understand the why and if the context is correct and its delivering a benefit to the customer than that’s half the problem solved,” she added.
The sentiment was echoed by Telstra’s Moore who was previously in a marketing focused role at the telco.
“It’s us being really clear and communicating effectively around what that specific benefit is. And being really clear about what we’re doing rather than using industry jargon,” she said.