You can’t secure what you can’t explain
Another area of research is explainability. Today, many AI and ML systems — including the AI- and ML-powered tools offered by many major cybersecurity vendors — are “black box” systems. “Vendors are not building explainability in,” says Sounil Yu, CISO-in-residence at YL Ventures. “In security, being able to explain what happened is a foundational component. If I can't explain why it happened, how can I fix it?”
For companies building their own AI or ML systems, when something goes wrong, they can go back to the training data or to the algorithms used and fix the problem. “If you’re building it from someone else, you have no idea what the training data was,” says Yu.
Need to secure more than just algorithms
An AI system isn't just a natural language processing engine or just a classification algorithm or just a neural network. Even if those pieces are completely secure, the system still must interact with users and back-end platforms.
Does the system use strong authentication and the principles of least privilege? Are the connections to the back-end databases secure? What about the connections to third-party data sources? Is the user interface resilient against injection attacks?
Another people-related source of insecurity is unique to AI and ML projects: data scientists. "They don't call them scientists for nothing," says Othot's Abbatico. "Good data scientists perform experiments with data that lead to insightful models. Experimentation, however, can lead to risky behavior when it comes to data security." They might be tempted to move data to insecure locations or delete sample data sets when done working with them. Othot invested in getting SOC II certification early on, and these controls help enforce strong data protection practices throughout the company, including when it comes to moving or deleting data.
“The truth is, the biggest risk in most AI models everywhere is not in the AI,” says Peter Herzog, product manager of Urvin AI, an AI agency, and co-founder of ISECOM, an international non-profit organization on security research. The problem, he says, is in the people. “There's no such thing as an AI model that is free of security problems because people decide how to train them, people decide what data to include, people decide what they want to predict and forecast, and people decide how much of that information to expose.”
Another security risk specific to AI and ML systems is data poisoning, where an attacker feeds information into a system to force it to make inaccurate predictions. For example, attackers may trick systems into thinking that malicious software is safe by feeding it examples of legitimate software that has indicators similar to malware.
It’s a high concern to most organizations, says Raff. “Right now, I’m not aware of any AI systems actually being attacked in real life,” he says. “It’s a real threat down the line, but right now the classic tools that attackers use to evade antivirus are still effective, so they don’t need to get fancier.”
Avoiding bias, model drift
When AI and ML systems are used for enterprise security — for user behavior analytics, to monitor network traffic or to check for data exfiltration, for example — bias and model drift can create potential risks. A training data set that under-represents particular attacks or that is out of date quickly can leave an organization vulnerable, especially as the AI is relied on more and more for defense. “You need to be constantly updating your model,” says Raff. “You need to make it a continuous thing.”
In some cases, the training can be automatic. Adapting a model to changing weather patterns or supply chain delivery schedules, for example, can help make it be more reliable over time. When the source of information involves malicious actors, then the training data sets need to be carefully managed to avoid poisoning and manipulation.
Enterprises are already dealing with algorithms creating ethical problems, such as when facial recognition or recruitment platforms discriminate against women or minorities. When bias creeps into algorithms, it can also create compliance problems, or, in the case of self-driving cars and medical applications, can kill people.
Just as algorithms can inject bias into predictions, they can also be used to control for bias. Othot, for example, helps universities with such goals as optimizing class sizes or achieving financial goals. Creating models without appropriate constraints can very easily create bias, says Othot's Abbatico. "Accounting for bias requires diligence. Adding goals related to diversity helps the modeling understand objectives and can help counter bias that could easily be incorporated in admissions if diversity goals weren't included as constraints."
The future of AI is cloudy
AI and ML systems require lots of data, complex algorithms, and powerful processors that can scale up when needed. All the major cloud vendors are falling over themselves to offer data science platforms that have everything in one convenient place. That means that data scientists don't need to wait for IT to provision servers for them. They can just go online, fill out a couple of forms, and they're in business.
According to the Deloitte AI survey, 93% of enterprises are using some form of cloud-based AI. “It makes it easier to get started,” says Deloitte’s Loucks. These projects then turn into operational systems, and as they scale up, the configuration issues multiply. With the newest services, centralized, automated configuration and security management dashboards may not be available, and companies must either write their own or wait for a vendor to step up and fill the gap.
When the people using the systems are citizen data scientists or theoretical researchers without strong backgrounds in security, this can be a problem. In addition, vendors historically roll out new features first and security second. That can be a problem when systems are rapidly deployed and then even more rapidly scaled. We’ve already seen this happen with IoT devices, cloud storage and containers.
AI platform vendors are becoming more aware of this threat and have learned from the mistakes says Raff. “I’m seeing more active inclusion of plans to include security than we might otherwise expect given the historic 'security comes last' mindset,” he says. “The ML community is more concerned about it, and the lag time is probably going to be shorter.”
Irfan Saif, principal and AI co-leader at Deloitte, agrees, especially when it comes to the major cloud platforms that support large enterprise AI workloads. “I would say, yes, they are more mature than maybe prior technologies have been in terms of the evolution of cybersecurity capabilities.”
Security checklist for AI projects
The following checklist to help secure AI projects is from Deloitte’s State of AI in the Enterprise, 3rd Edition:
- Keep a formal inventory of all AI implementations
- Align AI risk management with broader risk management efforts
- Have a single executive in charge of AI-related risks
- Conduct internal audit and testing
- Use outside vendors to conduct independent audits and testing
- Train practitioners how to recognize and resolve ethical issues around AI
- Collaborate with external parties on leading practices sound AI ethics
- Ensure that AI vendors provide unbiased systems
- Establish policies or a board to guide AI ethics
Editor's note: This article was originally published on CSO on 09/02/2019. It has since been updated with new information.