Credit: Dreamstime
Explainable AI at DARPA
DARPA, the Defense Advanced Research Projects Agency, has an active program on explainable artificial intelligence managed by Dr. Matt Turek. From the program’s website (emphasis mine):
The Explainable AI (XAI) program aims to create a suite of machine learning techniques that:
New machine-learning systems will have the ability to explain their rationale, characterise their strengths and weaknesses, and convey an understanding of how they will behave in the future. The strategy for achieving that goal is to develop new or modified machine-learning techniques that will produce more explainable models. These models will be combined with state-of-the-art human-computer interface techniques capable of translating models into understandable and useful explanation dialogues for the end user. Our strategy is to pursue a variety of techniques in order to generate a portfolio of methods that will provide future developers with a range of design options covering the performance-versus-explainability trade space.
- Produce more explainable models, while maintaining a high level of learning performance (prediction accuracy); and
- Enable human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners.
Google Cloud’s Explainable AI
The Google Cloud Platform offers Explainable AI tools and frameworks that work with its AutoML Tables and AI Platform services. These tools help you to understand feature attributions and visually investigate model behavior using the What-If Tool.
IDG
Feature attribution overlays from a Google image classification model.
AI Explanations give you a score that explains how each factor contributed to the final result of the model predictions. The What-If Tool lets you investigate model performances for a range of features in your dataset, optimisation strategies, and even manipulations to individual datapoint values.
Continuous evaluation lets you sample the prediction from trained machine learning models deployed to AI Platform and provide ground truth labels for prediction inputs using the continuous evaluation capability. The Data Labeling Service compares model predictions with ground truth labels to help you improve model performance.
Whenever you request a prediction on AI Platform, AI Explanations tells you how much each feature in the data contributed to the predicted result.
H2O.ai’s machine learning interpretability
H2O Driverless AI does explainable AI with its machine learning interpretability (MLI) module. This capability in H2O Driverless AI employs a combination of techniques and methodologies such as LIME, Shapley, surrogate decision trees, and partial dependence in an interactive dashboard to explain the results of both Driverless AI models and external models.
In addition, the auto documentation (AutoDoc) capability of Driverless AI provides transparency and an audit trail for Driverless AI models by generating a single document with all relevant data analysis, modeling, and explanatory results. This document helps data scientists save time in documenting the model, and it can be given to a business person or even model validators to increase understanding and trust in Driverless AI models.
DataRobot’s human-interpretable models
DataRobot, which I reviewed in December 2020, includes several components that result in highly human-interpretable models:
- Model Blueprint gives insight into the preprocessing steps that each model uses to arrive at its outcomes, helping you justify the models you build with DataRobot and explain those models to regulatory agencies if needed.
- Prediction Explanations show the top variables that impact the model’s outcome for each record, allowing you to explain exactly why your model came to its conclusions.
- The Feature Fit chart compares predicted and actual values and orders them based on importance, allowing you to evaluate the fit of a model for each individual feature.
- The Feature Effects chart exposes which features are most impactful to the model and how changes in the values of each feature affect the model’s outcomes.
DataRobot works to ensure that models are highly interpretable, minimising model risk and making it easy for any enterprise to comply with regulations and best practices.
Dataiku’s interpretability techniques
Dataiku provides a collection of various interpretability techniques to better understand and explain machine learning model behavior, including:
- Global feature importance: Which features are most important and what are their contributions to the model?
- Partial dependence plots: Across a single feature’s values, what is the model’s dependence on that feature?
- Subpopulation analysis: Do model interactions or biases exist?
- Individual prediction explanations (SHAP, ICE): What is each feature’s contribution to a prediction for an individual observation?
- Interactive decision trees for tree-based models: What are the splits and probabilities leading to a prediction?
- Model assertions: Do the model’s predictions meet subject matter expert intuitions on known and edge cases?
- Machine learning diagnostics: Is my methodology sound, or are there underlying problems like data leakage, overfitting, or target imbalance?
- What-if analysis: Given a set of inputs, what will the model predict, why, and how sensitive is the model to changing input values?
- Model fairness analysis: Is the model biased for or against sensitive groups or attributes?
Explainable AI is finally starting to receive the attention it deserves. We aren’t quite at the point where “glassbox” models are always preferred over black box models, but we’re getting close. To fill the gap, we have a variety of post-hoc techniques for explaining black box models.
