Satya Nadella (Microsoft)
Qantas is preparing to embark on the next phase of its trial using Microsoft’s HoloLens 2 mixed reality headset and technology to help train its engineers without putting them in harm’s way.
Working with Microsoft Mixed Reality Partner Program experts from US-based spatial computing software provider Altoura, Qantas has been putting a new Microsoft HoloLens 2 solution through its paces as part of a trial engineering training program.
Specifically, the proof of concept uses the HoloLens mixed reality technology to build a digital twin of a Boeing 737 flight deck, with the learner sitting in either the captain’s or first officer’s virtual chair to run through procedures and checklists.
“The key purpose is to allow learners to undertake an engine ground run in a safe, simulated environment – demonstrating responses to high risk events.” group learning technology manager for Qantas Amanda El Bahou said.
“What we are testing in the proof of concept is whether mixed reality will sufficiently simulate an experience that is a reliable test of competency.”
Making extensive use of Microsoft technologies, including HoloLens2 and Microsoft’s Mixed Reality Toolkit (MRTK), along with Azure as the underlying computational platform, Altoura developed a proof of concept over several months, which digitises a sample of steps within an engine ground run (EGR) test, according to Microsoft.
The solution sees Azure SQL handle all the system’s database requirements, with Azure Storage used extensively. Altoura has also created an Azure web app that runs its administrative portal so that end users, including Qantas, can create their own training content and protocols.
The proof of concept also integrates with Azure Spatial Anchors that allows physical and digital assets to co-exist, which is critical in these sorts of mixed reality environments.
While, by Microsoft’s own admission, it is too early to garner any metrics on Qantas’ proof of concept, mixed reality training systems that have been deployed in clinical healthcare settings have been shown to achieve equivalent training outcomes to more traditional live training techniques.
That said, Qantas has already been able to gain some insight into its trial training program. For example, Altoura has used Microsoft’s Mixed Reality Toolkit to track in fine detail, trainees’ hand movements, providing immediate feedback about the way they flicked a switch or moved the throttle.
Eye trackers, meanwhile, monitor what the user is looking at, and for how long. This also provides data that was previously not available that can be used to enhance training and ultimately boost safety in the air.
While the trial appears to be still in its early stages, Qantas is already looking forward to building it out into something bigger, with the next phase set to broaden the digitisation of the initial sample of steps needed for an engine ground run the full 45-step process, including responding to an engine fire.
It is understood that Qantas is also exploring how the technology could be deployed more widely across its business.
Details of the HoloLens2 trial come as Microsoft CEO Satya Nadella visits Australia for the company’s Innovate event in Sydney on 20 November, at Qantas, along with Telstra, Westpac and the CSIRO, among other local organisations, were showcased as emerging global innovation beacons.
Nadella used the event to reiterate Microsoft’s oft-stated mission to empower “every person and every organisation on the planet”.
“At a time when many are calling attention to the role technology plays in society broadly, our mission remains constant. It grounds us in the enormous opportunity and responsibility we have to ensure that the technology we create always benefits everyone on the planet, including the planet itself,” Nadella said.
