Tata Consultancy Services and Microsoft tackle AEMO big data
- 23 September, 2020 20:25
Tata Consultancy Services, along with Microsoft, has scored a project with the Australian Energy Market Operator (AEMO) testing an Azure-based big data Metering Data Management solution.
The new solution uses Cosmos DB as the main data store, leveraging Azure Kubernetes Services for the application and runtime layers.
One of AEMO’s principal responsibilities is settlement of the $16 billion-plus National Energy Market (NEM) which connects the grids of eastern and southern Australian states and territories to create a wholesale energy market.
Currently, the power to the NEM is dispatched in five-minute lots – but only settled every 30 minutes due to AEMO’s limited ability to segment its data.
The switch to five-minute settlement (5MS) scheduled for October 2021, aligns the dispatch and settlement times, and is expected to remove a number of potential market barriers for renewable energy providers for bidding and dispatching as well as encourage additional innovation.
There are more than 10 million meters connected in the retail and transmission network. With meter data storage requirements being three years, and the shift to 5MS, the volume of data to be stored will soar from 90 billion reads to around two trillion within the next five years.
AEMO’s digital transformation also includes leveraging Power BI to accelerate the production of reporting and the use of Databricks for the business intelligence and reporting services.
“Each day, we process more than seven million meter reads in the NEM, which takes several hours. Using this new technology, we can drastically reduce the processing time down to 45 minutes,” AEMO chief digital and business transformation officer Joe Locandro said.
“The platform has been designed to effectively scale its processing power in a more cost-effective way to deal with future growth, which is expected to triple to 20.5 million daily meter reads within the next four years, nearly a million an hour.”
Additionally, AEMO is also exploring the use of artificial intelligence and machine learning to enhance its analytics capability and build new classes of digital tools such as a next-generation simulator that can provide near real-time support for system security challenges, enhance forecasting and grid reliability.
As an example, AEMO is working on an energy simulator for the Australian east coast network, incorporating detailed models of wind and solar farms, and battery storage systems. It simulates their behaviour when connected to the grid to ensure the overall system is not negatively impacted, and the grid’s operation is simultaneously optimised for security and efficiency.
“At the new required level of modelling detail, running a one-minute simulation currently takes upwards of two hours of processing time – it is highly compute intensive,” he said. “We’re going to develop a simulator that will take it down to about three minutes to run a one minute of simulation without loss of fidelity.
"What that means is during a bushfire whereby powerlines are switched off or damaged, we will be able to quickly simulate and do ‘what-ifs’ to plan mitigation strategies.
“The simulator will be built in a hybrid platform, using cloud-based access to allow industry, researchers and developers to simulate the grid in an extremely fast fashion, which has never been done before.”