Credit: Dreamstime
As enterprises expand their machine learning (ML) capabilities to analyse data generated by increasingly complex applications, New Relic has updated its One full-stack observability application to include machine learning operations (MLOps) designed to help manage multiple data and ML models across different business units.
Along with application, network, infrastructure, browser monitoring, and log and error management, One is designed to allow data scientists and ML engineers to not only monitor ML model performance but also retrain models after raising alerts, said Guy Fighel, general manager of applied intelligence and group vice president of product engineering at New Relic.
Observability is a relatively new term in IT, used to describe the task of monitoring enterprise applications, data flow and distributed infrastructure. Systems that offer observability go beyond prior application performance monitoring (APM) programs, offering a high-level overview of IT infrastructure as well as granular metrics, to allow for efficient application, network, data, and security management.
According to a research report released by log-management application provider LogDNA, 75 per cent of responding companies are still struggling to achieve true observability despite substantial investments in tools.
The study showed that two-thirds of organisations currently spend US$100K or more annually on observability tools, with 38 per cent spending US$300K or more annually.
MLOps aids system observability
The One update is designed to help alleviate several pain points for data scientists, chief among them the changing nature of ML or AI models, as they depend on underlying data and code that may become irrelevant as real-world conditions change.
“The ML models deteriorate over the course of time," said Andy Thurai, research vice president and principal analyst at Constellation Research. "So you need model monitoring to measure the model performance, skew, staleness/freshness of the model, model recall, model precision, and model accuracy metrics. Depending on the application and usage, the models can change in a matter of seconds or can be valid for days/weeks/years in rare cases.”
The One update allows software engineers and data scientists to either import their own data or integrate with data science platforms, as well as monitor machine learning models and interdependencies along with other application components, including infrastructure, Fighel said.
Currently, New Relic supports data science platforms such as AWS SageMaker, DataRobot, Aporia, Superwise, Comet, DAGsHub, Mona and TruEra among others.
The company said that enterprises can create custom dashboards to track accuracy of machine learning models and generate alerts for unusual changes before they have an impact on the business or customers.
Observability to break data silos, speed devops
Another problem for enterprises deploying ML applications, according to New Relic’s Fighel, is how different teams across enterprises cannot work with each other efficiently because of disparate dashboards and separate interfaces.
“There is a major gap between the model producers, AKA data scientists, versus model implementors, AKA data engineering, and devops teams. By having tools like this, a model can be productionised easily,” Thurai said.
The One platform can help bring the teams together even if the enterprise has already invested in separate data science platforms, by providing a common interface that lets data scientists and other users import data from, and view models built on, different ML platforms, Fighel said.
This capability can also help to address vendor lock-ins, Fighel said. According to the LogDNA research report, more than half of professionals surveyed said that enterprises can’t implement the tools they want because of vendor lock-in.
Pricing and availability
The new machine learning monitoring capability, which is currently in general availability, is being offered for free on the One platform with a 100GB per month capping. However, Fighel said that the new system will soon follow a consumption pricing model.
Some of New Relic’s competitors include companies such as Sumo Logic, AppDynamics, Dynatrace, ManageEngine and Microsoft Azure Application Insights suite.
