Researchers break Azure PostgreSQL database-as-a-service isolation with cross-tenant attack
- 29 April, 2022 08:00
A team of researchers found two vulnerabilities in Microsoft's Azure PostgreSQL Flexible Server that when chained together allowed them to access the PostgreSQL databases of other cloud tenants.
The attack, dubbed ExtraReplica because it abused functionality related to database replication, combines a privilege escalation vulnerability that gave them the ability to execute code inside the container hosting their own database and another authentication bypass issue that allowed them to abuse the system's replication service to access other users' databases.
The flaws were patched server-side, so customers don't need to do anything to secure their instances, but the researchers from cloud security firm Wiz who found the issues call on Microsoft and other cloud providers to provide better documentation on the isolation models and architecture of their services to allow customers to better evaluate the risks to their data when onboarding.
From database into the underlying system
The Azure Database for PostgreSQL is a managed database-as-a-service based on the open-source PostgreSQL relational database engine that has been in development for over 30 years.
PostgreSQL is a complex piece of software with many features that emphasises stability, high-availability and scalability. Microsoft offers the Azure Database for PostgreSQL service in three deployment variations: Single Server, Flexible Server and Hyperscale (Citus).
The Flexible Server allows customers to optimise their costs by choosing high availability inside a single Azure availability zone or across multiple availability zones or by stopping and starting their servers and burstable compute tier as needed.
The architecture separates the compute and storage with the database engine running inside a container in a Linux virtual machine and data files residing on Azure storage in three locally redundant synchronous copies.
One of the features of the PostgreSQL engine is to execute native system commands through database queries. The Wiz researchers signed up for the server and started executing queries inside their database to understand the environment.
They were able to determine that the database user the service provided did not have the necessary role to execute code, such as bash commands. This led them to look for previous research on potential ways to escalate privileges inside PostgreSQL and eventually found a bug in Microsoft's implementation.
"While researching our instance, we found that Azure modified their PostgreSQL engine," the researchers said in a blog post.
"It’s likely that Azure introduced these changes to the PostgreSQL engine to harden their privilege model and add new features. We managed to exploit a bug in those modifications to achieve privilege escalation, allowing us to execute arbitrary queries as a superuser. Obtaining superuser privileges allowed us to execute OS-level commands on our instance."
While this vulnerability was reported to Microsoft in January and was patched, the researchers don't want to release exploitation details yet out of concern that other vendors might have made similar modifications to their PostgreSQL engines and might have introduced the same bug.
Once they gained the ability to execute system commands, the researchers determined that they were inside a Docker container running Ubuntu 18.04.6 LTS with a recent kernel. They also looked at the network interfaces and noticed an interface that allowed connections from an internal IP netblock.
They then created another database in a different Azure account and tried to access it from their first account using port 5342 (PostgreSQL) using the internal network interface.
The connection worked even when the firewall was set to block all connections, which meant connections between the databases of different tenants were possible through the internal network, though this didn't mean much because they still lacked proper credentials to access the other database and read or write to it.
Certificate-based client authentication gone wrong
The researchers then wondered why this internal network connection across-tenants was allowed in the first place and decided to look at two files found on the machine called pg_hba.conf and pg_ident.conf. According to PostgreSQL documentation, these files are responsible for client authentication and username mappings.
The pg_hba.conf file revealed that a user called replication was allowed to connect through the internal network using certificate-based authentication. This user is part of Azure's database replication feature that allows creating backup copies of databases or replicating databases across servers.
The researchers then looked at pg_ident.conf and found a regular expression that was meant to validate the client certificate by looking at the CN (common name) for which it was issued. This is usually a domain or subdomain name. In this case, the database was configured to allow connections as the replication user if the client connection presented a certificate matching replication.eee03a2acfe6.database.azure.com.
The eee03a2acfe6 is an identifier unique to their particular database instance, so such a certificate would only provide access to their own database but obtaining such a certificate from a trusted certificate authority would be impossible because the researchers didn't own the azure.com to pass validation.
However, the researchers noticed a mistake in the regular expression because it ended with the (.*) string. This means it would match not only a certificate for replication.eee03a2acfe6.database.azure.com but also replication.eee03a2acfe6.database.azure.com.something-else.com.
Therefore, the researchers set out to obtain a certificate from DigiCert -- one of the public CAs trusted by the system -- matching the expected subdomain but on a domain they owned. This worked and allowed them to connect to their own database as the replication service, which has full reading permissions.
Since connections to other databases via the internal network were possible and impersonating the replication account and its privileges was possible, the only piece of missing information to access some other customer's database was finding the unique ID such as eee03a2acfe6 assigned to the target database and obtaining a certificate with a CN that matched it.
This wasn't hard at all because it was included in the target database server's certificate when connecting to it over SSL. In certificate-based client authentication both the client and the server present their respective certificates to each other to verify their identities and establish an encrypted connection.
Furthermore, since these are publicly trusted certificates issued by a certificate authority, they are published in certificate transparency logs, so this can also be used to find the unique identifiers associated with a target database if its custom Azure subdomain is known.
The attack only works against databases from the same region but finding the Azure availability region for a particular database can be determined easily by looking at the IP addresses of the servers hosting them. Then potential attackers would only have had to create an account in the same region.
Impact of ExtraReplica
The Azure PostgreSQL Single Server offerings were also affected by the first PostgreSQL privilege escalation vulnerability, but not by the cross-tenant authentication bypass using the replication service. Furthermore, the Flexible Server instances were not impacted if they were configured for Private Access (VNet Integration). VNet is Azure's virtual network functionality.
When first configuring their Azure PostgreSQL database account, users are required to choose their network connectivity preference between public access via allowed IP addresses, which is the default option, or private access via VNet. This depends on how they expect their applications to communicate with the database and it's hard to say how many users choose the VNet option.
According to the Wiz researchers, Microsoft didn't disclose the number of potentially impacted customers but said they were not aware of any attempts to exploit the vulnerabilities.
"Microsoft and other CSPs typically publish documentation on their current isolation models and architecture," the researchers said.
"However, we noticed that the PostgreSQL Flexible Server lacks public isolation documentation, making it difficult for customers to evaluate the risk when they onboard such a service. This issue is not unique to Azure alone, as other cloud providers tend to share isolation model for only a limited number of services."
Cloud providers should be more transparent with their isolation architectures and customers should ask their providers for such documentation before using such services, the researchers said.
They also noted that unlike software security flaws, vulnerabilities and misconfigurations in cloud services do not get CVE IDs assigned to them, which makes them harder to track or monitor. Because of this, there are volunteer community-driven efforts to create a database for cloud security issues and incidents.