March 29, 2022
New zero-day vulnerability joins a chain of recently discovered vulnerabilities capable of operating an end-to-end attack on ESXi. Organizations should evaluate risk and apply vCenter client patches immediately.
Pentera Labs’ Senior Security Researcher, Yuval Lazar, discovered an Information Disclosure vulnerability impacting more than 500,000 appliances running default vCenter Server deployments.
This finding is critical given its potential global impact. According to VMware – more than 80 percent of virtualized workloads are running on VMware technology, including 100 percent of Fortune 500 and Fortune Global 100 companies. The ease and convenience that vCenter offers for managing virtualized hosts in enterprise environments provides cybercriminals with centralized access and the potential to inflict widespread damage on organizations.
In the full attack vector, threat actors can completely take over an organization’s ESXi’s deployed in a hybrid infrastructure and virtual machines hosted and managed by the hypervisor from just endpoint access to a host with a vCenter client.
VMware has issued a patch for the vulnerability that can be found here.
Installed in hundreds of thousands of organizations worldwide and used to manage some of their most critical asset and core systems, VMware vCenter Server is a high-priority target for cybercriminals. Once compromised, the ease and convenience that vCenter offers for managing virtualized hosts in enterprise environments will play into the adversary’s hands, providing centralized access and widespread damage.
Spurred by previously reported vulnerabilities, increasing demand from our customers, and threats observed in the wild, most notably recent reports of a python ransomware strain targeting ESXi, Pentera Labs has doubled down on exploring new vCenter attack vectors. The result of these explorations was the discovery of a new Information Disclosure vulnerability. The findings were proactively reported to VMware and later released under CVE-2022-22948.
The vulnerability described in this paper is part of a critical kill chain that leads to an ESXi takeover. In the full attack vector, which is a subject for a future article, we can see that initial endpoint access to a host with a vCenter client lends itself to a complete takeover of the organization’s ESXis. The devil and his merry band of demons are in the details of the chained vulnerabilities exploited as part of this attack vector.
From initial access to complete server takeover
Image 1 – Chaining multiple vulnerabilities for a complete view of cybersecurity risk
Pentera Labs firmly believes security readiness is not measured by a single vulnerability or the security team’s ability to discover or patch it. Rather, an organization’s security posture is determined by whether that vulnerability can be exploited directly or chained to preceding vulnerabilities across the complete attack operation.
In the case of VMware vCenter vulnerabilities, the adversary can take over the host where the vCenter application is running and all of its ESXi servers deployed in a hybrid infrastructure and virtual machines hosted and managed by the hypervisor.
Recent reports of ransomware gangs encrypting large amounts of virtual machines highlight the criticality of ransomware strains adding this vector to their arsenal of TTPs.
With our customers’ increased focus on virtualization infrastructure, Pentera Labs researchers set out on a mission to further enhance its existing attack frameworks used to identify, elevate, exploit and progress attack operations. As always, Pentera’s methods are to assume the attacker’s view and identify ways in which the adversary can take advantage of critical weaknesses – those known and those yet to be discovered.
During our research on the vCenter client, we located a file containing plaintext login credentials for the client’s postgresDB: “/etc/vmware-vpx/vcdb.properties”. This file is accessible to any of the users that are part of the “cis” group.
In other words, any user that is a member of the “cis” group can connect to the vCenter’s Postgres database.
Credentials to PostgresDB in vcdb.properties
Armed with the PostgresDB credentials, we proceeded to conduct an examination of the PostgresDB. We used the following command to connect to the DB, placing the credentials found in vcdb.properties to good use:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc -w <Password>
Here we discovered that it contains a wealth of information about the ESXi and vCenter, including information about the datacenters and virtual machines that are stored on them. For example:
VCDB query for information about virtual machines located on the ESXi
Another table is the table ‘vpx_host’ containing details for a user called ‘vpxuser’ and its password phrase. The vpx_host table holds a record for each managed ESXi, each containing a user called “vpxuser” and a unique password phrase.
So we retrieved the password phrase, using the command:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc -w <Password> -c 'SELECT user_name, password FROM vc.vpx_host;'
The vpxuser password phrase
Next, we dived in to investigate the “vpxuser”.
Turns out that “vpxuser” is a high-privileged user automatically created on the first connection between the ESXi to the vCenter.
The “vpxuser” user is created on the ESXi by default and is designed according to the principle of least-privilege, so it can be managed by the vCenter without the use of root. The user is created through a process called “vpxd” which is responsible for communication between the ESXi and the vCenter.
There isn’t a lot of information about this user, but we found a comment about it on the “passwd” file on the ESXi. The “passwd” file is usually used to keep track of users that have access to a system and it contains information about the user accounts. We were able to gather a lot of information about the user “vpxuser” from the file. Eventually, it was the description that led us to our next finding – “VMware VirtualCenter administration account”.
We started investigating the “vpxd” process using IDA in an effort to better understand how the “vpxuser” and its password are created and recorded in the PostgresDB that began this journey.
Our first step was to search for the string “VMware VirtualCenter administration account” using IDA’s Strings tab to see where else this string is referenced. This search led to the function related to the creation of the user on the ESXi. From there, we could identify the function where the password is created as the figure below shows.
Password creation function call
Diving into the “create_random_password” function, we identified the use of Random_Crypto (open-vm-tools), which suggests that the password for this user is randomly created.
Random_Crypto function in “create_random_password”
We knew that the password must be saved to the vCenter’s PostgresDB since it is randomly generated upon the first connection and is required for further communication. Yet, as we previously saw in the DB, the password is not saved in plaintext. We immediately noticed that the password phrase is encoded in base64, yet the encoded phrase was not readable, so we assumed it is also encrypted.
The vpxuser password phrase
We started exploring the processes the password goes through, as it is created, encrypted, encoded and finally, saved in the DB.
Eventually, we found the encryption function – only to discover that it uses OpenSSL Symmetric EVP common encryption method.
A call to the function that encrypts the password
Symmetric encryption uses the same secret key both to encrypt and decrypt the information. By using symmetric encryption algorithms, data is converted to a form that cannot be understood by anyone who does not possess the secret key.
The EVP functions provide a high level interface to OpenSSL cryptographic functions. In order to perform symmetric encryption or decryption using EVP you will need to know:
So now that we know what a symmetric encryption looks like, let’s go back and identify the password creation method. Since the IV is randomized, we deduced that it’s probably also saved to the DB. And it is! Once we found the function where the IV is created, this led the road to the decryption function!
The call to the OpenSSL decryption function
We were ready to proceed, having found all 3 prerequisites needed to decrypt a symmetric encryption:
There was still one missing piece to this puzzle. The “/etc/vmware-vpx/ssl/symkey.dat” file is only accessible to high-privileged users, so we needed a method that would give us root privileges to read it. Luckily, we had a method ready on stand-by thanks to our previous research and discovery of CVE-2021-22015, a privilege escalation vulnerability.
Once we extracted the key, we had all the ingredients necessary to decrypt the password. We wrote a python script to extract the cleartext password and got ourselves high-privilege credentials to ESXi!
To remediate CVE-2022-22948, apply the updates listed in VMware’s Advisory site. There is no known workaround.
I wanted to say thanks to everyone at VMware involved with the patch for this vulnerability. They were kind, prompt in their responses, and very easy to work with.
We were ready to verify that we had enough to decrypt the password. Using a script we created for this purpose, we entered the encoded-encrypted vpxuser password phrase and the key as arguments. This is the result we received:
Our Python script decrypted the password!
We used the vpxuser credentials to connect via SSH to the managed ESXi with high privileges and gained full control of the ESXi. We could control the ESXi as we wish – extract the memory of virtual machines, list inventory, get sensitive files, access sensitive information, you name it.
Ivanti Ground Zero On January 10, 2024, Ivanti disclosed two vulnerabilities, CVE-2023-46805 and CVE-2024-21887, impacting its Ivanti Connect Secure and Ivanti Policy Secure products in supported versions (9.x and 22.x). Successful exploitation can result in authentication bypass and command injection, leading to unauthenticated remote code execution and lateral movement inside the victim’s network. Then on […]
WebLogic is a popular enterprise middleware tool that orchestrates the interaction between backend systems and frontend clients. This makes it a valuable tool for attackers, who can exploit it to access and influence a wide range of organizational applications. In this blog post, we explore how to install a persistent backdoor on WebLogic Server. We […]
Today’s security leaders must manage a constantly evolving attack surface and a dynamic threat environment due to interconnected devices, cloud services, IoT technologies, and hybrid work environments. Adversaries are constantly introducing new attack techniques, and not all companies have internal Red Teams or unlimited security resources to stay on top of the latest threats. On […]