New zero-day vulnerability joins a chain of recently discovered vulnerabilities capable of operating an end-to-end attack on ESXi. Organizations should evaluate risk and apply vCenter client patches immediately.
Pentera Labs’ Senior Security Researcher, Yuval Lazar, discovered an Information Disclosure vulnerability impacting more than 500,000 appliances running default vCenter Server deployments.
This finding is critical given its potential global impact. According to VMware – more than 80 percent of virtualized workloads are running on VMware technology, including 100 percent of Fortune 500 and Fortune Global 100 companies. The ease and convenience that vCenter offers for managing virtualized hosts in enterprise environments provides cybercriminals with centralized access and the potential to inflict widespread damage on organizations.
In the full attack vector, threat actors can completely take over an organization’s ESXi’s deployed in a hybrid infrastructure and virtual machines hosted and managed by the hypervisor from just endpoint access to a host with a vCenter client.
VMware has issued a patch for the vulnerability that can be found here.
Installed in hundreds of thousands of organizations worldwide and used to manage some of their most critical asset and core systems, VMware vCenter Server is a high-priority target for cybercriminals. Once compromised, the ease and convenience that vCenter offers for managing virtualized hosts in enterprise environments will play into the adversary’s hands, providing centralized access and widespread damage.
Spurred by previously reported vulnerabilities, increasing demand from our customers, and threats observed in the wild, most notably recent reports of a python ransomware strain targeting ESXi, Pentera Labs has doubled down on exploring new vCenter attack vectors. The result of these explorations was the discovery of a new Information Disclosure vulnerability. The findings were proactively reported to VMware and later released under CVE-2022-22948.
Complete Attack Vector
The vulnerability described in this paper is part of a critical kill chain that leads to an ESXi takeover. In the full attack vector, which is a subject for a future article, we can see that initial endpoint access to a host with a vCenter client lends itself to a complete takeover of the organization’s ESXis. The devil and his merry band of demons are in the details of the chained vulnerabilities exploited as part of this attack vector.
Complete Attack Operation
From initial access to complete server takeover
Image 1 – Chaining multiple vulnerabilities for a complete view of cybersecurity risk
Pentera Labs firmly believes security readiness is not measured by a single vulnerability or the security team’s ability to discover or patch it. Rather, an organization’s security posture is determined by whether that vulnerability can be exploited directly or chained to preceding vulnerabilities across the complete attack operation.
In the case of VMware vCenter vulnerabilities, the adversary can take over the host where the vCenter application is running and all of its ESXi servers deployed in a hybrid infrastructure and virtual machines hosted and managed by the hypervisor.
Recent reports of ransomware gangs encrypting large amounts of virtual machines highlight the criticality of ransomware strains adding this vector to their arsenal of TTPs.
With our customers’ increased focus on virtualization infrastructure, Pentera Labs researchers set out on a mission to further enhance its existing attack frameworks used to identify, elevate, exploit and progress attack operations. As always, Pentera’s methods are to assume the attacker’s view and identify ways in which the adversary can take advantage of critical weaknesses – those known and those yet to be discovered.
The Credentials Disclosure
During our research on the vCenter client, we located a file containing plaintext login credentials for the client’s postgresDB: “/etc/vmware-vpx/vcdb.properties”. This file is accessible to any of the users that are part of the “cis” group.
In other words, any user that is a member of the “cis” group can connect to the vCenter’s Postgres database.
Credentials to PostgresDB in vcdb.properties
Exploring the vCenter internal DB
Armed with the PostgresDB credentials, we proceeded to conduct an examination of the PostgresDB. We used the following command to connect to the DB, placing the credentials found in vcdb.properties to good use:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc -w <Password>
Here we discovered that it contains a wealth of information about the ESXi and vCenter, including information about the datacenters and virtual machines that are stored on them. For example:
VCDB query for information about virtual machines located on the ESXi
Another table is the table ‘vpx_host’ containing details for a user called ‘vpxuser’ and its password phrase. The vpx_host table holds a record for each managed ESXi, each containing a user called “vpxuser” and a unique password phrase.
So we retrieved the password phrase, using the command:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc -w <Password> -c 'SELECT user_name, password FROM vc.vpx_host;'
The vpxuser password phrase
Next, we dived in to investigate the “vpxuser”.
vpxuser – who are you?
Turns out that “vpxuser” is a high-privileged user automatically created on the first connection between the ESXi to the vCenter.
The “vpxuser” user is created on the ESXi by default and is designed according to the principle of least-privilege, so it can be managed by the vCenter without the use of root. The user is created through a process called “vpxd” which is responsible for communication between the ESXi and the vCenter.
There isn’t a lot of information about this user, but we found a comment about it on the “passwd” file on the ESXi. The “passwd” file is usually used to keep track of users that have access to a system and it contains information about the user accounts. We were able to gather a lot of information about the user “vpxuser” from the file. Eventually, it was the description that led us to our next finding – “VMware VirtualCenter administration account”.
We started investigating the “vpxd” process using IDA in an effort to better understand how the “vpxuser” and its password are created and recorded in the PostgresDB that began this journey.
Our first step was to search for the string “VMware VirtualCenter administration account” using IDA’s Strings tab to see where else this string is referenced. This search led to the function related to the creation of the user on the ESXi. From there, we could identify the function where the password is created as the figure below shows.
Password creation function call
Diving into the “create_random_password” function, we identified the use of Random_Crypto (open-vm-tools), which suggests that the password for this user is randomly created.
Random_Crypto function in “create_random_password”
We knew that the password must be saved to the vCenter’s PostgresDB since it is randomly generated upon the first connection and is required for further communication. Yet, as we previously saw in the DB, the password is not saved in plaintext. We immediately noticed that the password phrase is encoded in base64, yet the encoded phrase was not readable, so we assumed it is also encrypted.
The vpxuser password phrase
Decrypting the vpxuser password
We started exploring the processes the password goes through, as it is created, encrypted, encoded and finally, saved in the DB.
Eventually, we found the encryption function – only to discover that it uses OpenSSL Symmetric EVP common encryption method.
A call to the function that encrypts the password
OpenSSL Symmetric Encryption EVP
Symmetric encryption uses the same secret key both to encrypt and decrypt the information. By using symmetric encryption algorithms, data is converted to a form that cannot be understood by anyone who does not possess the secret key.
The EVP functions provide a high level interface to OpenSSL cryptographic functions. In order to perform symmetric encryption or decryption using EVP you will need to know:
- The algorithm – There are several types of encryption algorithms that can be used, such as block or stream algorithms. As we later discovered, AES_256_CBC is the algorithm used here.
- The key – You need to possess the secret key to decrypt the message.
- The Initialization Vector (IV) – An Initialization Vector is a random number used in combination with a secret key as a means to encrypt data. This number is sometimes referred to as a nonce. An IV is used to avoid repetition during the data encryption process, making it impossible for hackers who use dictionary attacks to decrypt the exchanged encrypted message by discovering a pattern.
So now that we know what a symmetric encryption looks like, let’s go back and identify the password creation method. Since the IV is randomized, we deduced that it’s probably also saved to the DB. And it is! Once we found the function where the IV is created, this led the road to the decryption function!
The call to the OpenSSL decryption function
We were ready to proceed, having found all 3 prerequisites needed to decrypt a symmetric encryption:
- The algorithm – We could just tell that the algorithm is AES_256_CBC.
- The IV – We realized that the first 16 bytes of the password phrase are the IV.
- The key – We found that the encryption key is created in advance and saved to the file “/etc/vmware-vpx/ssl/symkey.dat”.
There was still one missing piece to this puzzle. The “/etc/vmware-vpx/ssl/symkey.dat” file is only accessible to high-privileged users, so we needed a method that would give us root privileges to read it. Luckily, we had a method ready on stand-by thanks to our previous research and discovery of CVE-2021-22015, a privilege escalation vulnerability.
Once we extracted the key, we had all the ingredients necessary to decrypt the password. We wrote a python script to extract the cleartext password and got ourselves high-privilege credentials to ESXi!
- Logged into the vCenter’s PostgresDB.
- Found the encoded and encrypted password for vpxuser, a high-privileged user automatically created on the first connection between the ESXi to the vCenter.
- Reverse-engineered the creation of vpxuser to devise a method for decrypting the vpxuser password.
- Leveraged vScalation (CVE-2021-22015), a Privilege Escalation method also discovered by Pentera Labs to gain root privileges necessary to decrypt the vpxuser password.
- Once decrypted, the compromised root account vpxuser confirms complete takeover of the ESXi server and a new zero-day is born.
To remediate CVE-2022-22948, apply the updates listed in VMware’s Advisory site. There is no known workaround.
I wanted to say thanks to everyone at VMware involved with the patch for this vulnerability. They were kind, prompt in their responses, and very easy to work with.
Proof of Concept
We were ready to verify that we had enough to decrypt the password. Using a script we created for this purpose, we entered the encoded-encrypted vpxuser password phrase and the key as arguments. This is the result we received:
Our Python script decrypted the password!
We used the vpxuser credentials to connect via SSH to the managed ESXi with high privileges and gained full control of the ESXi. We could control the ESXi as we wish – extract the memory of virtual machines, list inventory, get sensitive files, access sensitive information, you name it.
How we improved our QA with Shift-Left testing
This article is part of Pentera’s Engineering Series – a behind-the-scenes look at the technologies we develop to keep companies secure. In this piece, we look at the testing processes that we use to QA our platform and deliver a high-quality solution. It almost goes without saying that testing is a critical part of the...
Five steps to mitigate the risk of credential exposure
Every year, billions of credentials appear online, be it on the dark web, clear web, paste sites, or in data dumps shared by cybercriminals. These credentials are often used for account takeover attacks, exposing organizations to breaches, ransomware, and data theft. While CISOs are aware of growing identity threats and have multiple tools in their...
WiFi – The Untested Attack Surface
Much of a company’s assets are connected to Wi-Fi networks. However, security teams are often less likely to validate these networks. This pushed us to wonder what we might find if we were to test a corporate WiFi network. After running the Pentera platform™️ over Wi-Fi, we found several vulnerabilities, which helped us gain insight...