Immersive Labs recently discovered a security weakness in the AWS EC2 import process. When customer-created Windows virtual machines (VMs) are run through the Sysprep process and then uploaded to the AWS platform to create a reusable AMI or "template", answer files (also known as unattended setup files) were copied to the same directory and renamed by the AWS import process.
These additional files aren’t sanitized during the initial Sysprep first boot, which could leave cleartext credentials readable by all users who have access to the affected system. That is, if customers choose to use the answer file to set administrator or other user credentials on the affected template and don’t modify the file permissions of the answers file to prevent access by unprivileged users prior to initiating the sysprep process.
This weakness will be very rare to discover in the wild, as it requires a specific set of circumstances to be present. However, the Sysprep process is used to create templates from which many machines are deployed. If an organization has used the Sysprep process to set administrator or user credentials, a disproportionate number of Windows machines in its cloud-based estate could be affected. Given specific criteria, these credentials could be used to escalate privileges locally or move laterally to other machines deployed from the same template.
We initially reported this finding to AWS on 23rd February 2024. AWS accepted our bug report and reported that this condition was patched on 12th April 2024. We have confirmed that the answers file is no longer copied during VM import.
Pentesting lore
Since ancient times, pentesters and threat actors alike have scoured systems for credentials to further compromise and to move laterally or escalate privileges in a target network.
While seemingly one of the more benign and technically less sophisticated actions that can be used against a target, it’s one of the more effective – often requiring only a little effort and a high boredom threshold to be truly a master at the art of finding creds. This can also be harder to detect from a defender's perspective as it is usually only noticed by unusual processes accessing files.
As a pentester and red teamer by trade, I’ve lost count of the number of times that success was snatched from the jaws of defeat by simply finding cleartext credentials on a user’s desktop, in an open file share, stored in a browser, or any number of easy-to-reach places once I had initial access.
No matter how good an organization is at patching, if it doesn’t take steps to ensure good credential hygiene and privileged access management are correctly implemented, the fight can be over once an adversary gains access to the network. All it takes for the attacker is patience and the knowledge to know where to look for credentials.
One of those places is the answer files used by the Microsoft System Preparation Tool, otherwise known as Sysprep.
But what Is Sysprep?
As any Windows administrator will tell you, the Sysprep tool allows users to customize a Windows installation, then create an image ready to deploy across the estate. These images were once referred to as "gold" or "golden" images and were primarily created as a means to save time so new laptops, workstations, or servers could be deployed at speed and with the use of out-of-band (OOB) management technology (such as IPMI, iDrac, or vPro) performed remotely.
It’s true that this type of remote management has fallen out of style in favor of more modern solutions like Entra ID and InTune, but you can still find the artifacts of older deployment styles in target environments today.
So why might Windows systems need to be run through the Sysprep process? Why not just configure one system and take an image of that?
Again, as any Windows administrator will tell you, the Sysprep process will create a disk ready to be imaged that can be deployed on other hardware or hypervisors without unexpected issues or driver conflicts – like installing your own custom version of Windows from scratch.
Additionally, when you try to join a Windows server to a domain where the same (non-Sysprepped) image has been used as the domain controller, you’ll be greeted with the following error:
Lastly, the Sysprep process can be incredibly powerful in terms of customizing that image at boot time. All of these customizations can be pre-programmed with an answer file.
This blog post won’t cover all of the possible configuration options that’ll be of interest to administrative users, but it is important to call out that specific local users can be created or a default password set for the local administrator account, and other potentially sensitive information added like Multi-Activation License Keys (MAKs). Historically, it's easy to see why such a feature would be useful for remote Windows administrators, deploying a new system and then being able to connect remotely via RDP or another remote management protocol.
An opportunity
The offensive-minded would also see an opportunity here. Credentials that are stored in answer files (either in cleartext or Base64 encoded) are there for the taking – assuming that an attacker can access an answer file.
The earliest record I can find of a tool that would enumerate the contents of these files for credentials is in this pull request from the Metasploit framework in 2012, although this knowledge probably existed earlier elsewhere.
A typically useful characteristic of answer files is that if you can find them while pentesting, they’d be readable by all users, including standard or low-privilege users. This would lead to simple privilege escalation or lateral movement, assuming the same image had been deployed across the network.
Thankfully, when designing the Sysprep tool, Microsoft knew that if customers were going to use answer files to set user passwords, sensitive data should be removed from the world-readable file.
If you were to Sysprep a Windows machine today and set either the administrator user password or create a new user and set their password, when the newly deployed image boots for the first time, the Sysprep process creates those accounts, sets the passwords, and cleans the file, replacing sensitive information with the string *SENSITIVE*DATA*DELETED*:
Over time, the chances of finding an unattend.xml file or one of the other answer files with cleartext credentials have diminished for most pentest jobs. And if you do discover it, you’ll most likely find a sanitized file. But, if you do find one with credentials, it’s almost like finding MS08-067 or MS17-010 in a customer network…rare and yet trivial to exploit, and could well award the access required to reach the next objective.
Recently however, while making a custom Windows EC2 image for a Immersive Labs Cyber Ranges customer, I stumbled across a curious artifact in how AWS handles custom VM imports.
What does this have to do with AWS?
AWS allows its users to import custom-made VMs from outside of the AWS ecosystem. This functionality is especially useful when AWS isn’t able to supply specific operating systems from its marketplace or when using third-party-supplied AMIs isn’t desirable.
AWS gives some public guidance for this process, including how to prepare a Windows image for the import process as well as the import process itself to create a new AMI. What AWS doesn’t seem to provide publicly in any detail is how the import process manipulates the disk image during import and conversion to AMI (more on this shortly).
AWS’s guidance on creating the initial disk image says that disk images can be run through the Sysprep process prior to importing, rather than performing the Sysprep process once imported.
During testing, I found that performing the Sysprep process outside of AWS seemed to be preferential in terms of initial boot speeds once the AMI was created – and it can probably be expected that in other organizations that already have gold images created and stored in the required OVA format that they would also push an image that’s been through the initial Sysprep process.
When AWS imports the disk image, it adds additional software to that disk, including its EC2Launch application and the SSM Agent – all of which is done prior to the image’s first boot.
This is interesting as AWS is adding files and directories and likely altering the registry of the Windows image during this process without attaching the disk to a VM and starting it. These services are there to improve user experience and compatibility.
For example, when deploying a new AMI and specifying a larger disk size than the original image, the newly deployed AMI will resize the primary disk partition to occupy the assigned space.
But where this process becomes far more interesting is what happens to answer files like unattend.xml. The AWS import process created a duplicate of this file and copied it to the same directory, appending '.vmimport' to the filename.
Additionally, because this is performed before the first boot and the Microsoft Sysprep process has no knowledge of this file, it wasn’t modified to remove sensitive information.
The file was also readable by low-privilege users based on file permissions inherited from the directory, assuming that the original answers file permissions weren't modified to prevent unprivileged users from reading.
We’ve tested this across multiple versions of the Windows Server and Client, and the behavior from the AWS import process always remained the same. We can also confirm that this behavior appeared to be unique and specific to AWS and its VM import process only. We haven’t tested images in the third-party AMI marketplace, but encourage vendors using the ecosystem to check their deployed offerings if an imported Sysprepped image is used as the base.
Reporting to AWS and confirmation of fix
Immersive Labs reported this bug to AWS on 23rd February 2024, and indicated that conditions present in the VM import process, given certain conditions, could leave AWS customers affected by unnecessary risk. AWS asks for submitted vulnerabilities to include a CVSS 3.1 score, which we defined as 7.8 – High.
- Access vector: Local – An attacker requires local access to the affected VM to exploit it
- Attack complexity: Low – Credentials are in cleartext in a world-readable file
- Privileges required: Low – Any user on an affected system can read the file
- User interaction: None – User interaction isn’t required to perform file enumeration once access is achieved
- Scope: Unchanged – We view the scope as unchanged
- Confidentiality: High – Successful privilege escalation results in complete compromise of the system and can lead to complete loss of confidentiality
- Integrity: High – Successful privilege escalation results in complete compromise of the system and can lead to complete loss of integrity
- Availability: High – Successful privilege escalation results in complete compromise of the system and can lead to complete loss of availability
Take this score with a pinch of salt. This CVSS score is calculated in isolation and doesn’t make a clear distinction on the unique organization created conditions required for the bug to be present.
AWS accepted our report and confirmed that the condition was patched on 12th April 2024 meaning that the unattend.xml files are no longer copied during the VM import process. Immersive Labs has validated this claim and can confirm it to be true.
What does this mean for my organization?
To be upfront, this bug is very specific and is only likely to affect a very small number of AWS customers, given the conditions to create this weakness.
However, customers who have moved custom Windows VMs into AWS and have performed a Sysprep where credentials are hardcoded might be affected by this condition, and it could affect them across multiple hosts. Unfortunately, we don’t know how long this condition has existed and would have affected Windows images imported into AWS.
We also know that existing public pentesting tools don’t check for the presence of this copied file, so it may have been missed in historic assessments. The link to the AWS import process too, may not have been established where it has been discovered.
Customers who believe they may be affected should check their Windows estate in AWS for file names ending in '.vmimport' in directories associated with the Sysprep process and confirm that the contents don’t contain cleartext credentials that are pertinent or could be reused in their organization. These directories are:
- C:\
- C:\Windows\Panther\
- C:\Windows\Panther\Unattend\
- C:\Windows\system32\
- C:\Windows\system32\sysprep\
Customers should avoid using reused credentials across multiple administrative accounts wherever possible, instead employing a solution like Microsoft LAPS or another Privileged Access Management (PAM) solution.
Further advice can also be found from Microsoft in terms of implementing best practices when using answer files.
How did you find this?
Bug hunting takes time and luck. But a large part of it is having the knowledge, skills, and judgment to see the immediate issues presented by the data and what the second and third order effects could be, as well as the curiosity to test them.
This bug isn’t particularly complex or technical, but it could have significant connotations for affected organizations if not remediated.
Immersive Labs has a myriad of content across multiple cyber disciplines. Our core mission is not only to provide the knowledge to arm professionals against modern cyber threats in realistic, on-keyboard training labs but also to give teams the ability to find unique security issues like this in their own environments.
Want to know more? Book a demo today.