Here at Immersive Labs, we love to get exploits and mitigations in the hands of our users as quickly as possible. So, when SaltStack announced that there were several high and critical-level CVEs on their system, we knew it was time for a one-day exploit party.
A zero-day occurs when a vulnerability is exploited before it’s made public. One-day exploits are developed on the day the patches are released, a common practice by both security teams and attackers. And that’s where this story really begins.
SaltStack announced the CVEs in January 2021, and confirmed the patches would be released on 4th February. We’d been expecting the release, as our vulnerable application developer Mat Rollings had identified one of the included vulnerabilities. Saltstack doesn’t normally release the proof of concept (PoC) code for each of their exploits, but as an open-source project, we would be able to access their changes.
We are a UK based team, so the following time frames are given in UTC. You’ll see why this is important soon enough…
Z-Day – 1 hour
25th February, 18:00
One hour to go before the release notes and patches are made public. At this point, we’re sat in a Google Meet room getting our test environments set up, which look something like this:
Gaz, Alex, Stefan and Harris in their natural habitats.
- Local Linux OS:
- Docker
- Docker Compose
- Custom Compose file to start Salt with all services and an attached minion
- Burp
- Python3 virtual-env with msgpack, zmq and salt-pepper
- Local version of the source code prior to release
- AWS:
- Ubuntu 20.04 base AMI configured as a master
- Ubuntu 20.04 base AMI configured as a minion
Z-Day – 10 minutes
18:50
We hit our first stumbling block when we refreshed the official Salt notification page:
The release had been pushed back by several hours. The patches wouldn’t be released until 1AM, assuming there were no further delays.
We had a tough choice to make. Should we call it a night and pick it up in the morning, or should we push on through, hoping there wouldn’t be another delay? Of course, we’re Immersers, so we decided we would wait it out.
Z-Day + 0 hours
19:00
As the vulnerabilities were already in the code, we could start reviewing it to try and spot anything ahead of the release. At the very least, we’d get familiar with the code base.
As these CVEs were rated critical, we expected they would be in the network layer. So we focused our efforts on components like the API. Fortunately for us, that’s where the most impactful vulns were located.
Z-Day + 6 hours
26th February, 01:00
1AM was fast approaching. No further announcements had been made, so we assumed everything was still on track.
We paused what we were doing and cleaned our environments ready to test. Docker is great; just a couple of CLI commands and within seconds we have a fully operational Death Star SaltStack instance.
Someone shouts, “Notes are up!” in the Meet room. It’s go time.
There was a lot on the release notes page. Out of the 10 CVEs, one of them was our own discovery, and three immediately piqued our interest.
- CVE-2021-25281: Auth bypass in the API to run commands on the Salt master
- CVE-2021-25282: Directory traversal on a file write function related to the auth bypass
- CVE-2021-25283: Template injection that lists code execution as an impact
The patch notes were up and the code was still being merged, but with these starter flags we knew where to start looking.
Z-Day + 7 hours
02:00
It didn’t take long before we had a simple yet functional auth bypass.
The exploit works by using the asynchronous client. It creates a job which is queued to be run. We only get the job ID returned, which has no indication of success or not. Fortunately, we had full access to the local builds, so we shelled in and took a look at the jobs ourselves.
Z-Day + 8 hours
03:00
By this time, we were all getting tired. We should have got here quicker, but we soon realised that if we chained the auth bypass with the directory traversal we could, in some situations, gain code execution over the master.
There are many well-known techniques to gain code execution if you have write access as root – your imagination is the limit! We we tested these methods first with great success:
- Write an SSH key to /root/.ssh/
- Write a cronjob to spawn a reverse shell
- Write a service script to spawn a reverse shell
We then made a very important discovery that altered the effectiveness of this attack.
When using our docker images, the API was running as a low privileged user, not root. This meant we were limited to where we could write files. The default Ubuntu installations did not have the same issue. When using a default installation and starting the ‘salt-api’ service, this always ran as root.
Z-Day + 10 hours
05:00
With a fully functional exploit, all that remained was to create the final PoC scripts and build the practical labs to go alongside them. We could do this later after a few hours of much-needed sleep!
Z-Day + 16 hours
11:00
Well-rested and fuelled by more cups of tea, we returned to our Google Meet room and started putting together the practical labs. This took longer than the actual discovery process as we built everything with a stricter setup to ensure the labs were reliable once deployed into the platform.
For the defensive variant of this lab, we’d already done most of the leg work when we looked at SaltStack last year.
The analysis this time round was even easier. This exploit was against the ‘wheel_async’ client, which created a job for every request sent to the API. This meant that every attempted exploit and the output of every command was stored.
After some initial testing, we handed the labs over to the QA team, who checked that all the practical images worked, verified the information was factual and relevant, and fixed all the typos.
Z-Day + 23 hours
18:00
The labs were published live.
Our offensive lab shows the user how to exploit the vulnerability from a red team perspective to gain remote access as root on a target Salt master. From here you can run code against every connected minion.
The defensive version of the lab gives you access to a compromised Salt master. You must investigate the artifacts left on the system and identify what the attackers have done.
Sharing the tools
With the labs published and the PoCs in place, we wanted to share the exploit code we used and the tool we created to read jobs. These resources can be found in our public GitHub repository.
https://github.com/Immersive-Labs-Sec/CVE-2021-25281
https://github.com/Immersive-Labs-Sec/SaltStackTools