Both attackers and researchers often use msfvenom to generate shellcode. It allows them to quickly create code that can perform different actions on a host, which helps their exploits endlessly.But it can come with a problem.You’ve probably come across this scene before: After successfully finding a location to drop your shellcode in memory, you redirect the flow of the program to execute the beginning of your shellcode with execute permissions on the shellcode region so the code will run. In addition, a debugger is attached to follow the flow, and the shellcode starts to execute!Then you run into something like this:https://www.immersivelabs.com/wp-content/uploads/2023/02/image1-400x297.jpgWinDbg displays an access violation where an xor operation occurs on the memory located at (that’s 00000017 because eax is 00000000). Ultimately, the shellcode fails to execute fully, which results in a remote shell not landing on the target.https://www.immersivelabs.com/wp-content/uploads/2023/02/image2-400x266.jpg Shows fnstenv successfully executing. Shows the pop eax instruction placing 00000000 into eax. This is what gets used in the xor later. Shows the xor causing an access violation by operating on 00000000 + 17h.This can be a frustrating issue to encounter since a vulnerability has been located, exploited, and used trampolines or custom assembly instructions to direct the control flow to the shellcode. Then the generated shellcode fails before it even begins.Reverse engineers, exploit devs, and researchers often get frustrated by fnstenv failing because it’s not easy to debug or understand why the failure is happening.
What is fnstenv?
fstenv and fnstenv are two very similar assembly instructions. Fstenv stands for FPU Store Environment, an instruction that stores the state of the x87 FPU information in memory, generally used by exception handlers, system processes, and sometimes application processes. The difference between the two instructions is that fstenv does error checking before saving the state and fnstenv doesn’t, which is why your shellcode uses this instruction.
So why use it in shellcode?
fnstenv gives shellcode a way to understand where it’s located inside memory. This instruction is a popular way for the shikata encoder to decode itself. It needs to know where it resides in memory to locate the bytes that it decodes into the instructions, which gives you the reverse shell you’re after. It’s all part of the GetPC routine that’s been around for years.Before we move on to the issues, let’s look at the fnstenv instruction in more depth. When executed, the state of the FPU environment that’s stored looks like the following (taken from the intel manual):https://www.immersivelabs.com/wp-content/uploads/2023/02/image3-400x195.jpgThis shows the different pieces of information saved as part of the fnstenv instruction. Notice how the offset 12 (0xC) holds the FPU Instruction Pointer Offset. This is the location of the last executed FPU instruction. In theory, if the shellcode executes an FPU instruction (like fldz, fnsave, etc.), this would initialize the FPU environment and cause the FIP to be maintained. Then, running fnstenv will place the FIP offset to the top of the stack. Finally, the shellcode simply places that into a register like eax and the shellcode knows where it resides in memory.https://www.immersivelabs.com/wp-content/uploads/2023/02/fldz-400x127.png
Why does fnstenv fail?
Issue #1: Execution alignmentThe Fnstenv instruction should fall to an EIP location aligned to a multiple of four. If it’s not, the instruction can fail. Look at the image below:https://www.immersivelabs.com/wp-content/uploads/2023/02/image4-400x123.jpgThe red box shows that the instruction is executing at offset 0x018d5b69. A good way to fix this is to remove one of the no-operation instructions (NOPs) before the shellcode. This will land the instruction nicely on an offset that it prefers.
Issue #2: Bad characters
There can also be bad characters (badchars) in the shellcode’s location. Often when exploiting a program, the shellcode won’t reside straight after EIP takeover and will have to be searched using an egghunter. Sometimes, execution will need to be bounced around the process memory before successfully finding the shellcode location.In this case, there is potential for two sets of bad characters – the initial set of bad characters identified for the overflow to occur and any transformation of data that occurs when your shellcode is copied to another location (such as the heap). Both need to be looked at and analyzed for these changed characters.So how do you check badchars in a certain location? One of the best ways is to place an egg hunting unique character, such as “w00tw00t,” before the suspected badchars. This way, after sending your exploit to the program and breaking at the pivot instruction, you can manually search memory for the location of that string, knowing whatever follows will be the exact location of the shellcode.Below is the replacement of the shellcode with an updated badchars list with \x00 \x0a and \x20 taken out because they were identified in the initial set of bad character filtering.https://www.immersivelabs.com/wp-content/uploads/2023/02/without90s-400x513.pngNow, the shellcode with the badchars would look like this:https://www.immersivelabs.com/wp-content/uploads/2023/02/badchars_final-400x308.pngRun the program, attach your WinDbg, set your breakpoint at the pivot point, and search for the “w00tw00t” egg you placed before the bad chars.It should look something like this:https://www.immersivelabs.com/wp-content/uploads/2023/02/windbg1-400x604.pngThe above output of WinDbg shows a breakpoint being hit at a pivot point in the code. Then using the command s -a 0 L?80000000 “w00tw00t” you can search through the ASCII text in memory to match the egg. There’s only one place in memory that holds the shellcode!Add eight bytes to the memory location found, so you don’t start inspecting hex characters of “w00tw00t.” To view it better, look at the memory window and start scanning through the characters to see if any of them have changed – you might be surprised to see that some of them have been mangled when they were written to memory.https://www.immersivelabs.com/wp-content/uploads/2023/02/image7-400x376.jpgFrom inspecting the memory, you can see that \x8d and \xd3 have been mangled and changed to \x22.
Issue #3: No NOPs
One of the most discussed possible reasons for fnstenv failing is that there are no NOPs before the shellcode, so the shellcode starts decoding and fails. For this reason, it’s usually a good idea to have several NOPs leading up to your shellcode. This gives it some space to start decoding itself correctly.The image below shows the shellcode re-added to the exploit Python script with a number of \x90s – which is hex for the NOP instruction.https://www.immersivelabs.com/wp-content/uploads/2023/02/shellcode_final-400x510.png
Issue #4: ESP location
The next issue can be split into two parts; both are related to the ESP register. The stack pointer needs to point to a writable location and not too close to EIP.First, the writable part. It’s rare for the stack pointer not to point to a writeable place unless ESP has been mangled and points to an odd location. The other explanation is that it points to the stack base, which is defined in the threat environment block (TEB). This can sometimes happen when performing things like the SEH overflow, and you’ve had to overflow past nearly all the ExceptionList. To solve this, have some shellcode simply subtract a large value from ESP so that when the FPU environment is initialized and fnstenv places something on the stack, it has some space to do so.https://www.immersivelabs.com/wp-content/uploads/2023/02/windbg2-400x542.pngThe WinDbg output above shows that ESP is placed right by the StackBase, so anything placed on the stack will pass the StackBase location shown in the TEB.The other reason bundled into this issue could be that EIP and ESP are too close together, which means the decoder will start placing things on the stack that overwrite instructions needed for the shellcode to decode and function properly. As with the writable issue, you can solve this by simply adding in shellcode that places ESP elsewhere, so your shellcode has free reign to push and pop things off the stack.
Issue #5: You’re using a debugger
If none of these fixes work and you’re still seeing fnstenv fail, it may be because you’re stepping through with a debugger, which can cause an issue with fnstenv’s execution.So try this a few times without the debugger attached, and also with the debugger attached but not stepping through the individual instructions. Hopefully, you’ll get this stubborn instruction to work.If you’re working on a virtualized machine that abstracts you from the underlying PC, the instruction may not be implemented correctly, which means it will always fail. This has been seen many times with a number of pieces of software, such as QEMU.
Conclusion
It’s common to see the fnstenv instruction fail when using Msfvenom-generated shellcode. This can be because of misaligned execution, bad characters, a lack of NOPs, the ESP location, or because you’re using a debugger. Next time you run into this problem, try these fixes.Immersive Labs has a range of reverse engineering content, covering many aspects of exploit development in Windows and Linux.Click here to learn more about our extensive content library.