Ever had the trial version of an application run just fine, but fail horribly when you register a license key? I’ve had this happen with two different and completely unrelated software products – FLStudio (awesome music authoring application) and FinalBuilder (a NAnt/MSBuild replacement). In both cases, the cause was a Data Execution Prevention (DEP) violation. (DEP is on by default in 64-bit versions of Windows, which is why it is most often encountered there. On 32-bit Windows, DEP is enabled only for critical operating system files, though it is simple to turn it on for all files.) I suspect that the licensing scheme generates code from the license key, which is then executed. The code will be generated in a data region, which is marked with the NX (No eXecute) flag on modern processors. When a modern processor tries to execute code in a portion of memory marked with the NX flag, DEP kicks in and tears down the process. The rationale behind this is that you’re not supposed to be executing data! (Apologies to the LISP folks out there.) If you need to dynamically generate code on the heap, clear the NX flag to mark it as executable.
Now you may be wondering about the .NET Framework because the CLR takes MSIL and JITs it to native code, which lives on the unmanaged heap. The CLR clears the NX flag for the regions of the heap where JIT’d methods live. The same applies to JVMs in the Java world and any other JIT’d language.
The Fix
If you’re an end user, there is no (easy) way to mark the generated code as executable. (I suppose you could hook a debugger up to the process every time you launch the application and manually clear the NX flag in the correct region. Thanks, but no.) So the solution is to exempt the executable in question from data execution prevention (DEP). On Vista, go to Computer… Properties… Advanced system settings… Performance Settings… Data Execution Prevention tab… and add the EXE to the exception list. (You should find the DEP tab in a similar place on Longhorn Server, Windows XP SP2, and Windows Server 2003.) The process doesn’t have DEP protection, but at least you can run it.
The Real Fix
If you actually wrote the software, you can really fix the issue. You must clear the NX bit in the regions where you generate code to prevent DEP from tearing down your process. Rather than using malloc or HeapAlloc, you must use VirtualAlloc or VirtualProtect with one of the following flags: PAGE_EXECUTE, PAGE_EXECUTE_READ, PAGE_EXECUTE_READWRITE, or PAGE_EXECUTE_WRITECOPY.
Why Malware Can’t Sidestep DEP
Now you may think that this opens a security hole. Can’t malware simply clear the NX flag itself? Let’s walk through the hypothetical exploit:
- You have an unexploited application with an undiscovered buffer overflow bug.
- Your application gets a malformed request attempting to exploit the bug.
- The buffer is overflown and return address of the current function is set to somewhere in the buffer. (This is how a typical buffer overflow exploit occurs.)
- The processor tries to execute the data in the buffer, which would contain the attack code. The buffer is marked NX because it is part of the current thread’s stack (or less often, the heap).
- DEP kicks in and tears down the process.
At no time has the attacker’s code been given a chance to run. The only way that the attacker’s code could run and call VirtualAlloc/Protect is if your code called VirtualAlloc/Protect on the buffer, which would be a pretty silly thing to do. So if malware can call VirtualAlloc/Protect, it can already run code in your process and do much worse things than dynamically generate code and mark it as executable.
Hopefully this helps some folks solve mysterious licensing failures and software authors make their code work on DEP-enabled systems, which are being more and more common.
JetBrains is looking for feedback on 
