Why Mitigations Matter for Analysts
Modern operating systems do not rely on programmers writing bug-free code. They deploy hardware and software mitigations that make exploitation harder even when a vulnerability exists.
Analysts need to understand these mitigations because:
- Malware that bypasses them uses specific, recognisable techniques (ROP chains, information leaks)
- Understanding the mitigation explains why certain malware behaviours exist
- Bypass attempts leave forensic artefacts that appear in dynamic analysis
ASLR: Address Space Layout Randomisation
ASLR randomises the base addresses of the executable image, stack, heap, and loaded DLLs on each process execution. An attacker who discovers a buffer overflow cannot hardcode a jump address because the target address changes on every run.
Entropy by architecture
| Architecture | Entropy (Stack) | Brute-force Feasibility |
|---|---|---|
| 32-bit Windows | 8-9 bits | Feasible in seconds to minutes |
| 64-bit Windows (HEASLR) | 17+ bits | Infeasible |
| 64-bit Linux (PIE) | 28+ bits | Infeasible |
32-bit ASLR is weak. Malware targeting 32-bit systems sometimes brute-forces ASLR by repeatedly crashing and restarting the target process until a predictable address is hit.
ASLR bypass techniques
| Technique | Mechanism |
|---|---|
| Information disclosure | Leak a pointer to calculate actual base address |
| Partial overwrite | Modify only the lower bytes (unaffected by ASLR randomisation) |
| Non-ASLR modules | Target DLLs compiled without /DYNAMICBASE flag |
| Brute force | Feasible on 32-bit systems with low entropy |
| Heap spray | Fill large memory region with shellcode: increases probability of hitting it |
What to look for
In dynamic analysis, a crash followed by a retry loop in the malware suggests ASLR brute-forcing. An information disclosure followed by a calculated jump target suggests a leak-based bypass.
DEP: Data Execution Prevention
DEP marks the stack and heap as non-executable. The CPU refuses to execute instructions fetched from pages without the Execute permission, enforced via the NX (AMD) or XD (Intel) bit in page table entries.
Even if an attacker injects shellcode into a data region, the CPU raises an access violation when attempting to execute bytes from a non-executable page.
DEP bypass: Return-Oriented Programming (ROP)
ROP chains together small snippets of existing executable code called gadgets (sequences ending in RET) that already exist in loaded modules.
Stack layout for a ROP chain:
┌──────────────────────────────┐
│ Address of gadget 1 │ ← return address (overwritten)
│ Address of gadget 2 │ ← popped by gadget 1's RET
│ Address of gadget 3 │ ← popped by gadget 2's RET
│ Address of VirtualProtect() │ ← called to make shellcode executable
│ Arguments to VirtualProtect │
└──────────────────────────────┘
No injected code executes: only existing code in executable regions. DEP is bypassed entirely.
ROP gadgets look like this in disassembly:
pop eax ; ← this is the gadget
ret ; ← RET chains to next gadget address on stack
Finding ROP gadgets in loaded DLLs is automated by tools like ROPgadget and mona.py.
ASLR + DEP together
DEP prevents code injection. ASLR prevents reliable gadget addressing. Defeating both requires:
- An information disclosure to find module base addresses (defeats ASLR)
- A ROP chain using gadgets from those modules (defeats DEP)
Malware that uses both bypass techniques is sophisticated and likely from a professional threat actor.
Stack Canaries
A random value (the canary) is placed between local buffers and the saved return address during function prologue. Before RET executes, the epilogue verifies the canary value. If corrupted, the process terminates immediately.
Stack layout with canary:
┌─────────────────────────┐
│ Return Address │ ← target of overflow
│ Saved EBP │
│ Stack canary (random) │ ← verified before RET
│ Local buffer │ ← overflow starts here
└─────────────────────────┘
An overflow must corrupt the canary before reaching the return address: triggering immediate termination.
Bypass
Requires knowing the canary value in advance via an information disclosure vulnerability: reading the canary value from memory before triggering the overflow, then including the correct value in the exploit payload.
Compiler flags
| Compiler | Flag | Notes |
|---|---|---|
| MSVC | /GS | Default since Visual Studio 2003 |
| GCC | -fstack-protector | Protects functions with arrays |
| GCC | -fstack-protector-strong | Protects more functions |
Summary
| Mitigation | Defeats | Bypass Technique |
|---|---|---|
| ASLR | Hardcoded addresses | Info leak, brute force (32-bit), non-ASLR modules |
| DEP/NX | Shellcode injection | ROP chain using existing executable code |
| Stack canary | Return address overwrite | Info leak to read canary value first |
ASLR and DEP are most effective together. Bypassing both requires an information leak and a ROP chain: significantly raising the bar for exploitation.