Unpacking with a Debugger

The Goal: Reach the OEP

The unpacking stub must restore the original code in memory before transferring control to it. The point where control transfers from the stub to the original code is the Original Entry Point (OEP). The analyst''s goal is to reach the OEP, dump the process memory, and reconstruct the import table.

Packed file on disk → stub executes → decompresses payload in memory
                                                            ↓
                                               OEP reached → dump + fix IAT
                                                            ↓
                                               Analysable unpacked PE

The ESP Hardware Breakpoint Technique

Many common packers (UPX, ASPack, PECompact) begin with a PUSHAD instruction that saves all general-purpose registers to the stack. After the unpacking loop, a POPAD restores them. The transition from POPAD to the OEP jump is called the tail jump.

Why this works

PUSHAD writes all eight registers onto the stack in one operation, decrementing ESP by 32 bytes. After unpacking completes, POPAD reads those same bytes back. If we set a hardware breakpoint on the memory address where PUSHAD stored the values, the breakpoint fires exactly when POPAD accesses them: pausing execution just before the tail jump.

Execution flow:
  PUSHAD     → saves registers, ESP decrements 32 bytes
  [unpacking loop runs — decompresses payload]
  POPAD      → reads from ESP location → HARDWARE BREAKPOINT FIRES HERE
  JMP [OEP]  → tail jump to original code

Step-by-step procedure in x32dbg

1. Open packed sample in x32dbg
2. Run to entry point (F9 — stops at first instruction)
3. First instruction is PUSHAD — confirm this
4. Step over PUSHAD with F8
5. Note the current ESP value in the Registers panel
6. Right-click the ESP value in the Dump panel
   → Breakpoint → Hardware, Access → DWORD
7. Press F9 to run
8. Debugger pauses — POPAD is accessing the breakpoint address
9. Look at the next instruction after POPAD
   → There will be a JMP or RET to a distant address (the tail jump)
10. Step into the tail jump with F7
11. You are now at the OEP — the first instruction of the original code

Recognising the OEP

The OEP of a normally compiled Windows executable starts with the standard compiler prologue:

; MSVC-compiled EXE entry point (typical)
PUSH    EBP
MOV     EBP, ESP
SUB     ESP, some_value

; Or for a DLL:
PUSH    EBP
MOV     EBP, ESP
PUSH    ECX

If after the tail jump you see a function prologue pattern, you are at the correct OEP. If you see garbage bytes or a second jump, the packer has additional stages: repeat the process.

When PUSHAD/POPAD Is Not Used

Some packers do not use PUSHAD. Alternative breakpoint strategies:

Strategy	When to Use
VirtualAlloc breakpoint	When the packer allocates memory for the unpacked code: break when allocation returns and set memory execute breakpoint on that region
VirtualProtect breakpoint	When the packer marks the unpacked region as executable: fires just before OEP transfer
Memory execute breakpoint on .text section	Set on the first byte of the (currently empty) .text region: fires when the stub writes and then executes code there
Entropy-guided stepping	Run until entropy in memory drops: indicates decompression complete

Dumping Memory with Scylla

Once paused at the OEP, the unpacked code exists in memory but the on-disk file is still packed. Scylla is the standard tool for creating an analysable PE:

Scylla workflow in x32dbg

1. While paused at OEP in x32dbg
2. Open Scylla: Plugins → Scylla
3. Verify OEP address is correct in the OEP field
4. Click [IAT Autosearch] → Scylla locates the import table
5. Click [Get Imports] → Scylla resolves all imported function names
6. Review the imports list — remove any invalid entries (right-click → delete)
7. Click [Dump] → saves process memory to disk as unpacked_sample.exe
8. Click [Fix Dump] → patches the IAT into the dumped file
9. Result: standalone unpacked PE ready for static analysis

Verifying the dump

# Open dumped file in Detect-It-Easy
die unpacked_sample.exe
# Should now show compiler (e.g., "MSVC (19.xx)") instead of packer name

# Check entropy
pestudio unpacked_sample.exe
# .text section entropy should be 5.0–6.5, not 7+

# Check imports
# IAT should now contain full import list, not just LoadLibraryA

Common Unpacking Failures and Fixes

Problem	Cause	Fix
Scylla IAT Autosearch finds nothing	Wrong OEP: still in stub	Step further into the code; look for the function prologue pattern
Many invalid imports in Scylla	Partial unpack: some modules not yet loaded	Run further after OEP until all imports resolve, then dump
Dumped file crashes immediately	Stolen bytes: first instructions at OEP are missing	Manually restore the stolen instructions (covered in Topic 6)
DIE still shows packer after dump	Multi-stage packer: a second packer wraps the first	Repeat the ESP breakpoint technique on the newly dumped file

Anti-Dump Defences

Advanced packers add defences specifically against memory dumping:

• PE header erasure: zeros the first 0x1000 bytes after unpacking, destroying the DOS and PE headers
• Guard pages: marks unpacked memory pages as guard pages; any read by a dumper triggers an exception
• IAT destruction: corrupts the import table after resolution to prevent reconstruction

These advanced techniques are covered in Unit 6 (Self-Defending Malware).