Epilog detection unreliable when ip is on jmp with an 0x58 immediate before it

[ishitatsuyuki]

This is a very rare case of epilog analysis false-positive I found when checking epilog detection robustness for the WIP SEH backend, but I thought it was worth reporting.

The sample landed on an in-function jump:

0x35911b800      488b4258       mov rax, qword [rdx + 0x58] ; 0x58 looks like a pop but is actually an offset
0x35911b804      eba9           jmp 0x35911b7af             ; sample ip here

The function starts at 0x35911b6c0 and ends at 0x35911ba8c.

It looks like the current heuristic can break if you stress it with millions of samples. :P

framehop/src/x86_64/instruction_analysis/epilogue.rs

Lines 33 to 43 in 1517df9

	// This must be the first iteration. Look backwards.
	if let Some(potential_pop_byte) = slice_from_start.last() {
	// Get the previous byte. We have no idea how long the previous instruction
	// is, so we might be looking at a random last byte of a wider instruction.
	// Let's just pray that this is not the case.
	if potential_pop_byte & 0xf8 == 0x58 {
	// Assuming we haven't just misinterpreted the last byte of a wider
	// instruction, this is a `pop rXX`.
	break;
	}
	}

Wine uses something different: It checks if the jump lands within function bounds.

https://github.com/wine-mirror/wine/blob/d0ce5a77c60e0a9613f1af03d67d13b89816441b/dlls/ntdll/signal_x86_64.c#L720

This could be potentially more robust, but needs testing.

[ishitatsuyuki]

After analyzing more cases, I've realized that the jump target heuristic can't work for indirect jumps. (In general, the indirect jump identification is much harder, and there doesn't seem to be a reliable way to tell between indirect tail-calls and indirect switch-table jumps.) So the pop byte heuristic needs to stay at least for those cases.

[mstange]

I'm definitely in favor of adding a check for "is within function bounds" for the case where the target address is known.