Infinite recursion with unhandled managed exceptions and exception marshaling is set to throw #14796
Description
If managed exception marshalling is configured to throw an Objective-C exception (and vice versa for Objective-C exceptions), then we can run into an infinite loop in some cases when leaking managed exceptions to native code.
Example: the FinishedLaunching method throws an exception, and we have this stack trace:
(lldb) bt
* thread #1, name = 'tid_103', queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
* frame #0: 0x0000000100ca1c8d libxamarin-dotnet-debug.dylib`xamarin_process_managed_exception(exception=0x0000000100dc3910) at runtime.m:2278:11
frame #1: 0x0000000100caeac2 libxamarin-dotnet-debug.dylib`xamarin_invoke_trampoline(type=Tramp_Default, self=0x00006000026c0ff0, sel="application:didFinishLaunchingWithOptions:", iterator=(libxamarin-dotnet-debug.dylib`param_iter_next(IteratorAction, void*, char const*, unsigned long, void*, void**) at trampolines-x86_64.m:180), marshal_return_value=(libxamarin-dotnet-debug.dylib`marshal_return_value(void*, char const*, unsigned long, void*, _MonoType*, bool, _MonoMethod*, MethodDescription*, void**) at trampolines-x86_64.m:251), context=0x00007ff7bfceac50) at trampolines-invoke.m:778:3
frame #2: 0x0000000100cb6139 libxamarin-dotnet-debug.dylib`xamarin_arch_trampoline(state=0x00007ff7bfceac80) at trampolines-x86_64.m:491:2
frame #3: 0x0000000100cb731e libxamarin-dotnet-debug.dylib`xamarin_x86_64_common_trampoline at trampolines-x86_64-asm.s:51
frame #4: 0x00007fff2508a847 UIKitCore`-[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 214
frame #5: 0x00007fff2508c538 UIKitCore`-[UIApplication _callInitializationDelegatesWithActions:forCanvas:payload:fromOriginatingProcess:] + 4128
frame #6: 0x00007fff25091ed8 UIKitCore`-[UIApplication _runWithMainScene:transitionContext:completion:] + 1190
frame #7: 0x00007fff24602a92 UIKitCore`-[_UISceneLifecycleMultiplexer completeApplicationLaunchWithFBSScene:transitionContext:] + 122
frame #8: 0x00007fff24bf6ba6 UIKitCore`_UIScenePerformActionsWithLifecycleActionMask + 88
frame #9: 0x00007fff2460358c UIKitCore`__101-[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:]_block_invoke + 198
frame #10: 0x00007fff24602fc9 UIKitCore`-[_UISceneLifecycleMultiplexer _performBlock:withApplicationOfDeactivationReasons:fromReasons:] + 252
frame #11: 0x00007fff246033c2 UIKitCore`-[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:] + 831
frame #12: 0x00007fff24602c6f UIKitCore`-[_UISceneLifecycleMultiplexer uiScene:transitionedFromState:withTransitionContext:] + 354
frame #13: 0x00007fff2460d0b7 UIKitCore`__186-[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:]_block_invoke + 178
frame #14: 0x00007fff24ada115 UIKitCore`+[BSAnimationSettings(UIKit) tryAnimatingWithSettings:actions:completion:] + 859
frame #15: 0x00007fff24c1402d UIKitCore`_UISceneSettingsDiffActionPerformChangesWithTransitionContext + 246
frame #16: 0x00007fff2460cd3b UIKitCore`-[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:] + 346
frame #17: 0x00007fff2440a465 UIKitCore`__64-[UIScene scene:didUpdateWithDiff:transitionContext:completion:]_block_invoke.578 + 796
frame #18: 0x00007fff24408e37 UIKitCore`-[UIScene _emitSceneSettingsUpdateResponseForCompletion:afterSceneUpdateWork:] + 253
frame #19: 0x00007fff2440a025 UIKitCore`-[UIScene scene:didUpdateWithDiff:transitionContext:completion:] + 255
frame #20: 0x00007fff250907f9 UIKitCore`-[UIApplication workspace:didCreateScene:withTransitionContext:completion:] + 513
frame #21: 0x00007fff24b0b112 UIKitCore`-[UIApplicationSceneClientAgent scene:didInitializeWithEvent:completion:] + 355
frame #22: 0x00007fff26546cdd FrontBoardServices`-[FBSScene _callOutQueue_agent_didCreateWithTransitionContext:completion:] + 415
frame #23: 0x00007fff26573216 FrontBoardServices`__94-[FBSWorkspaceScenesClient createWithSceneID:groupID:parameters:transitionContext:completion:]_block_invoke.180 + 102
frame #24: 0x00007fff265550ef FrontBoardServices`-[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:] + 209
frame #25: 0x00007fff26572df5 FrontBoardServices`__94-[FBSWorkspaceScenesClient createWithSceneID:groupID:parameters:transitionContext:completion:]_block_invoke + 352
frame #26: 0x00007fff20115b25 libdispatch.dylib`_dispatch_client_callout + 8
frame #27: 0x00007fff20118c9f libdispatch.dylib`_dispatch_block_invoke_direct + 281
frame #28: 0x00007fff26599da3 FrontBoardServices`__FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ + 30
frame #29: 0x00007fff26599c99 FrontBoardServices`-[FBSSerialQueue _targetQueue_performNextIfPossible] + 174
frame #30: 0x00007fff26599dcb FrontBoardServices`-[FBSSerialQueue _performNextFromRunLoopSource] + 19
frame #31: 0x00007fff20373833 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
frame #32: 0x00007fff2037372b CoreFoundation`__CFRunLoopDoSource0 + 180
frame #33: 0x00007fff20372bf8 CoreFoundation`__CFRunLoopDoSources0 + 242
frame #34: 0x00007fff2036d2f4 CoreFoundation`__CFRunLoopRun + 871
frame #35: 0x00007fff2036ca90 CoreFoundation`CFRunLoopRunSpecific + 562
frame #36: 0x00007fff2cb72c8e GraphicsServices`GSEventRunModal + 139
frame #37: 0x00007fff2508e90e UIKitCore`-[UIApplication _run] + 928
frame #38: 0x00007fff25093569 UIKitCore`UIApplicationMain + 101
frame #39: 0x0000000105c60941
frame #40: 0x0000000105c5dc83
frame #41: 0x0000000101376fff libmonosgen-2.0.dylib`mono_jit_runtime_invoke + 2223
frame #42: 0x0000000101299318 libmonosgen-2.0.dylib`mono_runtime_invoke_checked + 136
frame #43: 0x00000001012a00bb libmonosgen-2.0.dylib`mono_runtime_exec_main_checked + 107
frame #44: 0x00000001013d90b2 libmonosgen-2.0.dylib`mono_jit_exec + 354
frame #45: 0x0000000100cb5f8d libxamarin-dotnet-debug.dylib`xamarin_main(argc=1, argv=0x00007ff7bfced648, launch_mode=XamarinLaunchModeApp) at monotouch-main.m:494:8
frame #46: 0x00000001002afc94 HelloiOS`main(argc=1, argv=0x00007ff7bfced648) at main.x86_64.mm:65:11
frame #47: 0x0000000100896f21 dyld_sim`start_sim + 10
frame #48: 0x0000000105c9c51e dyld`start + 462
The xamarin_process_managed_exception method will convert the managed exception to an Objective-C exception, and throw it.
The native _dispatch_client_callout function seems to catch all Objective-C exceptions and directly call objc_terminate:
(lldb) disass -n _dispatch_client_callout
libdispatch.dylib`_dispatch_client_callout:
0x7fff20115b1d <+0>: pushq %rbp
0x7fff20115b1e <+1>: movq %rsp, %rbp
0x7fff20115b21 <+4>: pushq %rbx
0x7fff20115b22 <+5>: pushq %rax
0x7fff20115b23 <+6>: callq *%rsi
0x7fff20115b25 <+8>: addq $0x8, %rsp
0x7fff20115b29 <+12>: popq %rbx
0x7fff20115b2a <+13>: popq %rbp
0x7fff20115b2b <+14>: retq
0x7fff20115b2c <+15>: movq %rax, %rdi
0x7fff20115b2f <+18>: callq 0x7fff2014a0ac ; symbol stub for: objc_begin_catch
0x7fff20115b34 <+23>: callq 0x7fff2014a0dc ; symbol stub for: objc_terminate
0x7fff20115b39 <+28>: ud2
0x7fff20115b3b <+30>: movq %rax, %rbx
0x7fff20115b3e <+33>: callq 0x7fff2014a0b8 ; symbol stub for: objc_end_catch
0x7fff20115b43 <+38>: movq %rbx, %rdi
0x7fff20115b46 <+41>: callq 0x7fff20149e54 ; symbol stub for: _Unwind_Resume
0x7fff20115b4b <+46>: callq 0x7fff2014a0dc ; symbol stub for: objc_terminate
This means that control will never return to the managed Main function, because the OS will intercept the Objective-C app and effectively try to terminate the app.
However, we install a handler for uncaught Objective-C exceptions (using NSSetUncaughtExceptionHandler). In this handler we'll try to convert the Objective-C exception to a managed exception, and throw that:
frame #15003: 0x00000001065f8aa0 libmonosgen-2.0.dylib`mono_amd64_throw_exception + 160
frame #15004: 0x0000000105b275b0
frame #15005: 0x0000000105e236f5 libxamarin-dotnet-debug.dylib`exception_handler(exc=0x00006000005f0f60) at runtime.m:1123:2
frame #15006: 0x00007fff2040707d CoreFoundation`__handleUncaughtException + 719
frame #15007: 0x00007fff20189d6e libobjc.A.dylib`_objc_terminate() + 90
frame #15008: 0x00007fff2025a9c7 libc++abi.dylib`std::__terminate(void (*)()) + 8
frame #15009: 0x00007fff2025a978 libc++abi.dylib`std::terminate() + 56
frame #15010: 0x00007fff201a4be4 libobjc.A.dylib`objc_terminate + 9
frame #15011: 0x00007fff20115b39 libdispatch.dylib`_dispatch_client_callout + 28
And then we'll convert this managed exception to an Objective-C exception, and throw that:
frame #14997: 0x00007fff201a4b68 libobjc.A.dylib`objc_exception_throw + 307
frame #14998: 0x0000000105e2300f libxamarin-dotnet-debug.dylib`xamarin_process_managed_exception(exception=0x0000000000000000) at runtime.m:2391:3
frame #14999: 0x0000000105e22c4b libxamarin-dotnet-debug.dylib`xamarin_process_managed_exception_gchandle(gchandle=0x0000000000001c36) at runtime.m:1099:2
frame #15000: 0x0000000105e22c05 libxamarin-dotnet-debug.dylib`xamarin_ftnptr_exception_handler(gchandle=0x0000000000001c36) at runtime.m:1087:2
frame #15001: 0x000000010659dc84 libmonosgen-2.0.dylib`mono_handle_exception_internal + 5556
frame #15002: 0x000000010659c6c2 libmonosgen-2.0.dylib`mono_handle_exception + 18
frame #15003: 0x00000001065f8aa0 libmonosgen-2.0.dylib`mono_amd64_throw_exception + 160
which will again call our uncaught Objective-C exception handler:
frame #14992: 0x00007fff2040707d CoreFoundation`__handleUncaughtException + 719
frame #14993: 0x00007fff20189d6e libobjc.A.dylib`_objc_terminate() + 90
frame #14994: 0x00007fff2025a9c7 libc++abi.dylib`std::__terminate(void (*)()) + 8
frame #14995: 0x00007fff2025d01e libc++abi.dylib`__cxxabiv1::failed_throw(__cxxabiv1::__cxa_exception*) + 27
frame #14996: 0x00007fff2025cfe5 libc++abi.dylib`__cxa_throw + 116
frame #14997: 0x00007fff201a4b68 libobjc.A.dylib`objc_exception_throw + 307
ad infinitum...
The actual bug is that the FinishedLaunching method shouldn't leak managed exceptions to native code. Any managed exceptions leaked to native code has the potential of this occurring. On the other hand, the resulting stackoverflow due to the infinite recursion makes these issues harder to diagnose, so we should figure out a way to improve the diagnostics at the very least.