Hanging CoreClr Debugging Events

georgevv
georgevv
edited August 19 in General

Hi,

I finally figured out why I had mysterious net5+ executables hang before my program even hit Main(). Where the only fix I had at the time was a computer reboot. It turns out linqpad8 8.10.4 (x64), or any other net5+ version of linqpad, has a bug where if you execute (F5) and then cancel (Shift + F5) for any reason before the script has finished executing naturally, i.e. you can get an exception and cancel it, put a breakpoint and cancel it. You will get a hanging windows event in \Sessions\1\BaseNamedObjects which is named TelestoStartupEvent_{PID in hex}

This event is something the core clr uses to make a program wait for a debugger to attach to it if a process has the same PID as the event.
https://github.com/dotnet/runtime/blob/main/src/coreclr/debug/ee/debugger.cpp#L1629

In essence if you use linqpad enough and execute/cancel enough times you will acquire dozens of telesto startup events with a PID that is no longer actively used by any process and is free to be recycled from a new process. Once a process that is in net5+ recycles a PID that has a telesto startup event, the coreclr is stuck before Main() indefinitely.

The workaround I have for now is to just close linqpad8 (kill any linqpad8 processes in task manager), which finally frees the telesto startup events and any future net5+ runs will be unaffected by the indefinite hang.

In order to see the telesto startup events you can get WinObj
https://learn.microsoft.com/en-us/sysinternals/downloads/winobj

and open it with Administrator privileges. You can then navigate to \Sessions\1\BaseNamedObjects and search for Telesto. You can see the acquisition of new events as you Execute/Cancel linqpad scripts and how they go away as you close linqpad.

I managed to reproduce the hanging executable by having a net8 executable with this code (any net5+ target framework will repro)

    using System;
    using System.Threading;

    public class Program
    {
        public static void Main(string[] args)
        {
            Random random = new Random();
            Thread.Sleep(random.Next(0, 1000));
            Console.WriteLine("I have successfully ran.");
        }
    }

Once you acquire a dozen or so telesto startup events by Execute/Cancel in linqpad, you can run the following powershell script to run the repro program executable.

1.. 1000 | %{ Start-Process -FilePath "D:\Path\To\The\Program.exe" -NoNewWindow }

This runs in about 30 seconds for me if there is no hang, if we hit a program.exe with the same PID as a telesto startup event, you can see that the executable hanging in task manager.

Once you dump it you can see the following stack trace which I acquired by making a dump of the hanging exe and opening it with WinDbg
https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/

[0x0]   ntdll!NtWaitForSingleObject+0x14   0xc61357ea68   0x7ffc17419d0e
[0x1]   KERNELBASE!WaitForSingleObjectEx+0x8e   0xc61357ea70   0x7ffb4851e3ee
[0x2]   coreclr!NotifyDebuggerOfStartup+0xdd33d   (Inline Function)   (Inline Function)
[0x3]   coreclr!Debugger::Startup+0xdd35e   0xc61357eb10   0x7ffb4849339f
[0x4]   coreclr!InitializeDebugger+0x93   0xc61357ebe0   0x7ffb4843b8f8
[0x5]   coreclr!EEStartupHelper+0x7c4   0xc61357ec10   0x7ffb4848c9b9
[0x6]   coreclr!EEStartup+0x2d   0xc61357ecf0   0x7ffb4848c955
[0x7]   coreclr!EnsureEEStarted+0x95   0xc61357ed30   0x7ffb4848c88b
[0x8]   coreclr!CorHost2::Start+0x5b   0xc61357ed70   0x7ffb4847ae29
[0x9]   coreclr!coreclr_initialize+0x179   0xc61357edb0   0x7ffb5c383931
[0xa]   hostpolicy!coreclr_t::create+0x2b1   0xc61357ee80   0x7ffb5c3a04d5
[0xb]   hostpolicy!`anonymous namespace'::create_coreclr+0x165   0xc61357f000   0x7ffb5c3a27cf
[0xc]   hostpolicy!corehost_main+0x15f   0xc61357f060   0x7ffb38e8dffb
[0xd]   hostfxr!execute_app+0x2bb   0xc61357f160   0x7ffb38e908bc
[0xe]   hostfxr!`anonymous namespace'::read_config_and_execute+0xac   0xc61357f1f0   0x7ffb38e92c96
[0xf]   hostfxr!fx_muxer_t::handle_exec_host_command+0x1d6   0xc61357f2f0   0x7ffb38e90dbd
[0x10]   hostfxr!fx_muxer_t::execute+0x2ad   0xc61357f3a0   0x7ffb38e884bf
[0x11]   hostfxr!hostfxr_main_bundle_startupinfo+0x16f   0xc61357f4d0   0x7ff7e708f65d
[0x12]   Program!exe_start+0x63d   0xc61357f5e0   0x7ff7e708fc96
[0x13]   Program!wmain+0x146   0xc61357f790   0x7ff7e70911b8
[0x14]   Program!invoke_main+0x22   (Inline Function)   (Inline Function)
[0x15]   Program!__scrt_common_main_seh+0x10c   0xc61357f800   0x7ffc18bd259d
[0x16]   kernel32!BaseThreadInitThunk+0x1d   0xc61357f840   0x7ffc19e2afb8
[0x17]   ntdll!RtlUserThreadStart+0x28   0xc61357f870   0x0

I should have also clarified, this only happens when you have the linqpad debugger attached. If you do not have the debugger attached, it will never fire up a telesto event.

So if you run with #LINQPad optimize+ you will never create new telesto events.

I guess the actual question here is: Is there anything that can be done on Linqpad side to fix this and correctly handle the hanging TelestoStartupEvents? I am also mostly posting here as a FYI as this has been plaguing me for the last few years and never even imagined I would pin this down to linqpad.

Happy to help with more details & guidance on the repro!

· Share on Facebook

Categories