Doing any kind of research into the Windows kernel requires working with a kernel debugger, mostly WinDbg (or WinDbg Preview). There are at least 3 “levels” of debugging the kernel.
Level 1: Local Kernel Debugging
The first is using a local kernel debugger, which means configuring WinDbg to look at the kernel of the local machine. This can be configured by running the following command in an elevated command window, and restarting the system:
bcdedit -debug on
You must disable Secure Boot (if enabled) for this command to work, as Secure Boot protects against putting the machine in local kernel debugging mode. Once the system is restarted, WinDbg launched elevated, select File/Kernel Debug and go with the “Local” option (WinDbg Preview shown):
If all goes well, you’ll see the “lkd>” prompt appearing, confirming you’re in local kernel debugging mode.
What can you in this mode? You can look at anything in kernel and user space, such as listing the currently existing processes (!process 0 0), or examining any memory location in kernel or user space. You can even change kernel memory if you so desire, but be careful, any “bad” change may crash your system.
The downside of local kernel debugging is that the system is a moving target, things change while you’re typing commands, so you don’t want to look at things that change quickly. Additionally, you cannot set any breakpoint; you cannot view any CPU registers, since these are changing constantly, and are on a CPU-basis anyway.
The upside of local kernel debugging is convenience – setting it up is very easy, and you can still get a lot of information with this mode.
Level 2: Remote Debugging of a Virtual Machine
The next level is a full kernel debugging experience of a virtual machine, which can be running locally on your host machine, or perhaps on another host somewhere. Setting this up is more involved. First, the target VM must be set up to allow kernel debugging and set the “interface” to the host debugger. Windows supports several interfaces, but for a VM the best to use is network (supported on Windows 8 and later).
First, go to the VM and ping the host to find out its IP address. Then type the following:
bcdedit /dbgsettings net hostip:172.17.32.1 port:55000 key:1.2.3.4
Replace the host IP with the correct address, and select an unused port on the host. The key can be left out, in which case the command will generate something for you. Since that key is needed on the host side, it’s easier to select something simple. If the target VM is not local, you might prefer to let the command generate a random key and use that.
Next, launch WinDbg elevated on the host, and attach to the kernel using the “Net” option, specifying the correct port and key:
Restart the target, and it should connect early in its boot process:
Microsoft (R) Windows Debugger Version 10.0.25200.1003 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Using NET for debugging
Opened WinSock 2.0
Waiting to reconnect...
Connected to target 172.29.184.23 on port 55000 on local IP 172.29.176.1.
You can get the target MAC address by running .kdtargetmac command.
Connected to Windows 10 25309 x64 target at (Tue Mar 7 11:38:18.626 2023 (UTC - 5:00)), ptr64 TRUE
Kernel Debugger connection established. (Initial Breakpoint requested)
************* Path validation summary **************
Response Time (ms) Location
Deferred SRV*d:\Symbols*https://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*d:\Symbols*https://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 10 Kernel Version 25309 MP (1 procs) Free x64
Edition build lab: 25309.1000.amd64fre.rs_prerelease.230224-1334
Machine Name:
Kernel base = 0xfffff801`38600000 PsLoadedModuleList = 0xfffff801`39413d70
System Uptime: 0 days 0:00:00.382
nt!DebugService2+0x5:
fffff801`38a18655 cc int 3
Enter the g command to let the system continue. The prompt is “kd>” with the current CPU number on the left. You can break at any point into the target by clicking the “Break” toolbar button in the debugger. Then you can set up breakpoints, for whatever you’re researching. For example:
1: kd> bp nt!ntWriteFile
1: kd> g
Breakpoint 0 hit
nt!NtWriteFile:
fffff801`38dccf60 4c8bdc mov r11,rsp
2: kd> k
# Child-SP RetAddr Call Site
00 fffffa03`baa17428 fffff801`38a81b05 nt!NtWriteFile
01 fffffa03`baa17430 00007ff9`1184f994 nt!KiSystemServiceCopyEnd+0x25
02 00000095`c2a7f668 00007ff9`0ec89268 0x00007ff9`1184f994
03 00000095`c2a7f670 0000024b`ffffffff 0x00007ff9`0ec89268
04 00000095`c2a7f678 00000095`c2a7f680 0x0000024b`ffffffff
05 00000095`c2a7f680 0000024b`00000001 0x00000095`c2a7f680
06 00000095`c2a7f688 00000000`000001a8 0x0000024b`00000001
07 00000095`c2a7f690 00000095`c2a7f738 0x1a8
08 00000095`c2a7f698 0000024b`af215dc0 0x00000095`c2a7f738
09 00000095`c2a7f6a0 0000024b`0000002c 0x0000024b`af215dc0
0a 00000095`c2a7f6a8 00000095`c2a7f700 0x0000024b`0000002c
0b 00000095`c2a7f6b0 00000000`00000000 0x00000095`c2a7f700
2: kd> .reload /user
Loading User Symbols
.....................
2: kd> k
# Child-SP RetAddr Call Site
00 fffffa03`baa17428 fffff801`38a81b05 nt!NtWriteFile
01 fffffa03`baa17430 00007ff9`1184f994 nt!KiSystemServiceCopyEnd+0x25
02 00000095`c2a7f668 00007ff9`0ec89268 ntdll!NtWriteFile+0x14
03 00000095`c2a7f670 00007ff9`08458dda KERNELBASE!WriteFile+0x108
04 00000095`c2a7f6e0 00007ff9`084591e6 icsvc!ICTransport::PerformIoOperation+0x13e
05 00000095`c2a7f7b0 00007ff9`08457848 icsvc!ICTransport::Write+0x26
06 00000095`c2a7f800 00007ff9`08452ea3 icsvc!ICEndpoint::MsgTransactRespond+0x1f8
07 00000095`c2a7f8b0 00007ff9`08452abc icsvc!ICTimeSyncReferenceMsgHandler+0x3cb
08 00000095`c2a7faf0 00007ff9`084572cf icsvc!ICTimeSyncMsgHandler+0x3c
09 00000095`c2a7fb20 00007ff9`08457044 icsvc!ICEndpoint::HandleMsg+0x11b
0a 00000095`c2a7fbb0 00007ff9`084574c1 icsvc!ICEndpoint::DispatchBuffer+0x174
0b 00000095`c2a7fc60 00007ff9`08457149 icsvc!ICEndpoint::MsgDispatch+0x91
0c 00000095`c2a7fcd0 00007ff9`0f0344eb icsvc!ICEndpoint::DispatchThreadFunc+0x9
0d 00000095`c2a7fd00 00007ff9`0f54292d ucrtbase!thread_start<unsigned int (__cdecl*)(void *),1>+0x3b
0e 00000095`c2a7fd30 00007ff9`117fef48 KERNEL32!BaseThreadInitThunk+0x1d
0f 00000095`c2a7fd60 00000000`00000000 ntdll!RtlUserThreadStart+0x28
2: kd> !process -1 0
PROCESS ffffc706a12df080
SessionId: 0 Cid: 0828 Peb: 95c27a1000 ParentCid: 044c
DirBase: 1c57f1000 ObjectTable: ffffa50dfb92c880 HandleCount: 123.
Image: svchost.exe
In this “level” of debugging you have full control of the system. When in a breakpoint, nothing is moving. You can view register values, call stacks, etc., without anything changing “under your feet”. This seems perfect, so do we really need another level?
Some aspects of a typical kernel might not show up when debugging a VM. For example, looking at the list of interrupt service routines (ISRs) with the !idt command on my Hyper-V VM shows something like the following (truncated):
2: kd> !idt
Dumping IDT: ffffdd8179e5f000
00: fffff80138a79800 nt!KiDivideErrorFault
01: fffff80138a79b40 nt!KiDebugTrapOrFault Stack = 0xFFFFDD8179E95000
02: fffff80138a7a140 nt!KiNmiInterrupt Stack = 0xFFFFDD8179E8D000
03: fffff80138a7a6c0 nt!KiBreakpointTrap
...
2e: fffff80138a80e40 nt!KiSystemService
2f: fffff80138a75750 nt!KiDpcInterrupt
30: fffff80138a733c0 nt!KiHvInterrupt
31: fffff80138a73720 nt!KiVmbusInterrupt0
32: fffff80138a73a80 nt!KiVmbusInterrupt1
33: fffff80138a73de0 nt!KiVmbusInterrupt2
34: fffff80138a74140 nt!KiVmbusInterrupt3
35: fffff80138a71d88 nt!HalpInterruptCmciService (KINTERRUPT ffffc70697f23900)
36: fffff80138a71d90 nt!HalpInterruptCmciService (KINTERRUPT ffffc70697f23a20)
b0: fffff80138a72160 ACPI!ACPIInterruptServiceRoutine (KINTERRUPT ffffdd817a1ecdc0)
...
Some things are missing, such as the keyboard interrupt handler. This is due to certain things handled “internally” as the VM is “enlightened”, meaning it “knows” it’s a VM. Normally, it’s a good thing – you get nice support for copy/paste between the VM and the host, seamless mouse and keyboard interaction, etc. But it does mean it’s not the same as another physical machine.
Level 3: Remote debugging of a physical machine
In this final level, you’re debugging a physical machine, which provides the most “authentic” experience. Setting this up is the trickiest. Full description of how to set it up is described in the debugger documentation. In general, it’s similar to the previous case, but network debugging might not work for you depending on the network card type your target and host machines have.
If network debugging is not supported because of the limited list of network cards supported, your best bet is USB debugging using a dedicated USB cable that you must purchase. The instructions to set up USB debugging are provided in the docs, but it may require some trial and error to locate the USB ports that support debugging (not all do). Once you have that set up, you’ll use the “USB” tab in the kernel attachment dialog on the host. Once connected, you can set breakpoints in ISRs that may not exist on a VM:
: kd> !idt
Dumping IDT: fffff8022f5b1000
00: fffff80233236100 nt!KiDivideErrorFault
...
80: fffff8023322cd70 i8042prt!I8042KeyboardInterruptService (KINTERRUPT ffffd102109c0500)
...
Dumping Secondary IDT: ffffe5815fa0e000
01b0:hidi2c!OnInterruptIsr (KMDF) (KINTERRUPT ffffd10212e6edc0)
0: kd> bp i8042prt!I8042KeyboardInterruptService
0: kd> g
Breakpoint 0 hit
i8042prt!I8042KeyboardInterruptService:
fffff802`6dd42100 4889542410 mov qword ptr [rsp+10h],rdx
0: kd> k
# Child-SP RetAddr Call Site
00 fffff802`2f5cdf48 fffff802`331453cb i8042prt!I8042KeyboardInterruptService
01 fffff802`2f5cdf50 fffff802`3322b25f nt!KiCallInterruptServiceRoutine+0x16b
02 fffff802`2f5cdf90 fffff802`3322b527 nt!KiInterruptSubDispatch+0x11f
03 fffff802`2f5be9f0 fffff802`3322e13a nt!KiInterruptDispatch+0x37
04 fffff802`2f5beb80 00000000`00000000 nt!KiIdleLoop+0x5a
Happy debugging!
Pavel Yosifovich 