Support for kdump/kexec on WSL2

[grahamm78]

I am trying to set up kernel core dumps (preferably kdump but vmcore is OK) on WSL2 4.19.128-microsoft-standard. I'm wanting to use this for testing, and debugging (I'm working on some kernel panic / dump parsers) and was hoping I could use WSL2 to generate kdumps via sysrq or driver causing an OOPS. I can install kdump-tools and kernel-kdump but I cannot get it all configured, such as setting crashkernel kernel variabl

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
Unclear if there are workarounds outside of not using WSL2 for kdump. For sysrq issue that can be resolved by building a driver to trigger a panic.

Additional context

kdump-config show

• kdump is not supported by this kernel
• no crashkernel= parameter in the kernel cmdline
  DUMP_MODE: kdump
  USE_KDUMP: 1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR: /var/crash
  crashkernel addr:
  /var/lib/kdump/vmlinuz
  kdump initrd:
  /var/lib/kdump/initrd.img
  cat: /sys/kernel/kexec_crash_loaded: No such file or directory
  /usr/sbin/kdump-config: line 155: [: too many arguments
  current state: Not ready to kdump

[WSLUser]

You need to enable the relevant config option and add it to .wslconfig

[DoctorWho8]

Where would I as a curious user, go to change that? Suppose, mind you suppose, I had an interest in using either that feature, or the kexec one?

[WSLUser]

So I don't know which kernel option enables these but you can create a .wslconfig with notepad or some other text editor and place it at C:/Users/user. You can find examples on what to put for config options here in the github. The release notes have the template (or you can just use a search engine, plenty of blogs about it).

[DoctorWho8]

I guess that will work. I'm also interested in kexec as well.

[WSLUser]

https://www.kernel.org/doc/html/v5.8/admin-guide/kdump/kdump.html should do the trick. Per the doc, kdump uses kexec to execute. Now we both know the kernel option needed. I would recommend rebuilding a new kernel with the options you want enabled (there are a number of them there to choose) and specifying it in .wslconfig.

[grahamm78]

Thanks, I did not know about .wslconfig I will check that out. I will search to learn some more, such as if the file goes in the root of your profile does that mean one config for all installed instances. Cheers, Graham

[DoctorWho8]

@WSLUser so you're saying that I should create configure in my user name file, that contains the appropriate terms in it? And of course it would be called .wslconfig? Interesting. @grahamm78 what's your take on this?

[WSLUser]

#4694 (comment) Here's an example for you and comes straight from one of the MS devs.

[grahamm78]

I've gotten a lot further now! I've still got a few pieces to figure out though.

I built 2 kernels with updated .config based on the 4.19.128-microsoft-standard kernel. I followed some steps herewhich were helpful on building the kernel, then went back to the kdump documentation to set config options

1. Build the kernels

System Kernel

cp Microsoft/wsl-config .config-syskernel

CONFIG_LOCALVERSION="grahamm_syskernel" # to easily tell it's loaded via uname -a 
CONFIG_KEXEC=y   # this is there but commented by default
CONFIG_DEBUG_INFO=y     # wasn't in there at all, perhaps superceded by CONFIG_KALLSYMS kdump doc may be old?
CONFIG_NET_9P_VIRTIO=y # based on reading as my host systems /mnt/c etc were not appearing

cp .config-syskernel .config
make (take some defaults on prompts for config options)
cp arch/x86_64/boot/bzImage /mnt/c/Windows/System32/lxss/tools/syskernel

Dump-capture kernel
cp Microsoft/wsl-config .config-dumpkernel

CONFIG_LOCALVERSION="grahamm_dumpkernel"
CONFIG_CRASH_DUMP=y
CONFIG_SMP=n
CONFIG_NET_9P_VIRTIO=y # based on reading as my host systems /mnt/c etc were not appearing

cp .config-dumpkernel .config
make (take some defaults on prompts for config options)
cp arch/x86_64/boot/bzImage /mnt/c/Windows/System32/lxss/tools/dumpkernel

2. wsl --shutdown

3. Created a %userprofile%.wslconfig file with custom kernel, crashkernel and some other options.

[wsl2]
kernel=c:\\Windows\\system32\\lxss\\tools\\syskernel
kernelcommandline= crashkernel=0M-2G:128M,2G-6G:256M,6G-8G:512M,8G-:768M, panic=30, oops=panic, panic_print=ff

Now I launched wsl and I saw the custom kernel and kernelcommandline take effect.

4. set up kexec to point to the new dumpkernel. I could not figure out some of the parameters like root device (I could not find the root device name for current OS) so I skipped some parameters as a test.

root@GRAHAMM-Z2:/dev# kexec -p /mnt/c/Windows/system32/lxss/tools/dumpkernel
root@GRAHAMM-Z2:/dev# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x18000000
/var/lib/kdump/vmlinuz
kdump initrd:
/var/lib/kdump/initrd.img
current state: ready to kdump

root@GRAHAMM-Z2:/dev# cat /sys/kernel/kexec_crash_loaded
1

But - I tried to test it with kdump-config test and it dies because KDUMP_KERNEL doesn't exist.
root@GRAHAMM-Z2:/dev# kdump-config test

• /etc/default/kdump-tools: KDUMP_KERNEL does not exist: /var/lib/kdump/vmlinuz

so I need to find the places to fix the rest of the configuration settings

• /etc/default/kdump-tools: KDUMP_KERNEL does not exist: /var/lib/kdump/vmlinuz
  USE_KDUMP: 1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR: /var/crash
  crashkernel addr: 0x18000000
  kdump kernel addr:
  kdump kernel:
  /var/lib/kdump/vmlinuz
  kdump initrd:
  /var/lib/kdump/initrd.img
  kexec command to be used:
  /sbin/kexec -p --command-line="initrd=\initrd.img panic=-1 nr_cpus=12 swiotlb=force pty.legacy_count=0 panic=30, oops=panic, panic_print=ff reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0" /var/lib/kdump/vmlinuz

EDIT

I found the config file nano /usr/share/kdump-tools/kdump-tools.conf
and also just copied my kernel to /var/lib/kdump/vmlinuz but I still have a problem with initrd. How do I get a ramdisk for my kernel?

root@GRAHAMM-Z2:/mnt/c/Users/grahamm# kdump-config test

• Invalid symlink : /var/lib/kdump/initrd.img
• Invalid symlink : /var/lib/kdump/vmlinuz
• /etc/default/kdump-tools: KDUMP_INITRD does not exist: /var/lib/kdump/initrd.img
  USE_KDUMP: 1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR: /var/crash
  crashkernel addr: 0x18000000
  kdump kernel addr:
  kdump kernel:
  /var/lib/kdump/vmlinuz
  kdump initrd:
  /var/lib/kdump/initrd.img: broken symbolic link to /var/lib/kdump/initrd.img-4.19.128-grahamm-syskernel
  kexec command to be used:
  /sbin/kexec -p --command-line="initrd=\initrd.img panic=-1 nr_cpus=12 swiotlb=force pty.legacy_count=0 panic=30, oops=panic, panic_print=ff reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll nousb ata_piix.prefer_ms_hyperv=0" /var/lib/kdump/vmlinuz

[grahamm78]

One note of advice - don't put your custom kernels in c:\Windows\system32\lxss\tools\
It looks like you lose them after a build upgrade..

I'll keep working on this and post an update if I find a solution.