| Debugging tips: | Kernel User environments |
|---|---|
| Reference: | JOS makefile JOS obj/ GDB QEMU Patched QEMU |
GDB is your friend. Use the qemu-gdb target (or its qemu-gdb-nox variant) to make QEMU wait for GDB to attach. See the GDB reference below for some commands that are useful when debugging kernels.
If you're getting unexpected interrupts, exceptions, or triple faults, you can ask QEMU to generate a detailed log of interrupts using the -d argument.
To debug virtual memory issues, try the QEMU monitor commands info mem (for a high-level overview) or info pg (for lots of detail). Note that these commands only display the current page table.
(Lab 4+) To debug multiple CPUs, use GDB's thread-related commands like thread and info threads.
GDB also lets you debug user environments, but there are a few things you need to watch out for, since GDB doesn't know that there's a distinction between multiple user environments, or between user and kernel.
You can start JOS with a specific user environment using make run-name (or you can edit kern/init.c directly). To make QEMU wait for GDB to attach, use the run-name-gdb variant.
You can symbolically debug user code, just like you can kernel code, but you have to tell GDB which symbol table to use with the symbol-file command, since it can only use one symbol table at a time. The provided .gdbinit loads the kernel symbol table, obj/kern/kernel. The symbol table for a user environment is in its ELF binary, so you can load it using symbol-file obj/user/name. Don't load symbols from any .o files, as those haven't been relocated by the linker (libraries are statically linked into JOS user binaries, so those symbols are already included in each user binary). Make sure you get the right user binary; library functions will be linked at different EIPs in different binaries and GDB won't know any better!
(Lab 4+) Since GDB is attached to the virtual machine as a whole, it sees clock interrupts as just another control transfer. This makes it basically impossible to step through user code because a clock interrupt is virtually guaranteed the moment you let the VM run again. The stepi command works because it suppresses interrupts, but it only steps one assembly instruction. Breakpoints generally work, but watch out because you can hit the same EIP in a different environment (indeed, a different binary altogether!).
When building JOS, the makefile also produces some additional output files that may prove useful while debugging:
See the GDB manual for a full guide to GDB commands. Here are some particularly useful commands for our course, some of which don't typically come up outside of OS development.
QEMU represents each virtual CPU as a thread in GDB, so you can use all of GDB's thread-related commands to view or manipulate QEMU's virtual CPUs.
QEMU includes a built-in monitor that can inspect and modify the machine state in useful ways. To enter the monitor, press Ctrl-a c in the terminal running QEMU. Press Ctrl-a c again to switch back to the serial console.
For a complete reference to the monitor commands, see the QEMU manual. Here are some particularly useful commands:
CS =0008 10000000 ffffffff 10cf9a00 DPL=0 CS32 [-R-]
ef7c0000-ef800000 00040000 urw efbf8000-efc00000 00008000 -rwtells us that the 0x00040000 bytes of memory from 0xef7c0000 to 0xef800000 are mapped read/write and user-accessible, while the memory from 0xefbf8000 to 0xefc00000 is mapped read/write, but only kernel-accessible.
VPN range Entry Flags Physical page [00000-003ff] PDE[000] -------UWP [00200-00233] PTE[200-233] -------U-P 00380 0037e 0037d 0037c 0037b 0037a .. [00800-00bff] PDE[002] ----A--UWP [00800-00801] PTE[000-001] ----A--U-P 0034b 00349 [00802-00802] PTE[002] -------U-P 00348This shows two page directory entries, spanning virtual addresses 0x00000000 to 0x003fffff and 0x00800000 to 0x00bfffff, respectively. Both PDE's are present, writable, and user and the second PDE is also accessed. The second of these page tables maps three pages, spanning virtual addresses 0x00800000 through 0x00802fff, of which the first two are present, user, and accessed and the third is only present and user. The first of these PTE's maps physical page 0x34b.
4: v=30 e=0000 i=1 cpl=3 IP=001b:00800e2e pc=00800e2e SP=0023:eebfdf28 EAX=00000005
EAX=00000005 EBX=00001002 ECX=00200000 EDX=00000000
ESI=00000805 EDI=00200000 EBP=eebfdf60 ESP=eebfdf28
...
The first line describes the interrupt. The 4: is
just a log record counter. v gives the vector number in
hex. e gives the error code. i=1 indicates
that this was produced by an int instruction (versus a
hardware interrupt). The rest of the line should be self-explanatory.
See info registers
for a description of the register dump that follows.
$ sudo apt-get -y install pkg-config zlib1g-dev libglib2.0-dev dh-autoreconf bison flex libsdl2-2.0-0 libsdl2-dev libsdl2-net-dev libsdl2-ttf-dev libsdl2-gfx-dev libsdl2-image-dev $ cd ~/ $ mkdir -p src $ cd src $ mkdir qemu $ cd qemu $ wget http://technion-csl.github.io/ose/software/qemu/qemu-2.3.0.tar.bz2 $ tar -jxvf qemu-2.3.0.tar.bz2 $ cd qemu-2.3.0 Now download and apply "info pg" command patches on qemu sources: $ wget http://technion-csl.github.io/ose/software/qemu/info_pg1.diff $ wget http://technion-csl.github.io/ose/software/qemu/info_pg2.diff $ patch -p1 < info_pg1.diff $ patch -p1 < info_pg2.diff Apply following patch to dump state and halt for inspection instead of resetting on triple fault: $ wget http://technion-csl.github.io/ose/software/qemu/triple_fault.diff $ patch -p1 < triple_fault.diff Apply following patch to generate E1000 debug messages (useful for lab6): $ wget http://technion-csl.github.io/ose/software/qemu/e1000_debug.diff $ patch -p1 < e1000_debug.diff Create "bin" directory in your home directory, compile and install qemu: $ mkdir ~/bin $ ./configure --prefix=$HOME/bin --disable-kvm --target-list=i386-softmmu --enable-sdl --with-sdlabi=2.0 $ make $ make install Now go to the source directory of jos and run: $ export QEMU=$HOME/bin/bin/qemu-system-i386 $ make qemu Go to qemu monitor using "CTRL-a c" sequence and run "info pg": QEMU 2.3.0 monitor - type 'help' for more information (qemu) info pg VPN range Entry Flags Physical page [00000-003ff] PDE[000] ----A--UWP [00001-00001] PTE[001] ---DA--UWP 0039a [00200-0020b] PTE[200-20b] ---DA--UWP 003b9 003b8 003b7 003b6 003b5 003b4 .. [00800-00bff] PDE[002] ----A--UWP [00800-0080b] PTE[000-00b] ---DA--UWP 003ad 003ab 003aa 003a9 003a8 003a7 .. [10000-103ff] PDE[040] ----A--UWP [10001-10001] PTE[001] ----A--UWP 0039d [10002-10003] PTE[002-003] -------UWP 0039b 00397 [10074-10075] PTE[074-075] ----A--UWP 00399 00398 [ee800-eebff] PDE[3ba] ----A--UWP [eebfd-eebfd] PTE[3fd] ---DA--UWP 003a0 [eebff-eebff] PTE[3ff] ---DA--UWP 0039eDone!