While mdb can be used to interact with the live kernel or probe into core dumps in complex and intimidating ways, it’s also not very hard to get some basic and very useful information from a core dump that will help you determine why the system crashed.
You can start mdb with the command “mdb -k” if you want to look into the live kernel. Use control-D to exit. For peering into core dumps, you would use a command like “mdb unix.0 vmcore.0” (adjust the dump numbers to match your core files).
The command for extracting the panic message from a core dump (you should also be able to find this in your messages files) is ::status as shown in the example shown here:
> ::status debugging crash dump vmcore.0 (64-bit) from boson operating system: 5.10 Generic_125100-05 (sun4u) panic message: forced crash dump initiated at user request dump content: kernel pages only |
In this case, we have a core dump that might appear to have been initiated by a normal user typing “panic” on the command line. Unix systems, as we know, don’t lend themselves to going down the tubes so easily. Instead, this crash was caused by some user level process that transgressed is some way the proper operating system etiquette. There are numerous reasons why systems panic, but they generally indicate that no safe alternative to crashing was available due to some very serious fault.
The command for displaying a traceback (::stack), displaying the process causing the crash and its predecessors, will often help to answer the key question, “Where did this crash come from?”. Here’s an example:
> ::stack vpanic(11f2d48, 1, 182c800, 182c800, 0, 182a000) kadmin+0x4a4(b4, 1, 0, 11f2c00, 5, 1) ntwdt_enforce_timeout+0x3c(60002d90ac0, 2a10058dcc0, 7bbabc00, 7bbabc00, 1910c00, 0) ntwdt_cyclic_softint+0xd0(60002dc72e8, 7bbabc00, 60002d90ac0, 0, 704f5c00, 7bbab4e4) intr_thread+0x170(0, b, 0, ffffffffffffffff, 300001ccaf8, 18) |
For ::stack output, you read backwards. The top line is the most recent event. In this case, we see the vpanic that caused the system to reboot and dump core.
To get more information on a panic, issue the ::panicinfo command as shown here:
> ::panicinfo
cpu 2
thread 2a10058dcc0
message forced crash dump initiated at user request
tstate 4480001607
g1 b
g2 6
g3 6
g4 30002e475c0
g5 88000000
g6 0
g7 2a10058dcc0
o0 11f2d48
o1 2a10058d938
o2 1
o3 1
o4 fffffffffffffff5
o5 9
o6 2a10058d001
o7 1060da0
pc 104460c
npc 1044610
y 0
|
Take note of the thread noted in the panicinfo output. You might be able to use this information to tie the panic to a particular process.
The command for displaying a list of all the processes that were running at the time of the crash is rather straightforward. You just type “::ps” at the mdb prompt:
> ::ps S PID PPID PGID SID UID FLAGS ADDR NAME R 0 0 0 0 0 0x00000001 000000000187c980 sched R 3 0 0 0 0 0x00020001 0000060002ebbb90 fsflush R 2 0 0 0 0 0x00020001 0000060002ebc778 pageout R 1 0 0 0 0 0x4a004000 0000060002ebd360 init R 1146 1 1146 1146 0 0x4a004000 00000600048503d0 msgmon R 1147 1 1147 1147 0 0x4a004000 0000060005a5afd8 advproc R 1148 1 1148 1148 0 0x4a004000 0000060005a58038 benproc R 2547 2247 2245 983 0 0x4a004000 000006000c3d5c18 suspit ... |
If you issue mdb “walk thread: and “findstack” commands using a process address (ADDR) as an argument as in “000006000c3d5c18::walk thread | ::findstack”, you will get a listing of the threads from that process. If you’re lucky, you might find one that refers to the thread noted in your panicinfo output. The thread output for a single process will look something like this:
stack pointer for thread 300028aa360: 2a1014b9071 [ 000002a1014b9071 cv_timedwait_sig+0x16c() ] 000002a1014b9121 cv_waituntil_sig+0x8c() 000002a1014b91f1 nanosleep+0x100() 000002a1014b92e1 syscall_trap32+0xcc() |
If the thread listed (in this case, 300028aa360) matches the one listed in your panicinfo output, you’ve found your culprit. You could page through the thread listing for all processes using a simple script like this one:
#!/bin/bash
# generate process listing
echo "::ps" | mdb -k unix.0 vmcore.0 | awk '$8 !~ /ADDR/ {print $8,$NF}' > ps$$
# review thread listing for each process
while read addr proc
do
echo $proc
echo $addr | awk '{print $1,"::walk thread | ::findstack"}' | mdb unix.0 vmcore.0
echo "================================================================="
done
|
If you’d like to see what files a particular process had opened at the time of the crash, use the ::pfiles command with the process address. Notice that this output displays standard in, standard out and standard err (file descriptors 0, 1 and 2) along with other devices and files.
> 000006000c3d5c18::pfiles FD TYPE VNODE INFO 0 REG 000006000ccc6900 /appdata/conf/applyperms 1 FIFO 000006000c2a2d00 2 FIFO 000006000c2a2d00 10 CHR 0000060005a2f800 /devices/pseudo/pts@0:1 11 REG 000006000c4088c0 /tmp/sh2547.1 62 REG 000006000c543400 /appdata/bin/applyfix |
These basic mdb commands can supply you with quite a bit of useful information for identifying the nature and cause of a system crash.
