To illustrate some of the many commands available with scat, I’m going to start up scat in its shorthand fashion — by providing the core dump “generation” as the only argument to the scat command.
# cd /var/crash # scat 4
The “scat 4” command is shorthand for “scat unix.4 vmcore.4”. You might have to add scat’s bin directory to your $PATH first:
# export PATH=$PATH:/opt/SUNWscat/bin
After identifying itself (e.g., Solaris[TM] CAT 4.1 …), scat opens the vmcore file (vmcore.4 in this example) and displays some information about the time of crash and the system on which it was generated.
To figure out where to go next, type “help” at the command prompt. A list of scat commands will be generated and categorized by type. Under the “Initial Investigation” label, you will see some of the more useful commands for taking a first stab at what went wrong. The analyze command performs a “quick triage” of the core file, highlighting some of its output in red to get your attention. You can generate an even more detailed report with “analyze -l”. The panic command will provide the panic string (e.g., forced crash dump initiated at user request), a traceback of system calls.
SolarisCAT(vmcore.4)> help
Initial Investigation:
analyze coreinfo msgbuf panic stack stat
toolinfo
General Commands:
analyze clockinfo coreinfo demangle eckstat environ
exit help intr kstat modinfo msgbuf
stat symbols taskq tunables
Memory Dump/Display Commands:
bigdump buf flip kseg mdump pdump
r rd rd16 rd32 rd64 rd8
rdb rdc rdd rdf rdh rdi
rdl rdq rds rdw sdump seg
skma strsum vmem wr
Data Conversion Commands:
2base 2dec 2double 2float 2hex 2string
2time bits calc decode deltatime pid2paddr
size
Process/Thread Information:
panic pid2paddr proc proctbl thread tlist
upcount
Stacks/Traps:
findaddr_stack frame regs stack trap
Searching Memory/Core:
findport findval stack tlist
File and Filesystem Information:
bigdump buf cgcheck dnlc findfiles getpath
inode node vfssw vfstype
CPU and Dispatch Queues:
callout cpu dispq disptbl
Memory Information:
anon iommu ipc kma kseg kstat
map memerr meminfo page resvswap seg
sfmmu shmchk skma swapinfo vmem
Network/IPC:
findport ifconf ipc ndd nstat pdump
stream
Type Database Commands:
sarray savl sdump skma slist stype
Lock Display Commands:
lck mutex mutextbl rwlock sema tstile
Device Information Commands:
dev
System Files/Tunables:
etcsystem msgbuf name2major path2inst tunables vfstab
Disassembler Commands:
codepath dis rdi sim
Data Structure Consistency Checks:
anon cgcheck dispq dnlc kma seg
sfmmu symbols upcount vmem
Miscellaneous Commands:
legal namelist patchdb pty sanitize toolinfo
trans
Tool Configuration:
base color refclock scatenv write
Usage: help [command]
help -? regular_expression
help all
|
To find out more about these and other commands available in scat, type “help” followed by the command name. The “help panic” command, for example, will give you a list of options available with the panic command.
SolarisCAT(vmcore.4)> help panic
panic [-l] [allpanicstr|buf|cpu|kmem|panic2|ptl1|string|thread]
Displays the specified panic information
-l display any thread(s) that have pinned the panic thread
allpanicstr prints the panic string and any other panic strings
found in the panic stack
buf Prints out the panicbuf structure, which contains
name-value pairs of interesting panic information, plus
the panic message.
cpu displays the cpu that had the panic
kmem prints the kmem_panic_info data
panic2 dumps the 2nd panic thread
ptl1 dumps the ptl1 panic info
string dumps the panic string
thread display the panic thread
|
The allpanicstr option shows that there’s only one panic string in the panic stack for this particular crash dump:
SolarisCAT(vmcore.4)> panic allpanicstr panic string: forced crash dump initiated at user request panicstr: forced crash dump initiated at user request
Some of the information you will find running the various scat commands will be redundant, of course, but …
SolarisCAT(vmcore.4)> panic buf
version: 1
panic message: forced crash dump initiated at user request
name value
---- -----
tstate 0x4400001604
%cwp:4 %pstate:0x16 (IE|PRIV|PEF|TSO)
%asi:0 %ccr:0x44 (IZ|XZ)
g1 0xb
g2 0x4
g3 0x4
...
|
The tlist command will match threads that match some particular criteria such as bound, door and pri. The tlist pinned command, for example, provides a list of “pinned” (i.e., interrupted) threads. You can add -l or -v for more detailed (verbose) output or such options as -f (include free threads), -h (table only) or other options to further modify what you see.
SolarisCAT(vmcore.4)> tlist pinned address translation failed for proc: 2624 bytes @ 0x0 ==== kernel thread: 0x180e000 ==== cmd: (unknown) t_stk: 0x180bae0 sp: 0x180b1d1 t_stkbase: 0x0 t_pri: 96(SYS) pctcpu: 0.000000 t_lwp: 0x0 bound psrset: 0 last cpuid: 0 idle: 255939 ticks (42 minutes 39.39 seconds) start: Sun Sep 18 19:14:08 3566 age: -2889264050706 seconds (33440556 days 3 hours 25 minutes 6 seconds) stime: 30064771072 (3479 days 16 hours 32 minutes 31.33 seconds later) interrupted (pinned) thread: 0x300001ccac0 |
Even with a smart tool, an armful of commands and just about all the data you could ask to see, analyzing core dumps is a difficult task. If you look at enough core dumps, however, the process just might get a little easier over time. At least I hope so!
See last week’s column: Probing into Core Dumps with mdb
