Friday, February 18, 2011

~Core-dump Analysis - I~

Today we would discuss the basics of core-dump analysis on GNU/Linux machine.
I would try to cover topics like:

* What extra do we need while compiling the program?
* Why & When Kernel try to make core-dump of a process?
* Do we need to set/enable parameter in order to create core-dump?
* How many ways we can get core-dump of a process?
* Sample Programs which sends various signals to kernel?
* Analysis of core-dump using gdb.



Q. What extra do we need while compiling the program?
Using -g flag in gcc generates the extra symbols which would be useful
while debugging and while analyzing the core-dump analysis. Even if you
don't have the object file compiled with -g option, we can have core-dump
file for a process. But sometime these core file may not be of use for
analysis. So if possible,try to compile your program with -g option. Following commands generates the object file "osource" for the source file
"source.c" with extra symbol.

mantosh@ubuntu:~$ gcc -Wall -g source.c -o osource
mantosh@ubuntu:~$ gcc -save-temps -g source.c -o osource



Q. Why & When Kernel try to make core-dump of a process?
Signal is a notification to a process that an event has occurred. Signals
are sometimes described as software interrupts. The usual source of many
signals sent to a process is the kernel. A signal is said to be generated
by some event. Once generated, a signal is later delivered to a process,
which then takes some action in response to the signal. Upon delivery of
a signal, a process carries out one of the following default actions,
depending on the signal: "ignored", "terminated", "core-dump file". Detailed information can be found on these by doing "man core" and
"man signal" on our machine terminal.




Q. Do we need to set/enable parameter in order to create core-dump?
Yes. On most of machine we need to set some kernel parameter in order to
enable the core-dump file. Just check the output of the following command.


mantosh@ubuntu:~$ ulimit -a
core file size (blocks, -c) 0 ==>default core file size is 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


mantosh@ubuntu:~$ cat /proc/sys/kernel/core_uses_pid
0 ==>By setting this parameter to 1,kernel would generate the core
==>file name ike core.pid in our current working directory.

mantosh@ubuntu:~$ cat /proc/sys/kernel/core_pattern
core


The core file size can be changed to unlimited size by the command.
mantosh@ubuntu:~$ ulimit -c unlimited

Similarly we can generate the core file like core.pid by setting the
mantosh@ubuntu:~$ echo 1 > /proc/sys/kernel/core_uses_pid
mantosh@ubuntu:~$ cat /proc/sys/kernel/core_uses_pid
1



Now kernel should be able to generate the core-dump file. Please note that
these setting should be done using the "root" user in order to take effect.






Q. How many ways we can get core-dump of a process?
We can get a core-dump in 3 ways. These are :
1. If program is running and does like invalid memory reference,floating point exception or trying to execute invalid instruction,kernel would
generate the core-dump file for that particular process.
2. we can call abort() system call inside our program. When program executes abort(),process will be aborted and kernel would take the
core-dump of a program which would contain the information of a program before abort() call.
3. While debugging our program using gdb, we can use "gcore" command which
generate the core file. The biggest advantage of "gcore" is that you can get core-dump at anytime,(i.e its on user when he/she want to generate core file).





=====================================================
Sample Program which i would be using to demonstrate the concept of core-
dump. I have written most of test program which uses multiple function
calls, global variable, multiple local variable, so that while analysing we
would have so many variables and stacktrace. Now lets start some actual stuff :)



// Test Program:: 1, Langauge :c++
// Purpose:: Fetch the basic information about a filename provided by the
user.


#include"iostream"
#include"cstring"
#include"cstdlib"
#include"fcntl.h"
#include"sys/stat.h"
using namespace std;
// class declaration
class fileinfo {
private:
char *filename;
int filedp;
struct stat status;
int success_flag;
public:
fileinfo(char *fname);
int file_open();
int file_stat();
int file_close();
void file_information();
~fileinfo();
};

// Constructor
fileinfo :: fileinfo(char *fname) {
filedp = -1;
success_flag = 0;
memset(&status, 0, sizeof(struct stat));
int length = strlen(fname);
filename = (char *)malloc(length);
strcpy(filename,fname);
}

// Member function for open the file by calling open()
int fileinfo :: file_open() {
filedp = open(filename,O_RDONLY);
if(-1 == filedp)
cout << "Error while opening the " < else
success_flag = 1;
return success_flag;
}

//Member function for getting information about particular file
int fileinfo :: file_stat() {
int retv;
retv = fstat(filedp, &status);
if(-1 == retv)
cout << "Error while getting status of "< else
success_flag = 1;
return success_flag;
}

// Main member function for displaying the basics information about file,
// if we are able to open a file and able to do successfull call of fstat()

void fileinfo :: file_information() {
int retv;
retv = file_open();
if(1 == retv) {
retv = file_stat();
if(1==retv) {
cout << "File: " << filename << "\n";
cout << "Inode: " << status.st_ino << "\n";
cout << "Links: " << status.st_nlink <<"\n";
cout <<"Size: " << status.st_size << "\n";
}
}
file_close();
}

// Member function for closing the file by calling close()
int fileinfo :: file_close() {
int retv;
retv = close(filedp);
if(-1== retv)
cout << "Error while closing the " << filename << " file\n";
else
success_flag = 1;
return success_flag;
}

// Destructor
fileinfo :: ~fileinfo() {

free(filename);
}

// Main program which usage all above functions.
int main(int argc, char *argv[]) {

if(argc !=2) {
cout <<"Invalid Usage:"<< argv[0] <<" \n";
return 1;
}

char *name = argv[1];
fileinfo f1(name);
f1.file_information();
return 0;
}




==> Now we should compile using g++ and run the program "otest" inside gdb,
==> so that we should be able to take core-dump using gcore command.


mantosh@ubuntu:~/Desktop$ gdb --args ./otest blog.c
(gdb) b main
Breakpoint 1 at 0x8048c84: file test.cc, line 93.
(gdb) b fileinfo :: ~fileinfo()
Breakpoint 2 at 0x8048956: file test.cc, line 87. (2 locations)
(gdb) r
Starting program: /home/mantosh/Desktop/otest blog.c

Breakpoint 1, main (argc=2, argv=0xbfc436d4) at test.cc:93
93 if(argc !=2) {
(gdb) bt
#0 main (argc=2, argv=0xbfc436d4) at test.cc:93
(gdb) n
98 char *name = argv[1];
(gdb)
99 fileinfo f1(name);
(gdb)
100 f1.file_information();
(gdb)
File: blog.c
Inode: 244585
Links: 1
Size: 4640
101 return 0;
(gdb) c
Continuing.

Breakpoint 2, ~fileinfo (this=0xbfc435c4) at test.cc:87
87 free(filename);
(gdb) bt
#0 ~fileinfo (this=0xbfc435c4) at test.cc:87
#1 0x08048d13 in main (argc=2, argv=0xbfc436d4) at test.cc:101
(gdb) help gcore
Save a core file with the current state of the debugged process.
Argument is optional filename. Default filename is 'core.'.
(gdb) gcore
Saved corefile core.9987 ==> core file name is " core.9987"
(gdb) q
The program is running. Exit anyway? (y or n) y




// Now we can do core-dump analysis which we just generated by the command // gcore inside gdb. See my explaination during the analysis which i have
// put using prefix ==>

mantosh@ubuntu:~/Desktop$ gdb

(gdb) core core.9987 ==>load core.9987 in gdb session
(no debugging symbols found)
Core was generated by `/home/mantosh/Desktop/otest blog.c'.
Program terminated with signal 5, Trace/breakpoint trap.
[New process 9987]
==> signal 5(SIGTRAP) was the reason for this core-file
==> process id was 9987.
#0 0x08048956 in ?? ()
(gdb) bt
==>backtrace of program without loading the symbol
#0 0x08048956 in ?? ()
#1 0x08048d13 in ?? ()
#2 0xb7dd5775 in ?? ()
#3 0x08048861 in ?? ()
(gdb) symbol ./otest
==> Load symbol otest
Reading symbols from /home/mantosh/Desktop/otest...done.
(gdb) bt
==>Now we get the actual backtrace of a prgram
#0 ~fileinfo (this=0xbfc435c4) at test.cc:87
#1 0x08048d13 in main (argc=2, argv=0xbfc436d4) at test.cc:101
(gdb) frame 1
==>select frame 1
#1 0x08048d13 in main (argc=2, argv=0xbfc436d4) at test.cc:101
101 return 0;
(gdb) info args
==>for displaying arguments information for frame 1
argc = 2
argv = (char **) 0xbfc436d4
(gdb) p argv[0]
$1 = 0xbfc456cb "/home/mantosh/Desktop/otest"
(gdb) p argv[1]
$2 = 0xbfc456e7 "blog.c"
(gdb) p argv[2]
$3 = 0x0
(gdb) x/3xw argv
==>display the addresses of argv[0], argv[1], note how
==> argv is null terminated.
0xbfc436d4: 0xbfc456cb 0xbfc456e7 0x00000000
(gdb) info locals
==> for displaying the locals variables inside frame 1
name = 0xbfc456e7 "blog.c"
f1 = {
filename = 0x896f008 "blog.c",
filedp = 6,
status = {
st_dev = 1792,
__pad1 = 0,
st_ino = 244585,
st_mode = 33188,
st_nlink = 1,
st_uid = 1000,
st_gid = 1000,
st_rdev = 0,
__pad2 = 0,
st_size = 4640,
st_blksize = 4096,
st_blocks = 16,
st_atim = {
tv_sec = 1298087485,
tv_nsec = 0
},
st_mtim = {
tv_sec = 1298085235,
tv_nsec = 0
},
st_ctim = {
tv_sec = 1298085235,
tv_nsec = 0
},
__unused4 = 0,
__unused5 = 0
},
success_flag = 1
}
(gdb) ptype f1
==>type information of any variable(Here object of class
==>fileinfo which is f1).
type = class fileinfo {
private:
char *filename;
int filedp;
stat status;
int success_flag;

public:
fileinfo(char *);
int file_open();
int file_stat();
int file_close();
void file_information();
~fileinfo(int);
}
(gdb) p sizeof(f1)
==> size of object f1
$4 = 100
(gdb) p name
$5 = 0xbfc456e7 "blog.c"
(gdb) x/10cb name
0xbfc456e7:98 'b' 108 'l' 111 'o' 103 'g' 46 '.' 99 'c' 0 '\0' 79 'O'
0xbfc456ef:82 'R' 66 'B'
(gdb) p &f1
==> address of object f1 of type fileinfo class
$6 = (fileinfo *) 0xbfc435c4
(gdb) x/25xw &f1
==> the above command would display the 25 unit of size word(which is 4byt)
==> in hexadecimal format.(x/nfu). 1st unit store the address of filename
==> which is(0x0896f008). Next unit store the variable filedp value which
==> is 6 and so on ...we can interpret in different ways of the information
==> stored into the address &f1.
0xbfc435c4: 0x0896f008 0x00000006 0x00000700 0x00000000
0xbfc435d4: 0x00000000 0x0003bb69 0x000081a4 0x00000001
0xbfc435e4: 0x000003e8 0x000003e8 0x00000000 0x00000000
0xbfc435f4: 0x00000000 0x00001220 0x00001000 0x00000010
0xbfc43604: 0x4d5f3e3d 0x00000000 0x4d5f3573 0x00000000
0xbfc43614: 0x4d5f3573 0x00000000 0x00000000 0x00000000
0xbfc43624: 0x00000001
(gdb) x/10cb 0x0896f008
==> this would print the f1->filename, since filename has been stored at
==> 1st offset of object f1.
0x896f008: 98 'b' 108 'l' 111 'o' 103 'g' 46 '.' 99 'c' 0 '\0' 0 '\0'
0x896f010: 0 '\0' 0 '\0'

(gdb) p f1.filedp
$7 = 6
(gdb) p/d 0x00000006
==> print the decimal eqv of hex val 0x00000006
$8 = 6
(gdb) p f1.status.st_dev
$9 = 1792
(gdb) p sizeof(f1.status.st_dev)
$10 = 8
(gdb) p/x 1792
$11 = 0x700
(gdb) p/d 0x00000700
$12 = 1792
(gdb) p f1.status.st_ino
$13 = 244585
(gdb) p f1.status.st_nlink
$14 = 1
(gdb) x/25dw &f1
==> print the 25 unit information in decimal from starting address &f1
0xbfc435c4: 144109576 6 1792 0
0xbfc435d4: 0 244585 33188 1
0xbfc435e4: 1000 1000 0 0
0xbfc435f4: 0 4640 4096 16
0xbfc43604: 1298087485 0 1298085235 0
0xbfc43614: 1298085235 0 0 0
0xbfc43624: 1
(gdb) p/x 144109576
$15 = 0x896f008
(gdb) bt
#0 ~fileinfo (this=0xbfc435c4) at test.cc:87
#1 0x08048d13 in main (argc=2, argv=0xbfc436d4) at test.cc:101
(gdb) f 0
#0 ~fileinfo (this=0xbfc435c4) at test.cc:87
87 free(filename);
(gdb) info locals
No locals.
(gdb) info args
==> Note how internally this pointer passed to its member function in cpp .
this = (fileinfo * const) 0xbfc435c4
(gdb) p sizeof(*this)
$16 = 100
(gdb) p filename
$17 = 0x896f008 "blog.c"
(gdb) info reg
==> This would display the value/state of various registers at the time of core-file was generated.
eax 0xbfc435c4 -1077660220
ecx 0x1 1
edx 0x1220 4640
ebx 0xb7f1dff4 -1208885260
esp 0xbfc43590 0xbfc43590
ebp 0xbfc43598 0xbfc43598
esi 0x8048de0 134516192
edi 0x8048840 134514752
eip 0x8048956 0x8048956 <~fileinfo+6 at test.cc:87>
eflags 0x200286 [ PF SF IF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
(gdb) p $eax
==> for printing the registers values use prefix $
$18 = -1077660220
(gdb) p/x -1077660220
==> we can see from here $eax store the "this" pointer.
$19 = 0xbfc435c4
(gdb) p this
$19 = (fileinfo * const) 0xbfc435c4
(gdb) x/i $eip
==> $eip store the next instruction which was supposed to be executed. Most
==> of time $eip would store the information of instruction which cause the
==> trouble. See your processor architecure manual for understanding of
==> various regeisters utility.
0x8048956 <~fileinfo+6 at test.cc:87>:Cannot access memory at address 0x8048956
(gdb) bt
#0 ~fileinfo (this=0xbfc435c4) at test.cc:87
#1 0x08048d13 in main (argc=2, argv=0xbfc436d4) at test.cc:101
(gdb) info frame
==> for getting additional information of a prticular selected frame
Stack level 0, frame at 0xbfc435a0:
eip = 0x8048956 in ~fileinfo (test.cc:87); saved eip 0x8048d13
called by frame at 0xbfc43640
source language c++.
Arglist at 0xbfc4358c, args: this=0xbfc435c4
Locals at 0xbfc4358c, Previous frame's sp is 0xbfc435a0
Saved registers:
ebp at 0xbfc43598, eip at 0xbfc4359c
(gdb) bt
#0 ~fileinfo (this=0xbfc435c4) at test.cc:87
#1 0x08048d13 in main (argc=2, argv=0xbfc436d4) at test.cc:101
(gdb) f 1
#1 0x08048d13 in main (argc=2, argv=0xbfc436d4) at test.cc:101
101 return 0;
(gdb) info frame
Stack level 1, frame at 0xbfc43640:
eip = 0x8048d13 in main (test.cc:101); saved eip 0xb7dd5775
caller of frame at 0xbfc435a0
source language c++.
Arglist at 0xbfc4359c, args: argc=2, argv=0xbfc436d4
Locals at 0xbfc4359c, Previous frame's sp at 0xbfc4362c
Saved registers:
ebx at 0xbfc43630, ebp at 0xbfc43638, esi at 0xbfc43634, eip at 0xbfc4363c
(gdb) q



Please see my the next article on the same topic.

No comments:

Post a Comment