gdb not loading symbols after attaching to process (MySQL debug binary)-CodePudding

I'm trying to use gdb to step-debug mysql internals.

I compiled the debug binary (Server version: 8.0.28-debug Source distribution ), but it doesn't seem like gdb can access most of the symbols.

When I attach gdb to a running server process, then send a MySQL query in another terminal, gdb breaks at some OS level function, and the query terminal does get paused. But when I try to "step into," the query result is returned without me getting to peek into any code.

When I run info sources or info functions in gdb, almost nothing seems to be from mysql, except a few under MySQL's extras directory. Scrolling way down the info functions output, below Non-debugging symbols I finally see a bunch of mysql functions. So, they don't appear to be loaded.

I started mysqld with the --gdb=ON --debug=ON flags, and also tried setting the SESSION variable for debugging in the MySQL terminal:

mysql> SET SESSION debug = 'd:t:i:o,/tmp/mysqld.trace:F:L:N:P';

Nothing I've tried has worked yet.

Here's my whole setup, from the point of launching a vagrant VM with Ubuntu 20.04. I basically followed the steps here, modified for a debug install.

$ apt update
$ apt upgrade -y
$ apt install libssl-dev libncurses5-dev pkg-config bison build-essential –y
$ cd ~
$ mkdir bld
$ cd bld
$ ################
$ ## BUILD STEP ##
$ ################
$ cmake /vagrant/mysql-server/ -DDOWNLOAD_BOOST=1 -DWITH_BOOST=$HOME/my_boost \
    -DWITH_DEBUG=true -DWITH_INNODB_EXTRA_DEBUG=true
$ # /vagrant folder is mounted from the Windows 10 host OS, contains the mysql git repo
$ make -j8
$ make install
$ ###
$ groupadd mysql
$ useradd -r -g mysql -s /bin/false mysql
$ cd /usr/local/mysql-test
$ mkdir mysql-files
$ chown mysql:mysql mysql-files
$ chmod 750 mysql-files
$ ###
$ /usr/local/bin/mysqld --initialize-insecure --user=mysql
$ # I dont need "secure" on local VM
$ 
$ ##################################
$ # START THE SERVER FOR DEBUGGING #
$ ##################################
$ /usr/local/bin/mysqld_safe --user=mysql --gdb=ON --debug=ON --thread_cache_size=5
$ # get mysqld PID
$ ps aux | grep mysql
root        2565  0.0  0.0  11856  4492 pts/0    S    07:12   0:00 sudo mysqld_safe --user=mysql --debug=ON --gdb=ON
root        2566  0.0  0.0   2608  1808 pts/0    S    07:12   0:00 /bin/sh /usr/local/bin/mysqld_safe --user=mysql --debug=ON --gdb=ON
mysql       2667  1.1  4.0 1818376 415952 pts/0  Sl   07:12   0:11 /usr/local/bin/mysqld --basedir=/usr/local --datadir=/usr/local/data --plugin-dir=/usr/local/lib/plugin --user=mysql --debug=ON --gdb=ON --log-error=vagrant.err --pid-file=vagrant.pid
vagrant     2745  0.0  0.0   8900   736 pts/1    S    07:29   0:00 grep --color=auto mysql
$ # START GDB
$ gdb
(gdb) attach 2667
Attaching to process 2667
[New LWP 2670]
[New LWP 2671]
[New LWP 2672]
[New LWP 2673]
[New LWP 2674]
[New LWP 2675]
[New LWP 2676]
[New LWP 2677]
[New LWP 2678]
[New LWP 2679]
[New LWP 2680]
[New LWP 2681]
[New LWP 2682]
[New LWP 2683]
[New LWP 2684]
[New LWP 2685]
[New LWP 2688]
[New LWP 2689]
[New LWP 2690]
[New LWP 2691]
[New LWP 2692]
[New LWP 2693]
[New LWP 2694]
[New LWP 2695]
[New LWP 2696]
[New LWP 2697]
[New LWP 2701]
[New LWP 2702]
[New LWP 2703]
[New LWP 2704]
[New LWP 2705]
[New LWP 2706]
[New LWP 2707]
[New LWP 2708]
[New LWP 2709]
[New LWP 2711]
[New LWP 2713]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f200fe2faff in __GI___poll (fds=0x7f1ffecb87d0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29      ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
(gdb)

None of the post-attach output mentions mysql. gdb does not seem to be importing the symbols.

This tutorial suggests that the gdb output after attach [PID] should look more like the following, with a line ike Reading symbols from ... mysqld...done.:

(gdb) attach 7506
Attaching to process 7506
Reading symbols from /home/cbell/source/bzr/mysql-5.6/sql/mysqld...done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7f7ba8e57700 (LWP 7524)]
[New Thread 0x7f7ba4a26700 (LWP 7523)]
[New Thread 0x7f7ba5227700 (LWP 7522)]
[New Thread 0x7f7ba5a28700 (LWP 7521)]
[New Thread 0x7f7ba6229700 (LWP 7520)]
[New Thread 0x7f7ba6a2a700 (LWP 7519)]
[New Thread 0x7f7ba722b700 (LWP 7518)]
[New Thread 0x7f7ba7a2c700 (LWP 7517)]
[New Thread 0x7f7ba822d700 (LWP 7516)]
[New Thread 0x7f7ba8a2e700 (LWP 7515)]
[New Thread 0x7f7ba9658700 (LWP 7513)]
[New Thread 0x7f7ba9e59700 (LWP 7512)]
[New Thread 0x7f7baa65a700 (LWP 7511)]
[New Thread 0x7f7baae5b700 (LWP 7510)]
[New Thread 0x7f7bab65c700 (LWP 7509)]
[New Thread 0x7f7bb085f700 (LWP 7508)]
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc  .so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc  .so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f7bb0939ae3 in poll () from /lib/x86_64-linux-gnu/libc.so.6

CodePudding user response：

$ make -j8
$ make install

It is exceedingly likely that make install runs strip on the binary before installing it into /usr/loca/bin (that's the standard procedure).

What you want to do is use the "as built" binary (which presumably still has debug info) to debug the process. Something like:

gdb ~/bld/.../mysqld -p 2667

This tutorial suggests that the gdb output after ...
Reading symbols from /home/cbell/source/bzr/mysql-5.6/sql/mysqld...done.

Note that they are not running installed binary, but the binary from their build tree.

CodePudding user response：

Here's the short answer. You need both -DWITH_DEBUG and -DWITH_VALGRIND.

cmake /vagrant/mysql-server/ -DDOWNLOAD_BOOST=1 -DWITH_BOOST=$HOME/my_boost \
    -DWITH_DEBUG=1 -DWITH_INNODB_EXTRA_DEBUG=1 -DWITH_VALGRIND=1

(Note: The MySQL documentation makes no mention of requiring VALGRIND for step debugging, as far as I can see)

It appears that -DWITH_DEBUG alone is more for debugging crashes, and it seems to provide symbols for system calls. Compiling with the -DWITH_VALGRIND flag appears to be what gets the vast quantity of internals' symbols loaded.

Running pretty much all these commands with sudo.

apt update
apt upgrade -y
apt install libssl-dev libncurses5-dev pkg-config bison build-essential –y
cd ~
mkdir bld
cd bld
################
## BUILD STEP ##
################
cmake /vagrant/mysql-server/ -DDOWNLOAD_BOOST=1 -DWITH_BOOST=$HOME/my_boost \
    -DWITH_DEBUG=true -DWITH_INNODB_EXTRA_DEBUG=true -DWITH_VALGRIND
# /vagrant is mounted from Windows host OS, contains the mysql git repo
make -j8
# make install # NOT NEEDED FOR DEBUGGING
###
groupadd mysql
useradd -r -g mysql -s /bin/false mysql
# cd /usr/local/mysql-test # instead, just go to build/mysql-test/
# or find . -name 'mysql-test'
cd ~/bld/mysql-test/
mkdir mysql-files
chown mysql:mysql mysql-files
chmod 750 mysql-files
###
~/bld/bin/mysqld --initialize-insecure --user=mysql
# I dont need "secure" on local VM    
##################################
# START THE SERVER FOR DEBUGGING #
##################################
# running mysqld directly - no mysqld_safe in bld
~/bld/bin/mysqld --user=mysql --gdb=1 --debug=1 --thread_cache_size=5
# get mysqld PID
vagrant@vagrant:/vagrant/mysql-server/sql$ ps aux | grep mysql
root       41292  0.0  0.0  11860  2112 pts/1    S    Feb12   0:00 sudo ./mysqld --user=mysql --gdb=1
mysql      41293  0.0  0.2 1819176 27684 pts/1   tl   Feb12   0:27 ./mysqld --user=mysql --gdb=1   
vagrant    44081  0.0  0.0   9032   736 pts/1    S    02:11   0:00 grep --color=auto mysql

# START GDB and attach to process owned by user mysql
gdb
(gdb) attach  41293

Notes: Although I've had issues with folders mounted from windows in vagrant in the past, that ended up being a non-issue here. It seems that VALGRIND was all I was really missing.

I'd also note that the build time on my vagrant machine took a long time with VALGRIND, and results in 16G size. With just VALGRIND (just DEBUG), the size was only 4.5G. Using a c5.2xlarge instance on AWS EC2, I was able to build much faster (down to 20-30 minutes vs over 1 hour).