Debugging OpenSSL with GDB and ltrace
This article is to note to debug OpenSSL with GDB. I use the case of this issue on OpenSSL making a segmentation fault on OpenSSL version 3.0.1, then fixed in OpenSSL version 3.0.2.
My environment
Fedora 37
GCC version 12.2.1
$ gcc --version gcc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-4) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
GDB 12.1
$ gdb --version GNU gdb (GDB) Fedora Linux 12.1-6.fc37 Copyright (C) 2022 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
OpenSSL 3.0.8
$ openssl version OpenSSL 3.0.8 7 Feb 2023 (Library: OpenSSL 3.0.8 7 Feb 2023)
An example of debugging OpenSSL
Initially I was trying to reproduce the issue with OpenSSL 3.0.1 built from the source. However I saw one runtime error related to one of the dependency library of OpenSSL when running the binary with GDB, I just explain with the system OpenSSL 3.0.8. The segmentation fault issue doesn't happen on the version. However it just explain how to debug on the OpenSSL.
Here is a minimal reproducing script to check the issue above.
$ cat repro.c
#include <openssl/hmac.h>
#define CF_CHECK_EQ(expr, res) if ( (expr) != (res) ) { goto end; }
#define CF_CHECK_NE(expr, res) if ( (expr) == (res) ) { goto end; }
int main(void)
{
const unsigned char key[24] = {0};
const unsigned char ct[166] = {0};
const EVP_MD* md = NULL;
EVP_PKEY *pkey = NULL;
EVP_MD_CTX* ctx = EVP_MD_CTX_new();
EVP_MD_CTX* ctx_tmp = NULL;
CF_CHECK_NE(pkey = EVP_PKEY_new_mac_key(EVP_PKEY_HMAC, NULL, key, sizeof(key)), NULL);
CF_CHECK_EQ(EVP_DigestSignInit(ctx, NULL, EVP_sha256(), NULL, pkey), 1);
ctx_tmp = EVP_MD_CTX_new();
CF_CHECK_EQ(EVP_MD_CTX_copy(ctx_tmp, ctx), 1);
EVP_MD_CTX_free(ctx);
ctx = ctx_tmp;
CF_CHECK_EQ(EVP_DigestSignUpdate(ctx, ct, sizeof(ct)), 1);
end:
return 0;
}
Compile with GCC
When I debug a code, I find the following command is useful.
$ gcc -g3 -ggdb3 -gdwarf-5 $(pkgconf --cflags --libs openssl) -o repro repro.c $ echo $? 0 $ ls -l repro -rwxr-xr-x. 1 jaruga jaruga 413600 Feb 17 17:46 repro*
Let's see each part of the command line one by one.
The -g* options are for debug. You can check it by man gdb or on the GDB official manual page - Debugging Options.
In my opinion, specifying the number of the debug options are a good practice, because you can be conscious about the which level (for -glevel and -ggdblevel) or version (-gdwarf-version) of the debug options. In new gcc version, the number can be default value. However when you work with a variety of gcc versions, specifying the level or version is useful.
To make this sustainable, you can set the alias below in your .bashrc.
$ alias gcc-debug='gcc -g3 -ggdb3 -gdwarf-5'
You can check the used DWARF version by the command below.
$ objdump -W ./repro | grep 'DWARF Version' DWARF Version: 5
For the pkgconf --cflags --libs openssl, the --cflags is to print the -I options, and --iibs is to print the -l options of the compiler options built to create the package.
You can see like this.
$ pkgconf --list-all | grep openssl openssl OpenSSL - Secure Sockets Layer and cryptography libraries and tools $ pkgconf --cflags --libs openssl -lssl -lcrypto
When running the reproducer with system OpenSSL 3.0.8, it exists successfully without error.
$ ./repro $ echo $? 0
Build OpenSSL 3.0.1 from the source
Next, let's install OpenSSL 3.0.1 to compare the issue by running the reproducing script. Check the INSTALL.md on the OpenSSL repository.
$ git clone https://github.com/openssl/openssl.git
$ cd openssl
$ git checkout openssl-3.0.1
$ ./Configure \
--prefix=${HOME}/local/openssl-3.0.1 \
linux-x86_64
$ make
You can run the make test optionally. I saw some test failures. However, I moved forward.
$ make test ... Test Summary Report ------------------- 80-test_ssl_new.t (Wstat: 256 (exited 1) Tests: 30 Failed: 1) Failed test: 12 Non-zero exit status: 1 Files=242, Tests=3281, 364 wallclock secs ( 6.11 usr 0.58 sys + 296.88 cusr 57.21 csys = 360.78 CPU) Result: FAIL make[1]: *** [Makefile:3252: run_tests] Error 1 make[1]: Leaving directory '/home/jaruga/var/git/openssl' make: *** [Makefile:3248: tests] Error 2
$ make install $ ~/local/openssl-3.0.1/bin/openssl version OpenSSL 3.0.1 14 Dec 2021 (Library: OpenSSL 3.0.8 7 Feb 2023)
Let's compile with OpenSSL 3.0.1.
$ gcc -g3 -ggdb3 -gdwarf-5 -lssl -lcrypto -I$HOME/local/openssl-3.0.1/include -L$HOME/local/openssl-3.0.1/lib64 -o repro-with-openssl-3.0.1 repro.c $ echo $? 0 $ ls -l repro-with-openssl-3.0.1 -rwxr-xr-x. 1 jaruga jaruga 412960 Feb 17 18:10 repro-with-openssl-3.0.1*
Check the link info. As a default, the binary links to the system OpenSSL 3.0.8.
$ ldd repro-with-openssl-3.0.1
linux-vdso.so.1 (0x00007fff1c4e1000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007f6407811000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f6407200000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6407635000)
libz.so.1 => /lib64/libz.so.1 (0x00007f64071e6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f64078cd000)
With the LD_LIBRARY_PATH, it links to the OpenSSL 3.0.1.
$ LD_LIBRARY_PATH=$HOME/local/openssl-3.0.1/lib64 ldd repro-with-openssl-3.0.1
linux-vdso.so.1 (0x00007fff32507000)
libssl.so.3 => /home/jaruga/local/openssl-3.0.1/lib64/libssl.so.3 (0x00007f12fcea2000)
libcrypto.so.3 => /home/jaruga/local/openssl-3.0.1/lib64/libcrypto.so.3 (0x00007f12fca00000)
libc.so.6 => /lib64/libc.so.6 (0x00007f12fc824000)
/lib64/ld-linux-x86-64.so.2 (0x00007f12fcf4c000)
The binary exits successfully without error.
$ ./repro-with-openssl-3.0.1 $ echo $? 0
And the binary exists with segmentation fault as expected.
$ LD_LIBRARY_PATH=$HOME/local/openssl-3.0.1/lib64 ./repro-with-openssl-3.0.1 Segmentation fault (core dumped)
Prepare GDB setting
When debugging the binary with system OpenSSL 3.0.8 on the GDB, it can be like this.
$ gdb -q -ex "set breakpoint pending on" -ex "set debuginfod enabled on" --args ./repro
Reading symbols from ./repro...
(gdb) b main
Breakpoint 1 at 0x401191: file repro.c, line 8.
(gdb) r
Starting program: /home/jaruga/doc/memo/20230227_day_of_learning/openssl_test/repro
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Breakpoint 1, main () at repro.c:8
8 const unsigned char key[24] = {0};
However, to make this step easy and sustainable, you can set the init files. The set startup-quietly on is the same with the gdb -q. See man gdbinit.
$ cat ~/.config/gdb/gdbinit set breakpoint pending on set debuginfod enabled on $ cat ~/.config/gdb/gdbearlyinit set startup-quietly on
Or the following alias might also be useful.
$ alias gdb-args='gdb -q -ex "set breakpoint pending on" -ex "set debuginfod enabled on" --args'
Debug with GDB
So, you can run the gdb simply. I ran b main (break main), r (run), then c (continue), and bt (backtrace).
$ gdb --args ./repro
Reading symbols from ./repro...
(gdb) b main
Breakpoint 1 at 0x401191: file repro.c, line 8.
(gdb) r
Starting program: /home/jaruga/doc/memo/20230227_day_of_learning/openssl_test/repro
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Breakpoint 1, main () at repro.c:8
8 const unsigned char key[24] = {0};
(gdb)
I faced the following issue when debugging with the source-built OpenSSL 3.0.1. It seems an issue related to openldap. So, I couldn't debug this case.
$ LD_LIBRARY_PATH=$HOME/local/openssl-3.0.1/lib64 gdb --args ./repro-with-openssl-3.0.1 gdb: symbol lookup error: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0 $ echo $? 127
Debug with ltrace
The ltrace is library call tracer. It's useful when you debug libraries.
Here is the installed version on my environment.
$ sudo dnf install ltrace $ rpm -q ltrace ltrace-0.7.91-46.fc37.x86_64 $ ltrace --version ltrace 0.7.91 Copyright (C) 2010-2013 Petr Machata, Red Hat Inc. Copyright (C) 1997-2009 Juan Cespedes <cespedes@debian.org>. License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
Then run the commands.
The producer exists successfully with the system OpenSSL 3.0.8 libraries.
$ ltrace ./repro EVP_MD_CTX_new(1, 0x7ffd7cc7c3f8, 0x7ffd7cc7c408, 0x403df8) = 0x17f92a0 EVP_PKEY_new_mac_key(855, 0, 0x7ffd7cc7c2a0, 24) = 0x180e530 EVP_sha256(7, 0x17f9010, 6170, 10) = 0x7ff59b3e4ec0 EVP_DigestSignInit(0x17f92a0, 0, 0x7ff59b3e4ec0, 0) = 1 EVP_MD_CTX_new(0x1818710, 0, 0, 0x18246d0) = 0x1824750 EVP_MD_CTX_copy(0x1824750, 0x17f92a0, 0x17f92a0, 0x1824750) = 1 EVP_MD_CTX_free(0x17f92a0, 0x18245b0, 24, 0x1824ba0) = 0 EVP_DigestSignUpdate(0x1824750, 0x7ffd7cc7c1f0, 166, 0x7ffd7cc7c1f0) = 1 +++ exited (status 0) +++
The segmentation fault happens with the OpenSSL 3.0.1 libraries.
$ LD_LIBRARY_PATH=$HOME/local/openssl-3.0.1/lib64 ltrace ./repro EVP_MD_CTX_new(1, 0x7fff121e58d8, 0x7fff121e58e8, 0x403df8) = 0xbee2a0 EVP_PKEY_new_mac_key(855, 0, 0x7fff121e5780, 24) = 0xc027f0 EVP_sha256(7, 0xbee010, 3117, 10) = 0x7f07b4815ba0 EVP_DigestSignInit(0xbee2a0, 0, 0x7f07b4815ba0, 0) = 1 EVP_MD_CTX_new(0xc2cc00, 0, 0, 0xc472d0) = 0xc02e20 EVP_MD_CTX_copy(0xc02e20, 0xbee2a0, 0xbee2a0, 0xc02e20) = 1 EVP_MD_CTX_free(0xbee2a0, 0, 0xbee2a0, 0xc02e20) = 0 EVP_DigestSignUpdate(0xc02e20, 0x7fff121e56d0, 166, 0x7fff121e56d0 <no return ...> --- SIGSEGV (Segmentation fault) --- +++ killed by SIGSEGV +++
$ LD_LIBRARY_PATH=$HOME/local/openssl-3.0.1/lib64 ltrace ./repro-with-openssl-3.0.1 EVP_MD_CTX_new(1, 0x7fff718b5828, 0x7fff718b5838, 0x403df8) = 0x8a22a0 EVP_PKEY_new_mac_key(855, 0, 0x7fff718b56d0, 24) = 0x8b67f0 EVP_sha256(7, 0x8a2010, 2273, 10) = 0x7fbdd2015ba0 EVP_DigestSignInit(0x8a22a0, 0, 0x7fbdd2015ba0, 0) = 1 EVP_MD_CTX_new(0x8e0c00, 0, 0, 0x8fb2d0) = 0x8b6e20 EVP_MD_CTX_copy(0x8b6e20, 0x8a22a0, 0x8a22a0, 0x8b6e20) = 1 EVP_MD_CTX_free(0x8a22a0, 0, 0x8a22a0, 0x8b6e20) = 0 EVP_DigestSignUpdate(0x8b6e20, 0x7fff718b5620, 166, 0x7fff718b5620 <no return ...> --- SIGSEGV (Segmentation fault) --- +++ killed by SIGSEGV +++
That's all. Happy debugging!
Exploring a DIY split keyboard and CAD file
I am testing my 24 keys keyboard layout on Moonlander. And I think it's time to consider buying a new "DIY" split keyboard. Below is my current preference.
A new DIY split keyboard
- PCB: Sweep Bling LP (GitHub, Shop) or Aurora Sweep is a candidate. Because my 24 keys layout is similar to the 34 keys. And I want to use the hot-swappable Kailh Low Profile Choc V1 key switches.
- Key switches: Kailh Low Profile Choc V1 - Light Blue (20gf, Linear) or the closest one to the Framework Laptop's key switches.
- Keycaps: Kailh Choc V1 flat black or/and flat transparent
- Case: Silver aluminum or black or none.
- Firmware: QMK over ZMK. Because I want to use mouse emulation and want to have the same experience with the Moonlander.
Take a look at Sweep Bling LP CAD files
However, the PCB is for 34 keys, not for 24 keys. So, how can I create a PCB for 24 keys? Perhaps, I may be able to edit the original 34 keys CAD file for 24 keys, and upload the file to a PCB manufacturer, and get it.
Ben Vallack's this video gave me a clue. For go to https://github.com/davidphilipbarr/Sweep. And here is the PCB of the Sweep Bling LP.
$ git clone https://github.com/davidphilipbarr/Sweep.git $ cd Sweep $ cd 'Sweep Bling LP'/pcb
Install KiCad.
$ sudo dnf install kicad $ pwd /home/jaruga/git/Sweep/Sweep Bling LP/pcb $ kicad &
Then you see the file lists included in the "sweepbling-lp__pcb.kicad_pro" file (the "pro" is a project file?) from the KiCad.
Then click the "sweepbling-lp__pcb.kicad_pcb".
For the PCB manufacturers I can upload the CAD file, JLCPCB and PCBWay were introduced by the video above and a split keyboard community.
I still have no idea about how to edit to update with the 24 keys. :)
