Debug Shared Libraries Loading using LD_DEBUG on Linux

When working with Linux binaries, one common headache is shared libraries loading issues. You might also run into runtime problems where symbol resolution goes wrong, and the wrong version of a library (or no version at all) is being used. These issues can be tricky to diagnose because the dynamic linker (ld-linux.so) is usually invisible - it silently loads and links libraries before the program even starts running. That's where the LD_DEBUG environment variable can be useful. This tutorial explains how to debug shared libraries loading using LD_DEBUG on Linux.

The LD_DEBUG variable controls the amount of information printed by the dynamic linker. First, you can check available options with:

LD_DEBUG=help ls

This prints the list of supported options:

Valid options for the LD_DEBUG environment variable are:

  libs        display library search paths
  reloc       display relocation processing
  files       display progress for input file
  symbols     display symbol table processing
  bindings    display information about symbol binding
  versions    display version dependencies
  scopes      display scope information
  all         all previous options combined
  statistics  display relocation statistics
  unused      determined unused DSOs
  help        display this help message and exit

If you suspect a library isn't being found in the right directory, use the libs option:

LD_DEBUG=libs ls

You'll see exactly where the linker looks for each .so file:

34622: find library=libselinux.so.1 [0]; searching
34622:  search cache=/etc/ld.so.cache
34622:   trying file=/lib/x86_64-linux-gnu/libselinux.so.1
34622: 
34622: find library=libc.so.6 [0]; searching
34622:  search cache=/etc/ld.so.cache
34622:   trying file=/lib/x86_64-linux-gnu/libc.so.6
34622: 
34622: find library=libpcre2-8.so.0 [0]; searching
34622:  search cache=/etc/ld.so.cache
34622:   trying file=/lib/x86_64-linux-gnu/libpcre2-8.so.0
...

To check how much work the linker does while starting the program, use the statistics option:

LD_DEBUG=statistics ls

Sample output:

14938:    runtime linker statistics:
14938:      total startup time in dynamic loader: 204755 cycles
14938:                time needed for relocation: 59539 cycles (29.0%)
14938:                     number of relocations: 337
14938:          number of relocations from cache: 3
14938:            number of relative relocations: 1587
14938:               time needed to load objects: 88255 cycles (43.1%)

This is useful for performance tuning of applications with heavy dynamic linking.

For the most detailed output, use the all option:

LD_DEBUG=all ls

You'll see detailed logs, including symbol lookups and bindings:

15065:    symbol=__vdso_clock_gettime;  lookup in file=linux-vdso.so.1 [0]
15065:    binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_clock_gettime' [LINUX_2.6]
15065:    symbol=__vdso_gettimeofday;  lookup in file=linux-vdso.so.1 [0]
15065:    binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_gettimeofday' [LINUX_2.6]
15065:    symbol=__vdso_time;  lookup in file=linux-vdso.so.1 [0]
15065:    binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_time' [LINUX_2.6]
15065:    symbol=__vdso_getcpu;  lookup in file=linux-vdso.so.1 [0]
15065:    binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_getcpu' [LINUX_2.6]
15065:    symbol=__vdso_clock_getres;  lookup in file=linux-vdso.so.1 [0]
15065:    binding file linux-vdso.so.1 [0] to linux-vdso.so.1 [0]: normal symbol `__vdso_clock_getres' [LINUX_2.6]
15065:    
15065:    file=libselinux.so.1 [0];  needed by ls [0]
15065:    find library=libselinux.so.1 [0]; searching
15065:     search cache=/etc/ld.so.cache
15065:      trying file=/lib/x86_64-linux-gnu/libselinux.so.1
...

This level of detail helps pinpoint which path the linker uses, which library is actually loaded, and how symbols are resolved.