Using strace and ltrace to help with troubleshooting on Linux

Both strace and ltrace are powerful command-line tools for debugging and troubleshooting programs on Linux: Strace captures and records all system calls made by a process as well as the signals received, while ltrace does the same for library calls.

If a program acts differently than you expect, you can use these tools to see “behind the curtain” and maybe get some clues as to what is going on.

Be forewarned, though. When you use either of these commands, you will end up with a lot of output to look through. Still, that can tell you quite a bit about how a process is working and sometimes give you important insights.

strace

To run strace against a program, use a command like what is shown below—the strace command followed by the program name. The output below has been cut short.

$ strace who
execve(“/usr/bin/who”, [“who”], 0x7ffe889f45c0 /* 41 vars */) = 0
brk(NULL)                               = 0x55e7d6720000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fff38c72b20) = -1 EINVAL (Invalid argument)
access(“/etc/ld.so.preload”, R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, “/etc/ld.so.cache”, O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, “”, {st_mode=S_IFREG|0644, st_size=60195, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 60195, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd1ba4d7000
close(3)
…

The command above runs the who command and reports on the system calls that it makes. The start of each line (e.g., execve, brk and arch_prctl) shows the system call being made.

To run strace against a running process to see what it’s doing, use the command with the -p option followed by the process ID. Note that the strace output starts in the 3rd line below after the process is attached.

$ strace -p 34512
strace: Process 34512 attached
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 34516
…

NOTE: You cannot redirect the output of the strace command through a pipe, but you can have it sent to a file using the -o option. In the example below, the date command output is shown. After that, the top of the output file is displayed using the head command.

$ strace -o outputfile date
Tue May  3 03:33:52 PM EDT 2022
$ head -10 outputfile
execve(“/usr/bin/date”, [“date”], 0x7ffc30f3bd00 /* 41 vars */) = 0
brk(NULL)                               = 0x55e78c077000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe52bf2ec0) = -1 EINVAL (Invalid argument)
access(“/etc/ld.so.preload”, R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, “/etc/ld.so.cache”, O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, “”, {st_mode=S_IFREG|0644, st_size=60195, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 60195, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f589661b000
close(3)                                = 0
openat(AT_FDCWD, “/lib64/libc.so.6”, O_RDONLY|O_CLOEXEC) = 3
read(3, “177ELF2113