GDB: Dissecting Programs

This article is part of the BUAA Operating Systems course preparatory tutorial. This version was written by me.

In 2024, the course lab environment was changed from GXemul to QEMU. To help students adapt to the new environment, two new articles were added to the preparatory tutorial: “GDB: Dissecting Programs” and “Introduction to the QEMU Emulator.”

Think back to when you first entered the world of programming. Everyone has had this experience: a carefully written program just won’t produce the correct result, and even after reviewing the code from top to bottom several times, the hidden bug remains elusive. You may know your code inside out, but code is ultimately static — it can’t reflect what actually happens at runtime. Various test cases may help you discover errors, but the program remains a black box with only inputs and outputs, leaving its inner workings a mystery.

You’ve surely tried many ways to escape this predicament — the famous “printf” method, for instance. But inserting meaningless output into existing logic only makes the code structure messier, and excessive output may further obscure the truth, taking you further from finding the bug. We need a different approach — one that can trace the program’s execution without intruding on the code’s original logic, thereby discovering runtime errors.

Introducing GDB

A program that can trace and control another program’s execution is called a debugger. Different languages have different debuggers — Python has PDB, Java has JDB. The one we’ll cover in this article is GDB, the “GNU Debugger.”

GDB’s mascot — an archerfish. It’s skilled at shooting water jets to knock insects (bugs) off plants by the water’s edge.

GDB is an extremely powerful debugger, supporting C, C++, Go, Rust, and many other languages. It was originally written by GNU Project founder Richard Matthew Stallman as part of the GNU Project. According to the GDB website, GDB’s main features include:

• Starting a program and specifying anything that may affect its behavior.
• Making the program stop under specified conditions.
• Examining what happened when the program stopped.
• Changing things in the program so you can experiment with correcting one bug and continue investigating another.

We’ll gradually explore these features below, seeing how GDB dissects a program’s inner workings like a scalpel to locate the root cause of bugs.

Preparation

Two notes before we begin:

1. The following content will be done on Ubuntu, matching this course’s lab environment. I also recommend using Linux as much as possible for learning and development, since many projects only support Linux or only provide Linux tutorials and documentation.
2. To better understand GDB commands, you should follow along and practice while reading.

The course’s jump server comes with all required environments pre-installed, so you can also use it to follow along. However, the jump server may have issues with setting breakpoints due to being unable to disable address space randomization. This can be resolved by entering set disable-randomization off in the GDB interface.

Installing GDB

GDB should be pre-installed on Ubuntu. Check with:

$ gdb -v       
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

If not installed:

$ apt-get update
$ apt-get install gdb

Loading a Program

Let’s use a simple program as an example:

// adds.c
#include <stdio.h>

int main() {
    int n;
    scanf("%d", &n);

    int c = 0;
    for (int i = 1; i <= n; i++) {
        c += i;
        printf("%d, ", c);
    }
    printf("\n");

    return 0;
}

From the earlier compilation tutorial, we know we can compile this with:

$ gcc adds.c -o adds

But this produces a binary stripped of runtime-debugging information, making debugging impossible. We need to add the -g flag, which preserves additional information for the debugger. Without and with -g, the outputs are the program’s Release build and Debug build, respectively.

$ gcc -g adds.c -o adds

So using a debugger also changes the program, but unlike the printf method which requires modifying source code, the debug build doesn’t alter the original code logic — the developer doesn’t need to care. In other words, debug mode is “transparent” to the developer.

Now we can use GDB on the program. There are two ways to tell GDB which program to debug:

• Specify it directly on the command line. This enters GDB’s interactive interface with adds as the program to debug:

  $ gdb adds
  

• Start GDB first, then load the executable with file <executable>:

  $ gdb
  (gdb) file adds
  

If you see this, GDB has successfully loaded adds:

Reading symbols from adds...

For now, let’s skip tracing and just let the program run to completion. Use the run command:

Reading symbols from adds...
(gdb) run
Starting program: /your/path/to/program/adds 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
10
1, 3, 6, 10, 15, 21, 28, 36, 45, 55, 
[Inferior 1 (process 85435) exited normally]

Standard I/O also works in the GDB environment.

Type quit to exit GDB.

Passing Arguments

The above is the simplest case. Often you need to pass arguments to the program. Let’s use an echo implementation:

// echo.c
#include <stdio.h>
#include <stdlib.h>

void usage() {
    printf("usage: echo <string>\n");
    exit(-1);
}

int main(int argc, char *argv[]) {
    if (argc != 2) {
        usage();
    }

    printf("%s\n", argv[1]);
    return 0;
}

Note: use ./echo, not echo:

$ gcc -g echo.c -o echo 
$ ./echo 'hello, world!'
hello, world!

Our echo program needs an argument. Like loading a program, there are two ways:

• Pass arguments on the command line. For ./echo 'hello, world!', the GDB equivalent is:

  $ gdb --args ./echo 'hello, world!'
  

  Note that --args includes the program ./echo.

• Set arguments inside GDB with set args [arg]...:

  $ gdb
  (gdb) file ./echo
  (gdb) set args 'hello, world!'
  

Now the program is properly loaded into GDB. Use run to execute it:

(gdb) run
Starting program: /your/path/to/program/echo 'hello, world!'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
hello, world!
[Inferior 1 (process 84663) exited normally]

Actually, set args [arg]... and run can be combined: run [arg]....

Let’s summarize the commands so far:

Step by Step: Tracing Execution

If all GDB did was make running programs more tedious, we might as well not use it. So let’s explore GDB’s first real feature: tracing program execution.

Tracing means seeing each instruction as it executes. Let’s use this Fibonacci program:

// fibo.c
#include <assert.h>
#include <stdio.h>

int fibo(int n) {
    assert(n > 0);
    if (n <= 2) {
        return 1;
    } else {
        return fibo(n - 1) + fibo(n - 2);
    }
}

int main(int argc, char *argv[]) {
    int n = 10;
    int fibo_n = fibo(n);
    printf("fibo(%d) = %d\n", n, fibo_n);

    return 0;
}

Compile and load into GDB:

$ gcc -g fibo.c -o fibo
$ gdb ./fibo

This time, instead of run, we use start. start makes the program stop at the beginning of main:

(gdb) start
Temporary breakpoint 1 at 0x11eb: file fibo.c, line 16.
Starting program: /your/path/to/program/fibo 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Temporary breakpoint 1, main (argc=1, argv=0x7fffffffd9a8) at fibo.c:15
15          int n = 10;
(gdb)

Like run, start also accepts arguments: start [arg]....

Notice 15 int n = 10; — this is the next line to be executed. Checking the source, this is indeed the first statement in main.

To execute step by step, use step:

(gdb) step
16          int fibo_n = fibo(n);
(gdb) 

Press Enter (no need to retype step) to execute the next step. We enter fibo with n = 10:

(gdb) 
fibo (n=10) at fibo.c:6
6           assert(n > 0);
(gdb) 

Two more Enters and we’re about to execute return fibo(n - 1) + fibo(n - 2);:

(gdb) 
7           if (n <= 2) {
(gdb) 
10              return fibo(n - 1) + fibo(n - 2);
(gdb)

Another Enter would call fibo(n - 1), recursing into fibo with n = 9:

(gdb) 
fibo (n=9) at fibo.c:6
6           assert(n > 0);
(gdb) 

This isn’t ideal — truly stepping through would endlessly recurse into called functions. We need to skip over function internals.

First, let’s exit fibo(9). The command for exiting the current function is finish. The output shows we didn’t descend into the fibo(9) -> fibo(8) -> fibo(7)... recursion — fibo(9) completed as if executed in one go, returning $1 = 34:

(gdb) finish
Run till exit from #0  fibo (n=9) at fibo.c:6
0x00005555555551c1 in fibo (n=10) at fibo.c:10
10              return fibo(n - 1) + fibo(n - 2);
Value returned is $1 = 34
(gdb)

We’ve exited fibo(9), but if we use step again, we’d enter fibo(8). So we introduce next — it executes the current line while skipping over any function calls within it.

After next, GDB shows line 12, the closing brace. What does this mean?

(gdb) next
12      }
(gdb) 

This represents the moment after the last instruction in the function but before returning. Since GDB shows the next line to execute, the result of the last instruction isn’t visible until the next line. But after the function’s last instruction, the function should return, at which point the function’s state is no longer accessible. So to represent “after the last instruction,” GDB uses the closing brace line.

One more next (or step) truly exits fibo(10). I’ll press Enter to repeat next:

(gdb) 
main (argc=1, argv=0x7fffffffd9a8) at fibo.c:17
17          printf("fibo(%d) = %d\n", n, fibo_n);
(gdb) 

To skip into printf, press Enter again. The program correctly outputs fibo(10) = 55:

(gdb) 
fibo(10) = 55
19          return 0;
(gdb) 

We can’t actually step into printf since we don’t have its source code.

At this point we don’t need to trace further. Options:

• continue — resume normal execution until the program exits:

  (gdb) continue
  Continuing.
  [Inferior 1 (process 102673) exited normally]
  

• kill — kill the program process:

  (gdb) kill
  Kill the program being debugged? (y or n) y
  [Inferior 1 (process 108565) killed]
  

• start — restart the program:

  (gdb) start
  The program being debugged has been started already.
  Start it from the beginning? (y or n) y
  ...
  

By the way, if you’ve used an editor or IDE’s debugging feature, you’ll recognize those universal debugger buttons. They correspond directly to the commands in this section. Here are VS Code’s buttons, from left to right:

1. Continue — continue
2. Step Over — next
3. Step Into — step
4. Step Out — finish
5. Restart — start (while running)
6. Stop — kill

Summary of this section:

The Heavy Sword: Breakpoints

Step-by-step execution shows every detail, but when debugging we usually only care about a few potentially buggy modules. We want the program to run freely through uninteresting parts. But how does a freely-running program know where the developer wants to stop? This is where breakpoints come in. A breakpoint marks certain lines — when execution reaches that point and conditions are met, control transfers from automatic execution back to GDB. Breakpoints seem simple but are incredibly versatile; they are the debugger’s most important feature.

Regular Breakpoints

Let’s modify fibo.c and use it as an example:

// fibo.c
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>

int fibo(int n) {
    assert(n > 0);
    if (n <= 2) {
        return 1;
    } else {
        return fibo(n - 1) + fibo(n - 2);
    }
}

void usage() {
    printf("usage: fibo <number>");
    exit(-1);
}

int main(int argc, char *argv[]) {
    if (argc != 2) {
        usage();
    }

    int n = atoi(argv[1]);

    for (int i = 1; i <= n; i++) {
        int fibo_i = fibo(i);
        printf("fibo(%d) = %d\n", i, fibo_i);
    }

    return 0;
}

Compile and enter GDB:

$ gcc -g fibo.c -o fibo 
$ gdb fibo

Suppose we want to skip ahead and stop at the first fibo call, then step into it. How?

We need a breakpoint at int fibo_i = fibo(i);. But we may not know the line number yet. First, use list [position] to view source code. Usage: list, list <line>, list <file>:<line>, list <function>, or list <file>:<function>. Pressing Enter after list shows more source.

Using list main then Enter reveals line 28:

(gdb) list main
...
25          int n = atoi(argv[1]);
26
27          for (int i = 1; i <= n; i++) {
28              int fibo_i = fibo(i);
29              printf("fibo(%d) = %d\n", i, fibo_i);
30          }
...
(gdb)

Now set the breakpoint with break 28:

(gdb) break 28
Breakpoint 1 at 0x1280: file fibo.c, line 28.

Run ./fibo 10 with run 10. Without breakpoints, run would complete immediately. But now it stops at the breakpoint:

(gdb) run 10
Starting program: /your/path/to/program/fibo 10
...

Breakpoint 1, main (argc=2, argv=0x7fffffffd988) at fibo.c:28
28              int fibo_i = fibo(i);
(gdb) 

Now step enters fibo:

(gdb) step
fibo (n=1) at fibo.c:7
7           assert(n > 0);
(gdb) 

If we continue now, what happens? Since the breakpoint is inside the loop, the program stops at it again after one iteration, having printed fibo(1) = 1:

(gdb) continue
Continuing.
fibo(1) = 1

Breakpoint 1, main (argc=2, argv=0x7fffffffd988) at fibo.c:28
28              int fibo_i = fibo(i);
(gdb) 

To avoid stopping repeatedly, we can delete or disable the breakpoint. Use info breakpoints to see all breakpoints:

(gdb) info breakpoints 
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000555555555280 in main at fibo.c:28
        breakpoint already hit 2 times

The breakpoint number is 1. Use disable breakpoints 1 to disable it:

(gdb) disable breakpoints 1
(gdb) info breakpoints 
Num     Type           Disp Enb Address            What
1       breakpoint     keep n   0x0000555555555280 in main at fibo.c:28
        breakpoint already hit 2 times

Re-enable with enable breakpoints [num].... To permanently delete: delete breakpoints 1. After deletion, continue lets the program finish:

(gdb) delete breakpoints 1
(gdb) continue
Continuing.
fibo(2) = 1
...
fibo(10) = 55
[Inferior 1 (process 109274) exited normally]

Temporary Breakpoints

Restart with start 10. Notice “Temporary breakpoint” — start is essentially a breakpoint at main:

(gdb) start 10
Temporary breakpoint 2 at 0x555555555251: file fibo.c, line 21.
...

A temporary breakpoint stops the program exactly once, then auto-deletes. Use tbreak (same syntax as break):

(gdb) tbreak 27
Temporary breakpoint 3 at 0x555555555277: file fibo.c, line 27.
(gdb) continue
Continuing.

Temporary breakpoint 3, main (...) at fibo.c:27
27          for (int i = 1; i <= n; i++) {

So start [arg]... equals tbreak main + run [arg]....

After hitting it, the temporary breakpoint disappears:

(gdb) info breakpoints 
No breakpoints or watchpoints.

Conditional Breakpoints

Breakpoints as described are powerful, but still insufficient for tracing. What if we want to stop only on the 9th iteration of the loop? A regular breakpoint would stop every iteration; a temporary breakpoint would stop only once. We need something in between: a conditional breakpoint.

A conditional breakpoint only stops when a condition is true. Create one with break [position] if <condition>. <condition> should be a valid expression in the current context, like a >= b + 1 where a and b are defined. A simple rule: if adding if (<condition>){} wouldn’t cause compile or runtime errors, the expression is usable.

Similarly: tbreak <position> if <condition> for conditional temporary breakpoints.

Let’s use break 28 if i == 9. After continue, the output confirms it stopped on the 9th iteration:

(gdb) break 28 if i == 9
Breakpoint 4 at 0x555555555280: file fibo.c, line 28.
(gdb) continue
Continuing.
fibo(1) = 1
...
fibo(8) = 21

Breakpoint 4, main (...) at fibo.c:28
28              int fibo_i = fibo(i);

We can also modify a breakpoint’s condition with condition <num> <condition>. This turns a regular breakpoint into a conditional one, or changes an existing condition. Change ours to i == 10:

(gdb) condition 4 i == 10
(gdb) continue
Continuing.
fibo(9) = 34

Breakpoint 4, main (...) at fibo.c:28
28              int fibo_i = fibo(i);

For loops without explicit counters (linked list traversal, iterators), use ignore <num> <times> to skip the first <times> hits. Combined with a temporary breakpoint: tbreak <position> then ignore <num> <K-1>.

Multiple breakpoints can now exist at the same location. Use clear <position> to remove all breakpoints at a location.

Watchpoints

Conditional breakpoints give more flexibility, but they still require a specific location — meaning you must roughly locate the bug before setting breakpoints. For small programs this is feasible; for large programs with heavy memory operations, the point of failure often differs from the bug’s origin. Regular conditional breakpoints struggle here.

GDB introduces watchpoints. A watchpoint specifies an expression (no location needed). GDB continuously monitors the expression’s value; when its value changes, execution stops. Think of watchpoints as special conditional breakpoints — condition-only, no location — where the condition is “the expression’s value changed.”

Let’s revisit the “stop on the 9th iteration” problem using watchpoints instead.

First, enter the loop:

$ gdb fibo
...
(gdb) tbreak 27
(gdb) run 10
...
Temporary breakpoint 1, main (...) at fibo.c:27
27          for (int i = 1; i <= n; i++) {

Set a watchpoint: watch i >= 9. When entering the 9th iteration, i >= 9 changes from false to true, triggering the stop:

(gdb) watch i >= 9
Hardware watchpoint 2: i >= 9
(gdb) continue
Continuing.
fibo(1) = 1
...
fibo(8) = 21

Hardware watchpoint 2: i >= 9

Old value = 0
New value = 1
0x00005555555552ad in main (...) at fibo.c:27
27          for (int i = 1; i <= n; i++) {

Two steps confirm we’re on the 9th iteration:

(gdb) step
28              int fibo_i = fibo(i);
(gdb) 
fibo (n=9) at fibo.c:7
7           assert(n > 0);

GDB also offers read watchpoints (rwatch <expression> — stop when the expression is read) and access watchpoints (awatch <expression> — stop when the expression is read or written). Their usage is similar to regular watchpoints.

Note the difference between a watchpoint’s value changing and an access watchpoint’s write.

Watchpoints need not be conditional expressions. They’re commonly used for debugging memory issues, often with pointer or address expressions. Here’s an example with a dangling pointer:

// watch.c
#include <stdio.h>
#include <stdlib.h>

void mysterious_func(int **pptr) {
    free(*pptr);
    // *pptr = NULL;
}

int main() {
    int *ptr = NULL;

    ptr = malloc(sizeof(int));
    *ptr = 10;
    printf("value is %d\n", *ptr);

    mysterious_func(&ptr);
    if (ptr != NULL) {
        printf("value is %d\n", *ptr);
    }

    return 0;
}

The missing *pptr = NULL leaves a dangling pointer. The output shows garbage:

$ gcc -g ./watch.c -o watch
$ ./watch 
value is 10
value is 1648141505

A watchpoint easily locates the issue:

$ gdb watch
(gdb) start
(gdb) watch *ptr
(gdb) continue
(gdb) continue
(gdb) continue
Continuing.
value is 10

Hardware watchpoint 2: *ptr

Old value = 10
New value = 1431655769
tcache_put (tc_idx=0, chunk=0x555555559290) at ./malloc/malloc.c:3160

For now we only know the bad value comes from malloc.c. With the call stack (next section), we can trace back to mysterious_func.

Another common bug: array out-of-bounds. Example:

// watch2.c
#include <stdio.h>

int careless_sum(int arr[], int n) {
    int i = 0, total = 0;
    while (i < n) {
        total += arr[++i];
        // total += arr[i++];
    }
    return total;
}

int main() {
    int arr[10] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
    printf("total = %d\n", careless_sum(arr, 10));
    return 0;
}

Using ++i instead of i++ causes out-of-bounds access:

$ gcc -g ./watch2.c -o watch2
$ ./watch2
total = -3779795

To debug this, watch when arr[10] (the first 4 bytes after the array) is accessed:

$ gdb watch2
(gdb) start
(gdb) next
(gdb) awatch arr[10]
Hardware access (read/write) watchpoint 2: arr[10]
(gdb) continue
Continuing.

Hardware access (read/write) watchpoint 4: arr[10]

Value = 1896377344
careless_sum (arr=0x7fffffffd910, n=10) at ./watch2.c:7
7               total += arr[++i];

Summary of this section: