We’ve all learned that a process’s memory consists of the heap and the stack. But this model is still too abstract — it conceals many operating system details. So here’s a brief rundown of process memory management.
Heap Growth
libc provides the malloc function for heap memory allocation. Under the hood, it relies on the brk system call.
From the kernel’s perspective, the heap is a simple structure. It consists of a fixed heap base (the end symbol) and a movable heap top (called the program break). All the kernel needs to do is mark the memory between the heap base and the user-specified program break as valid. The brk system call sets a given address as the heap top.
On top of the brk system call, glibc provides two functions: int brk(void *addr) and void *sbrk(intptr_t increment). The former directly sets the program break address, while the latter adjusts the program break position based on increment.
Here’s an example using sbrk() for memory allocation. sbrk() returns the previous program break address before the call, so you need to do some extra arithmetic to get the current address. This semantics makes sense — both brk(N) and malloc(N) return the starting address of the newly allocated memory.
#include <stdio.h>
#include <unistd.h>
extern char end;
int main() {
printf("Address of end symbol: %p\n", (void *)&end);
printf("Current program break: %p\n", sbrk(0));
printf("New program break: %p\n", sbrk(1024) + 1024);
printf("After deallocation: %p\n", sbrk(-512) - 512);
}
Output:
Address of end symbol: 0x55b76abe2040
Current program break: 0x55b77f1aa000
New program break: 0x55b77f1aa400
After deallocation: 0x55b77f1aa200
We can see that end <= program break. The difference here is because glibc has already used heap memory before main().
Also, although the program break address isn’t necessarily page-aligned, heap space is still allocated in pages (of course). This means accessing memory beyond the heap top might not trigger a SIGSEGV. This can lead to subtle bugs.
The following program sets the program break to a page-aligned address plus 1 byte. The entire page can then be accessed. Only after exceeding one page does SIGSEGV occur.
#include <stdio.h>
#include <unistd.h>
int main() {
const size_t PAGE_SIZE = sysconf(_SC_PAGESIZE);
void *origin_ptr = sbrk(0);
size_t origin_ptr_size = (size_t)origin_ptr;
size_t ptr_size_up_aligned =
(origin_ptr_size + PAGE_SIZE - 1) & ~(PAGE_SIZE - 1);
void *ptr_up_aligned = (void *)ptr_size_up_aligned;
// Allocate 1 byte more
brk(ptr_up_aligned + 1);
char *ptr = (char *)ptr_up_aligned;
for (int i = 0; i < PAGE_SIZE; i++) {
ptr[i] = 1;
}
write(1, "ok\n", 3);
ptr[PAGE_SIZE] = 2;
write(1, "not ok\n", 9);
}
Output:
ok
fish: Job 1, './a.out' terminated by signal SIGSEGV (Address boundary error)
Also note that heap space requested via brk isn’t fully mapped to physical memory (also expected). Physical memory allocation only happens on actual access. The following code allocates total pages of heap memory and accesses access of them. We run the program and count page faults with perf.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char *argv[]) {
if (argc < 3) {
fprintf(stderr, "Usage: %s <total> <access>\n", argv[0]);
exit(1);
}
int total = atoi(argv[1]);
int access = atoi(argv[2]);
if (access > total) {
fprintf(stderr, "Access exceeds allocated memory\n");
exit(1);
}
const size_t PAGE_SIZE = sysconf(_SC_PAGESIZE);
char *ptr = sbrk(total * PAGE_SIZE);
for (int i = 0; i < access; i++) {
ptr[i * PAGE_SIZE] = 1;
}
return 0;
}
Results: page-faults grow linearly with the number of pages accessed. This shows that brk doesn’t allocate physical memory — it only marks the corresponding page table entries as valid.
$ sudo perf stat -e 'page-faults' -- ./test_page_fault 10000 0
Performance counter stats for './test_page_fault 10000 0':
54 page-faults
...
$ sudo perf stat -e 'page-faults' -- ./test_page_fault 10000 10
Performance counter stats for './test_page_fault 10000 10':
63 page-faults
...
$ sudo perf stat -e 'page-faults' -- ./test_page_fault 10000 100
Performance counter stats for './test_page_fault 10000 100':
154 page-faults
...
$ sudo perf stat -e 'page-faults' -- ./test_page_fault 10000 1000
Performance counter stats for './test_page_fault 10000 1000':
1,054 page-faults
...
$ sudo perf stat -e 'page-faults' -- ./test_page_fault 10000 10000
Performance counter stats for './test_page_fault 10000 10000':
10,054 page-faults
...
malloc Memory Management
If all you need is heap allocation, sbrk() and malloc() are no different. The real difference lies in the free() function. malloc’s mechanism enables memory reuse, slowing memory usage growth, preventing memory leaks, and reducing the number of brk system calls.
Considering malloc’s implementation is an interesting exercise. malloc manages dynamically growing heap memory, which requires maintaining a dynamically growing data structure — and this data structure itself needs heap memory to store. So malloc must manage not only user allocations but also its own metadata.
glibc’s implementation stores chunk metadata right before the user data. This way, when free() is called, it knows the size of the memory block to release.
chunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size of previous chunk, if unallocated (P clear) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size of chunk, in bytes |A|M|P|
mem-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| User data starts here... .
. .
. (malloc_usable_size() bytes) .
. |
nextchunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (size of chunk, but used for application data) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size of next chunk, in bytes |A|0|1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
For free chunks, the space normally used for user data is repurposed to maintain a free list. When the user calls malloc() to request memory, the allocator can select an appropriate chunk from this free list.
chunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size of previous chunk, if unallocated (P clear) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
`head:' | Size of chunk, in bytes |A|0|P|
mem-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Forward pointer to next chunk in list |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Back pointer to previous chunk in list |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Unused space (may be 0 bytes long) .
. .
. |
nextchunk-> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
`foot:' | Size of chunk, in bytes |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Size of next chunk, in bytes |A|0|0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The above ASCII diagrams are from glibc/malloc/malloc.c.
By maintaining both user data and malloc metadata on the same data structure, malloc() achieves flexible heap memory management. However, if a user program doesn’t use memory properly and writes out of bounds, it can easily corrupt malloc’s data structures, leading to vulnerabilities or crashes.
The following program tries to modify data in malloc_chunk:
#include <stdlib.h>
struct malloc_chunk {
size_t mchunk_prev_size; /* Size of previous chunk (if free). */
size_t mchunk_size; /* Size in bytes, including overhead. */
struct malloc_chunk *fd; /* double links -- used only if free. */
struct malloc_chunk *bk;
/* Only used for large blocks: pointer to next larger size. */
struct malloc_chunk *fd_nextsize; /* double links -- used only if free. */
struct malloc_chunk *bk_nextsize;
};
struct malloc_chunk *container_of(void *p) {
size_t off = (size_t)(&((struct malloc_chunk *)NULL)->fd);
return (struct malloc_chunk *)(p - off);
}
int main() {
void *ptr = malloc(32);
struct malloc_chunk *chunk = container_of(ptr);
chunk->mchunk_size = 1;
free(ptr);
return 0;
}
struct malloc_chunkis from glibc/malloc/malloc.c.
Output: free() calls abort() to terminate the program.
free(): invalid pointer
fish: Job 1, './a.out' terminated by signal SIGABRT (Abort)
Modifying the free list data also causes a crash. The following program modifies the pointer’s data even after calling free(). This overwrites the forward and back pointers in the free list.
#include <stdlib.h>
#include <string.h>
int main() {
void *a = malloc(32);
void *b = malloc(32);
free(a);
free(b);
memset(b, 1, 32);
b = malloc(32);
a = malloc(32);
return 0;
}
Output: when malloc() is called again, the pointers in the malloc_chunk for b point to invalid addresses, causing a crash.
malloc(): unaligned tcache chunk detected
fish: Job 1, './a.out' terminated by signal SIGABRT (Abort)
A good habit is to set a pointer to NULL after calling
free(). Then the program would crash atmemset()instead of ata = malloc(32).
Stack Allocation
Usually, the stack size is determined at compile time. But there are exceptions. The alloca() function can allocate dynamically-sized space on the stack.
LLVM’s
allocainstruction may have some connection to this.
The following program demonstrates alloca(). Behaviorally, alloca() is equivalent to a variable-length array T arr[N].
#include <alloca.h>
#include <stdio.h>
int main() {
int n;
scanf("%d", &n);
int *arr = alloca(sizeof(int) * n);
for (int i = 0; i < n; i++) {
scanf("%d", &arr[i]);
}
for (int i = n - 1; i >= 0; i--) {
printf("%d ", arr[i]);
}
printf("\n");
return 0;
}
alloca() is implemented via a compiler builtin — it’s backed by __builtin_alloca. Let’s look at how it works with an example.
// alloca_example.c
#include <alloca.h>
void func(int m, int n) {
for (int i = 0; i < m; i++) {
int *ptr = alloca(sizeof(int) * n);
}
}
Compile with clang, targeting riscv64 (it’s the only one I can read…).
$ clang -S alloca_example.c --target=riscv64-linux-gnu
The resulting assembly, with comments highlighting the important parts. Within the stack frame, when extra stack memory is needed, the sp register moves toward lower addresses, and the moved address becomes the return value of alloca(). The fp register remains unchanged throughout, and local variables are addressed relative to fp. When the function returns, sp is reset to its original position based on the fp register.
func:
addi sp, sp, -48
sd ra, 40(sp)
sd s0, 32(sp)
addi s0, sp, 48 # s0 = fp; since fp doesn't change within the frame, use fp for addressing
sw a0, -20(s0)
sw a1, -24(s0)
li a0, 0
sw a0, -28(s0)
j .LBB0_1
.LBB0_1:
lw a0, -28(s0)
lw a1, -20(s0)
bge a0, a1, .LBB0_4
j .LBB0_2
.LBB0_2:
lw a0, -24(s0)
slli a0, a0, 2
addi a0, a0, 15
andi a1, a0, -16
mv a0, sp
sub a0, a0, a1 # sp moves down, stack frame grows
mv sp, a0
sd a0, -40(s0)
j .LBB0_3
.LBB0_3:
lw a0, -28(s0)
addiw a0, a0, 1
sw a0, -28(s0)
j .LBB0_1
.LBB0_4:
addi sp, s0, -48 # restore sp based on fp register
ld ra, 40(sp)
ld s0, 32(sp)
addi sp, sp, 48
ret
Note that in the source code, alloca() is inside a loop. In the assembly, the sp register movement is also inside the loop. This means memory allocated by alloca() in one iteration is not released when that iteration ends. All alloca() allocations are only freed when the function returns.
From the assembly, we can see the fp register plays a critical role here — it records the bottom of the current stack frame. Generally, when the compiler enables optimizations (-Oxx), it includes “frame pointer elimination.” But when we enable this optimization here, functions using alloca() still preserve the fp register.
$ clang -S alloca_example.c --target=riscv64-linux-gnu -fomit-frame-pointer
Although “frame pointer elimination” is usually enabled, it’s better to keep it off.
Despite all this, alloca() is rarely useful. Each thread’s stack space is limited (usually 8MB), making it impractical to allocate large amounts on the stack. Even if you increase the stack size, alloca() still isn’t very useful because its lifetime is tied to the stack frame — when the function returns, all alloca() memory is automatically freed. If you want to extend the lifetime of this data, you’d have to copy it; you can’t just pass a pointer.
The Memory Mapping Region
We always say memory is divided into heap and stack. So where does mmap()-allocated memory belong? It’s definitely not on the stack, but it’s also not on the heap. mmap()-allocated memory resides in the memory mapping region between the heap and the stack.
The memory mapping region sits between the heap and the stack, above the kernel constant TASK_UNMAPPED_BASE. Unlike malloc(), mmap() memory allocation is handled by the kernel. mmap() also allocates memory in pages.
The following program calls mmap() to allocate memory:
#include <sys/mman.h>
#include <stdio.h>
#include <unistd.h>
int main() {
int size = 10;
void *ptr = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
printf("ptr: %p\n", ptr);
pause();
munmap(ptr, size);
return 0;
}
Run the program and inspect its memory layout via proc:
$ clang run_mmap.c -o run_mmap
$ ./run_mmap &
ptr: 0x7f81ffca2000
$ cat /proc/$!/maps
The process’s memory layout is shown below. ===== separates the different sections of memory space. * marks the memory allocated by our mmap() call.
=====
55a9bde3a000-55a9bde3b000 r--p 00000000 103:05 8392660 /path/to/run_mmap
55a9bde3b000-55a9bde3c000 r-xp 00001000 103:05 8392660 /path/to/run_mmap
ELF 55a9bde3c000-55a9bde3d000 r--p 00002000 103:05 8392660 /path/to/run_mmap
55a9bde3d000-55a9bde3e000 r--p 00002000 103:05 8392660 /path/to/run_mmap
55a9bde3e000-55a9bde3f000 rw-p 00003000 103:05 8392660 /path/to/run_mmap
=====
Heap 55a9c4507000-55a9c4528000 rw-p 00000000 00:00 0 [heap]
=====
7f81ffa83000-7f81ffa86000 rw-p 00000000 00:00 0
7f81ffa86000-7f81ffaa8000 r--p 00000000 103:06 9063362 /usr/lib/x86_64-linux-gnu/libc.so.6
7f81ffaa8000-7f81ffc20000 r-xp 00022000 103:06 9063362 /usr/lib/x86_64-linux-gnu/libc.so.6
7f81ffc20000-7f81ffc78000 r--p 0019a000 103:06 9063362 /usr/lib/x86_64-linux-gnu/libc.so.6
7f81ffc78000-7f81ffc7c000 r--p 001f1000 103:06 9063362 /usr/lib/x86_64-linux-gnu/libc.so.6
7f81ffc7c000-7f81ffc7e000 rw-p 001f5000 103:06 9063362 /usr/lib/x86_64-linux-gnu/libc.so.6
MMap 7f81ffc7e000-7f81ffc8b000 rw-p 00000000 00:00 0
* 7f81ffca2000-7f81ffca5000 rw-p 00000000 00:00 0
7f81ffca5000-7f81ffca6000 r--p 00000000 103:06 9063067 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f81ffca6000-7f81ffcce000 r-xp 00001000 103:06 9063067 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f81ffcce000-7f81ffcd8000 r--p 00029000 103:06 9063067 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f81ffcd8000-7f81ffcda000 r--p 00033000 103:06 9063067 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7f81ffcda000-7f81ffcdc000 rw-p 00035000 103:06 9063067 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
=====
Stack 7ffe63299000-7ffe632ba000 rw-p 00000000 00:00 0 [stack]
=====
7ffe6332c000-7ffe63330000 r--p 00000000 00:00 0 [vvar]
7ffe63330000-7ffe63332000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
=====
From the table above, virtual memory space is laid out from low to high addresses as: program (ELF segments), heap, memory mapping region, and stack. Memory allocated by mmap() resides in the memory mapping region — this includes both anonymous mappings and file mappings. We can also see two shared libraries (libc.so.6 and ld-linux-x86-64.so.2) in the memory mapping region, because dynamic library loading (dlopen()) is actually implemented on top of mmap(). dlopen() reads the shared library file and maps its segments into memory at the appropriate positions, with the appropriate read, write, and execute permissions.
vvar,vdso, andvsyscallare memory segments added to accelerate certain system calls.
Thread Stacks for Non-Main Threads
When a process has only one thread, it only needs a single stack space, which is pre-allocated to a given size before the program runs. But with multiple threads, you need multiple thread stacks. These thread stacks are located differently from the main thread’s stack. Non-main thread stacks actually reside in the memory mapping region.
Here’s a simple pthread multithreaded program:
#include <pthread.h>
#include <unistd.h>
void *thread_function(void *arg) {
return NULL;
}
int main() {
pthread_t thread;
pthread_create(&thread, NULL, thread_function, NULL);
pthread_join(thread, NULL);
return 0;
}
Let’s trace the system calls during execution with strace:
$ clang test_thread.c -o test_thread
$ strace ./test_thread
Results:
execve("./test_thread", ["./test_thread"], 0x7ffc659bbb00 /* 73 vars */) = 0
brk(NULL) = 0x55d5e4bb3000
...
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fb614801000
mprotect(0x7fb614802000, 8388608, PROT_READ|PROT_WRITE) = 0
...
Focus on the mmap system call. Its flags include MAP_STACK, indicating that the allocated address is intended as a program stack. Right after mmap comes an mprotect call, which marks part of the mmap-allocated memory as readable and writable. This is the actual stack space.
We can also find this memory mapping in /proc/<PID>/maps, marked by * below. This confirms that for non-main threads, the thread stack is allocated by mmap and resides in the memory mapping region.
55d5b2fd6000-55d5b2fd7000 r--p 00000000 103:05 8392953 /path/to/test_thread
55d5b2fd7000-55d5b2fd8000 r-xp 00001000 103:05 8392953 /path/to/test_thread
55d5b2fd8000-55d5b2fd9000 r--p 00002000 103:05 8392953 /path/to/test_thread
55d5b2fd9000-55d5b2fda000 r--p 00002000 103:05 8392953 /path/to/test_thread
55d5b2fda000-55d5b2fdb000 rw-p 00003000 103:05 8392953 /path/to/test_thread
55d5e4bb3000-55d5e4bd4000 rw-p 00000000 00:00 0 [heap]
* 7fb614801000-7fb614802000 ---p 00000000 00:00 0
* 7fb614802000-7fb615005000 rw-p 00000000 00:00 0
7fb615005000-7fb615027000 r--p 00000000 103:06 9063362 /usr/lib/x86_64-linux-gnu/libc.so.6
7fb615027000-7fb61519f000 r-xp 00022000 103:06 9063362 /usr/lib/x86_64-linux-gnu/libc.so.6
7fb61519f000-7fb6151f7000 r--p 0019a000 103:06 9063362 /usr/lib/x86_64-linux-gnu/libc.so.6