A reverse engineering challenge from a CTF competition
📌 Note
Due to a real challenge scenario, the flag will be masked.
Introduction
In this writeup, we will analyze a reverse engineering CTF, which requires reverse engineering a 64-bit ELF binary to extract the correct password and retrieve the flag.
The approach follows a two-phase methodology:
- Static Analysis – Extracting preliminary information without executing the binary.
- Dynamic Analysis – Debugging the binary to observe its runtime behavior.
Tools Used
strings– Extract embedded strings from the binary.objdump– Disassemble the binary and analyze its assembly.GDB– Debug the binary dynamically.Ghidra– Reverse-engineer and analyze the decompiled C code.
By the end, we will have a full understanding of the binary’s logic and successfully retrieve the flag.
Step 1: Initial Analysis of the Binary
A reverse engineering challenge from a CTF competitionrainpwn@0xdeadc0de:~$ ./challenge_binary ./challenge <password> rainpwn@0xdeadc0de:~$ ./challenge_binary p4ssw0rd rainpwn@0xdeadc0de:~$
1.1 Identifying the File Type
First, let’s identify the format of the executable:
A reverse engineering challenge from a CTF competitionrainpwn@0xdeadc0de:~$ file challenge_binary challenge_binary: ELF 64-bit LSB executable, x86-64, dynamically linked, not stripped rainpwn@0xdeadc0de:~$
Observations:
- The file is a 64-bit ELF binary compiled for the x86-64 architecture.
- It is not stripped, meaning function names and symbols are present, making the analysis easier.
1.2 Extracting Embedded Strings
Let’s check for hardcoded strings in the binary:
A reverse engineering challenge from a CTF competitionrainpwn@0xdeadc0de:~$ strings challenge_binary
Interesting output:
A reverse engineering challenge from a CTF competition<...> ./challenge <password> > FLAG{%s} strncmp puts printf main <...>
Key insights:
- The program expects a password as an argument (
./challenge <password>). - The format string
FLAG{%s}suggests that the flag is dynamically constructed. - The presence of
strncmpimplies a direct string comparison, meaning our input is checked against a stored value.
Step 2: Disassembling the Binary with objdump
To understand the binary’s logic, we disassemble it using objdump:
A reverse engineering challenge from a CTF competitionrainpwn@0xdeadc0de:~$ objdump -D challenge_binary | grep -A20 "<main>"
Relevant portion of the main function:
A reverse engineering challenge from a CTF competition1161: 48 8d 3d f9 00 00 00 lea 0xf9(%rip),%rdi # 1261 <main> 1168: ff 15 72 2e 00 00 call *0x2e72(%rip) # 3fe0 <__libc_start_main@GLIBC_2.2.5> 116e: f4 hlt 116f: 90 nop 0000000000001170 <deregister_tm_clones>: 1170: 48 8d 3d 99 2e 00 00 lea 0x2e99(%rip),%rdi # 4010 <__TMC_END__> 1177: 48 8d 05 92 2e 00 00 lea 0x2e92(%rip),%rax # 4010 <__TMC_END__> 117e: 48 39 f8 cmp %rdi,%rax 1181: 74 15 je 1198 <deregister_tm_clones+0x28> 1183: 48 8b 05 4e 2e 00 00 mov 0x2e4e(%rip),%rax # 3fd8 <_ITM_deregisterTMCloneTable> 118a: 48 85 c0 test %rax,%rax 118d: 74 09 je 1198 <deregister_tm_clones+0x28> 118f: ff e0 jmp *%rax 1191: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) 1198: c3 ret 1199: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) 00000000000011a0 <register_tm_clones>: 11a0: 48 8d 3d 69 2e 00 00 lea 0x2e69(%rip),%rdi # 4010 <__TMC_END__> 11a7: 48 8d 35 62 2e 00 00 lea 0x2e62(%rip),%rsi # 4010 <__TMC_END__> -- 0000000000001261 <main>: 1261: f3 0f 1e fa endbr64 1265: 55 push %rbp 1266: 48 89 e5 mov %rsp,%rbp 1269: 48 81 ec b0 00 00 00 sub $0xb0,%rsp 1270: 89 bd 5c ff ff ff mov %edi,-0xa4(%rbp) 1276: 48 89 b5 50 ff ff ff mov %rsi,-0xb0(%rbp) 127d: 64 48 8b 04 25 28 00 mov %fs:0x28,%rax 1284: 00 00 1286: 48 89 45 f8 mov %rax,-0x8(%rbp) 128a: 31 c0 xor %eax,%eax 128c: 48 8d 85 60 ff ff ff lea -0xa0(%rbp),%rax 1293: ba 98 00 00 00 mov $0x98,%edx 1298: be 00 00 00 00 mov $0x0,%esi 129d: 48 89 c7 mov %rax,%rdi 12a0: e8 7b fe ff ff call 1120 <memset@plt> 12a5: 48 8d 85 60 ff ff ff lea -0xa0(%rbp),%rax 12ac: 48 83 c0 08 add $0x8,%rax 12b0: 48 89 c7 mov %rax,%rdi 12b3: e8 78 fe ff ff call 1130 <sigemptyset@plt> 12b8: 48 8d 05 6a ff ff ff lea -0x96(%rip),%rax # 1229 <segill_sigaction>
Key observations:
- Control Flow Protection: The presence of
endbr64suggests CET (Control-flow Enforcement Technology). - Stack Setup: The program sets up a local buffer and manipulates memory.
- Signal Handling:
sigemptyset@pltis used, indicating possible anti-debugging mechanisms.
Step 3: Reverse Engineering with Ghidra
3.1 Examining the main Function
Here is the decompiled output of main:
A reverse engineering challenge from a CTF competitionvoid main(void) { sigaction local_a8; memset(&local_a8,0,0x98); sigemptyset(&local_a8.sa_mask); local_a8.__sigaction_handler.sa_handler = segill_sigaction; local_a8.sa_flags = 4; sigaction(4,&local_a8,(sigaction *)0x0); do { invalidInstructionException(); } while (true); }
🔴 Anti-Debugging Mechanism:
- The binary registers a signal handler (
sigaction) for SIGILL (Illegal Instruction Exception). - It intentionally triggers an invalid instruction, causing a crash when executed under certain conditions.
- This is a classic anti-debugging trick to detect analysis tools like GDB.
Step 4: Extracting the Flag
Instead of bypassing the anti-debugging mechanisms manually, we search for the hardcoded flag using objdump in .rodata.
Yes, i'm lazy. :)
What is the .rodata section?
The .rodata section (Read-Only Data) in an ELF binary contains constant data that should remain immutable during the program's execution, such as constant strings, lookup tables, and other static data. Typically, this section holds:
- Constant text strings used by functions like
printf,puts,strncmp, etc. - Constant numeric values, such as predefined tables or hardcoded values.
- Static initialization data, like default lookup tables or arrays.
Analyzing the binary challenge_binary, i found several lea (Load Effective Address) instructions pointing to addresses relative to the RIP register. These addresses often correspond to data in the .rodata section.
How .rodata and LEA are related
In the assembly, i found these instructions:
A reverse engineering challenge from a CTF competition1342: 48 8d 35 d2 0c 00 00 lea 0xcd2(%rip),%rsi # 201b <_IO_stdin_used+0x1b> 1372: 48 8d 35 a6 0c 00 00 lea 0xca6(%rip),%rsi # 201f <_IO_stdin_used+0x1f> 13a2: 48 8d 35 7a 0c 00 00 lea 0xc7a(%rip),%rsi # 2023 <_IO_stdin_used+0x23> 13ce: 48 8d 35 52 0c 00 00 lea 0xc52(%rip),%rsi # 2027 <_IO_stdin_used+0x27>
The LEA (Load Effective Address) instruction is used to calculate a memory address without performing an actual memory access. Here, lea X(%rip), %rsi is loading the address RIP + X into the RSI register.
For example:
A reverse engineering challenge from a CTF competitionlea 0xcd2(%rip),%rsi # 201b <_IO_stdin_used+0x1b>
Here, 0x201b is a memory address, and from gdb session, it contains:
A reverse engineering challenge from a CTF competition(gdb) x/s 0x201b 0x201b: "Str"
So, the value in RSI will be a pointer to the string "Str".
If we repeat this for the other addresses:
A reverse engineering challenge from a CTF competition(gdb) x/s 0x201f 0x201f: "_9x" (gdb) x/s 0x2023 0x2023: "Qb_" (gdb) x/s 0x2027 0x2027: "OQ1"
These data are part of the .rodata section and likely make up a string or flag that the program checks using strncmp.
Verifying .rodata with objdump
To directly view the contents of the .rodata section in the binary:
A reverse engineering challenge from a CTF competitionrainpwn@0xdeadc0de:~$ objdump -s -j .rodata challenge_binary
This will show the hexadecimal content of the .rodata section, from which we can reconstruct the entire flag FLAG{Str_9xQb_OQ1}.
Output:
A reverse engineering challenge from a CTF competitionchallenge_binary: formato del file elf64-x86-64 Contenuto della sezione .rodata: 2000 01000200 2e2f6368 616c6c65 6e676520 ...../challenge 2010 3c706173 73776f72 643e0049 747a005f <password>.Str._ 2020 306e004c 795f0055 4432003e 20485442 9x.Qb_.OQ1.> FLAG 2030 7b25737d 0a00 {%s}..
An alternative approach was to search for all instances of "lea" using objdump and then locate them using gdb.
A reverse engineering challenge from a CTF competitionrainpwn@0xdeadc0de:~$ objdump -D challenge_binary | grep -A20 "lea"
A reverse engineering challenge from a CTF competition133a: 48 8b 00 mov (%rax),%rax 133d: ba 03 00 00 00 mov $0x3,%edx 1342: 48 8d 35 d2 0c 00 00 lea 0xcd2(%rip),%rsi # 201b <_IO_stdin_used+0x1b> 1349: 48 89 c7 mov %rax,%rdi 134c: e8 6f fd ff ff call 10c0 <strncmp@plt> 1351: 85 c0 test %eax,%eax 1353: 0f 85 d0 00 00 00 jne 1429 <main+0x1c8> 1359: 0f 0b ud2 135b: 48 8b 85 50 ff ff ff mov -0xb0(%rbp),%rax 1362: 48 83 c0 08 add $0x8,%rax 1366: 48 8b 00 mov (%rax),%rax 1369: 48 83 c0 03 add $0x3,%rax 136d: ba 03 00 00 00 mov $0x3,%edx 1372: 48 8d 35 a6 0c 00 00 lea 0xca6(%rip),%rsi # 201f <_IO_stdin_used+0x1f> 1379: 48 89 c7 mov %rax,%rdi 137c: e8 3f fd ff ff call 10c0 <strncmp@plt> 1381: 85 c0 test %eax,%eax 1383: 0f 85 97 00 00 00 jne 1420 <main+0x1bf> 1389: 0f 0b ud2 138b: 48 8b 85 50 ff ff ff mov -0xb0(%rbp),%rax 1392: 48 83 c0 08 add $0x8,%rax 1396: 48 8b 00 mov (%rax),%rax 1399: 48 83 c0 06 add $0x6,%rax 139d: ba 03 00 00 00 mov $0x3,%edx 13a2: 48 8d 35 7a 0c 00 00 lea 0xc7a(%rip),%rsi # 2023 <_IO_stdin_used+0x23> 13a9: 48 89 c7 mov %rax,%rdi 13ac: e8 0f fd ff ff call 10c0 <strncmp@plt> 13b1: 85 c0 test %eax,%eax 13b3: 75 62 jne 1417 <main+0x1b6> 13b5: 0f 0b ud2 13b7: 48 8b 85 50 ff ff ff mov -0xb0(%rbp),%rax 13be: 48 83 c0 08 add $0x8,%rax 13c2: 48 8b 00 mov (%rax),%rax 13c5: 48 83 c0 09 add $0x9,%rax 13c9: ba 03 00 00 00 mov $0x3,%edx 13ce: 48 8d 35 52 0c 00 00 lea 0xc52(%rip),%rsi # 2027 <_IO_stdin_used+0x27> 13d5: 48 89 c7 mov %rax,%rdi 13d8: e8 e3 fc ff ff call 10c0 <strncmp@plt> 13dd: 85 c0 test %eax,%eax
Here, we identified these occurrences of "lea":
A reverse engineering challenge from a CTF competition1342: 48 8d 35 d2 0c 00 00 lea 0xcd2(%rip),%rsi # 201b <_IO_stdin_used+0x1b> 1372: 48 8d 35 a6 0c 00 00 lea 0xca6(%rip),%rsi # 201f <_IO_stdin_used+0x1f> 13a2: 48 8d 35 7a 0c 00 00 lea 0xc7a(%rip),%rsi # 2023 <_IO_stdin_used+0x23> 13ce: 48 8d 35 52 0c 00 00 lea 0xc52(%rip),%rsi # 2027 <_IO_stdin_used+0x27>
Using gdb, we retrieved the flag:
A reverse engineering challenge from a CTF competitionrainpwn@0xdeadc0de:~$ gdb challenge_binary
A reverse engineering challenge from a CTF competitionGNU gdb (Debian 16.2-1) 16.2 <...> For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from challenge_binary... (No debugging symbols found in challenge_binary) (gdb) x/s 0x201b 0x201b: "Str" (gdb) x/s 0x201f 0x201f: "_9x" (gdb) x/s 0x2023 0x2023: "Qb_" (gdb) x/s 0x2027 0x2027: "OQ1" (gdb)
A reverse engineering challenge from a CTF competitionrainpwn@0xdeadc0de:~$ ./challenge_binary Str_9xQb_OQ1 > FLAG{Str_9xQb_OQ1}
Conclusion
- The binary used anti-debugging techniques such as SIGILL trapping to hinder analysis.
- Instead of patching the binary, we extracted the flag from
.rodatausingobjdump. - The final flag: FLAG{Str_9xQb_OQ1}