x86 Calling Conventions
“The great thing about calling conventions on the x86 platform is that there are so many to choose from!” Raymond Chen, Microsoft
Calling conventions act as a contract between subroutines at the assembly level. They describe how and where arguments are passed and who is responsible for cleaning up the stack. Closely related considerations like which registers can be “clobbered,” which must be preserved and where return values are located are associated with a specific application binary interface (ABI), a more encompassing subject not given treatment here.
I imagine it’s possible to have a long and fulfilling career as a C/C++ developer without ever truly understanding the various calling conventions and their nuances. After all, they are a concern of the compiler and generally not the developer (unless you’re doing some low level debugging).
As a reverse engineer working on binary targets, it’s helpful to understand calling conventions and learn how to spot them in the code you’re reversing. Here we look at a few x86 calling conventions, the properties that make them unique and view some samples of what they look like when disassembled.
Caller/callee
You’re going to see the terms caller and callee used a lot here which can be a little confusing. The caller is simply a function that calls another function. The callee is a function that is called from another function. Consider the following C program that contains two functions, main and printFavNums.
In the code above the caller is main and the callee is printFavNums. Why? Because main calls another function (printFavNums) and printFavNums is called from another function (main).
If you want to copy and paste code from this article, check out this Github Gist. Those sick source code images are generated by Carbon. ❤
Who decides which calling convention is used?
The compiler determines which calling convention is used for all functions. Many compilers, however, allow the programmer to set specific calling conventions on a per-function basis as we’ll do with GCC in this article.
It’s important to remember that these are just conventions and a compiler has the choice to obey or disobey any convention. The calling conventions you’ll encounter are typically a result of the default calling convention set by the compiler, or a calling convention officially adopted by a target platform, like the Win32 API’s use of stdcall.
x86 (32-bit)
The x86 architecture has numerous adopted calling conventions prior to 64-bit processors. Here we’ll take a look at four of the most common 32-bit calling conventions, their defining characteristics and what they look like in Intel-flavor disassembly.
cdecl
The cdecl (pronounced see-dec-el, short for “C declaration”) calling convention is extremely common as it’s the default x86 (32-bit) calling convention on today’s most popular compilers including GCC, MSVC, and LLVM. It has the following properties:
- The caller places all arguments to the callee on the stack
- Arguments are pushed to the stack from right to left
- Stack cleanup is performed by the caller
Let’s consider the simple program we referenced earlier. We don’t need to explicitly set cdecl because it’s the default calling convention of GCC, the compiler we’re using here.
Here we use
-m32to tell GCC we want a 32-bit binary and-mpreferred-stack-boundary=2to tell GCC we want 2² (4-byte) stack alignment. We set this stack alignment because it’s much easier to confirm who cleans up the stack. Without setting this option the ABI dictates a 2⁴ (16-byte) stack alignment, which just adds unnecessary confusion.
Let’s take a look at this simple program (just main and printFavNums) disassembled with Binary Ninja and look for the cdecl characteristics we expect to see.
Here we can see main (caller) is pushing arguments for printFavNums (callee) to the stack from right to left. It first pushes 8 then 2 and calls printFavNums. This confirms ‘the caller places all arguments to the callee on the stack’ and ‘arguments are pushed to the stack from right to left’.
But what about the last characteristic? Who is cleaning up the stack here? If you look just after the call to printFavNums, main cleans up 8-bytes of the stack frame(add esp, 0x8), which accounts for both 4-byte arguments passed to the callee. This confirms the caller, main is cleaning up the stack for the callee. But just to make sure there’s no funny business here, let’s look at printFavNums.
We can see the three 4-byte arguments to printf are pushed to the stack from right to left and after the call is made, the caller, printFavNums cleans up the stack for the callee (printf) by adding 0xC (12) to the stack pointer (esp), just as we’d expect from cdecl. printFavNums never accounts for its own arguments when cleaning the stack frame. That’s for its caller (main) to do in this convention.
fastcall
The fastcall calling convention gets its name because it’s faster than other calling conventions that pass their arguments on the stack. On 32-bit systems, there are a limited number of registers to utilize for argument passing (compared to x86-64 and some other architectures with more general-purpose registers). Because of this, only the first 2 arguments are passed by register in fastcall, while any remaining arguments are passed on the stack. fastcall has the following properties:
- The caller places the first 2 arguments to the callee in registers, the rest on the stack
- Arguments are loaded to registers / pushed to the stack from right to left
- Stack cleanup is performed by the callee
Let’s consider a program very similar to the simple C program we referenced earlier. This time we’ve added one more argument and we explicitly set fastcall as a function attribute (this is GCC-specific syntax).
Let’s take a look at what the compiler did in this version of our simple program, again disassembled with Binary Ninja.
Here we can see main (caller) is pushing the last argument to the stack, then loading the second and first arguments into the edx and ecx registers respectively. This confirms that in fastcall the caller places the first 2 arguments to the callee in registers, the rest on the stack and arguments are processed right to left.
Notice how main does not clean up the stack after calling printFavNums. Let’s take a look at printFavNums to make sure it cleans up its own arguments.
If we look below the call to printf (which is still cdecl) we see printFavNums cleaning up 0x10 (16) bytes to account for the four 4-byte arguments passed to printf. But where is printFavNums cleaning up its own three 4-byte arguments? Well, remember that in fastcall two of those three were passed though registers which don’t need to be cleaned up. This leaves only one 4-byte argument to clean up, but where is that done?
The printFavNums function does this by using an optional operand to the retn instruction of 0x4. This causes an extra 4-bytes to be ‘cleaned up’ during the return operation. It certainly looks a lot different than we’ve seen so far, with the stack pointer (esp) being directly manipulated, but the effect is the same. Hence, the callee cleans up after itself.
stdcall
stdcall (short for “standard call”) is a common x86 calling convention similar to cdecl however the callee is responsible for cleaning its own arguments off the stack. If you’ve ever done any Windows development you probably know this calling convention as WINAPI which is just an alias for stdcall set in windef.h (#define WINAPI __stdcall). This calling convention has the following properties:
- The caller places all arguments to the callee on the stack
- Arguments are pushed to the stack from right to left
- Stack cleanup is performed by the callee
Let’s consider the same simple C program we’ve been using, this time with stdcall set for printFavNums.
Now let’s check out our two disassembled functions.
As we expect, main pushes the arguments to printFavNums to the stack from right to left. Notice how main does not clean up the stack after calling printFavNums.
Taking a look in printFavNums we can see retn 0x8 which accounts for the two 4-byte arguments passed into this function. As we expected, this stdcall function cleans up after itself.
thiscall
The thiscall calling convention is a pretty interesting beast. It’s intended to be used on C++ class member functions that need to reference their “this pointer” so the function can access class instance variables. Here are the properties for this calling convention:
- Arguments are pushed to the stack from right to left
- The “this pointer” is passed to the callee via the ECX register
- Stack cleanup is performed by the callee
Let’s consider a C++ version of the C program we’ve been using where printFavNums is a member of the FavoriteNumbers class.
Based on everything I’ve ever read and known about thiscall, GCC should make printFavNums a thiscall function because we’re calling a C++ class member function which requires access to its own instance variables. So let’s see…
Wait a minute. The ecx register is not being used for a this pointer, and the stack is clearly being cleaned by the caller (main). printFavNums fits the description of cdecl to a tee. So what happened here?
It turns out g++ (the C++ compiler command for GCC) made this function a cdecl function but inserted the this pointer as the first argument to pritnFavNums. So the function was given three arguments when compiled, even though the C++ prototype only has two!
There’s a lot¹ of information online² and in books³ that tell you there are two versions of thiscall. One for GCC and one for MSVC. This can be thought of in two ways; 1. There are two unique versions of thiscall, 2. GCC uses cdecl instead of thiscall by default, injecting the implied “this pointer” as the first argument.
Both of these descriptions are acceptable, and I personally like to use the second as my understanding of what’s happening here. GCC’s funky business with thiscall can be traced back to version 4.7.0⁴ where the developers switched from one form to the other in an attempt to become more compatible with the MSVC implementation of thiscall.
This caused problems⁵ with developers who weren’t expecting this change. Eventually, GCC switched back to doing it their own way — but adding a thiscall function attribute⁶ allowing developers to force thiscall instead of GCC’s odd “cdecl with injected this pointer” convention.
Let’s use that same code, but this time we’ll explicitly set printFavNums to thiscall using the GCC function attribute.
Now we’ll compile and disassemble it to see what happened this time.
That’s more like it! The printFavNums function is passed two arguments on the stack and ecx is loaded with the this pointer just before the call. We can also see that main is not cleaning up the stack after the call to printFavNums. printFavNums is now a thiscall function!
Special considerations
The subject of calling conventions can become complex under certain conditions (thiscall was bad enough). In many cases the information presented up to this point holds true, but in some cases, there are catches you must consider. Here we discuss some of those special cases.
Variable argument functions
In this article, you’ve learned about some calling conventions that are “callee clean up”, meaning the callee cleans up the stack itself, but there’s a catch. Callees cannot clean up their own stack if they are passed a variable number of arguments!
Variable argument functions (also called “vararg” or “variadic” functions) are functions that take a variable number of arguments. An example of a vararg function is given below.
The above code will compile with no issues because the default calling convention is cdecl, a caller cleanup convention. However, if we attempt to set this function as any of the callee cleanup functions we’ve learned about, say stdcall…
… the compiler will throw a warning and fall back to cdecl because it cannot make a variadic function a callee cleanup convention.
So why can’t variadic callees clean up their own stack? Can’t they just dynamically adjust the stack pointer before returning, or use retn X to clean up?
At compile time, a variadic callee has no guess as to how many arguments it could be passed. It could be zero, one hundred, or anything else. So the compiler can’t determine an effective strategy for stack cleanup.
Theoretically, could the variadic function be modified by the compiler to dynamically resolve its own argument count and size, then clean up its own stack? In most cases, I believe so. But thinking about possible edge cases and how to formally verify this makes my head hurt. With both caller and callee clean up conventions available, it just doesn’t make sense to go through all this trouble when an easier solution exists; just don’t use caller clean up conventions with variadic functions.
Calling thiscall anytime!
So we learned about the intended use of thiscall which is to be used with C++ class member functions that need access to their own class instance. But thiscall can be used wherever you want (when using GCC). In fact, this code is perfectly valid.
You might think the compiler (GCC 5.4.0 in this case) would complain. This isn’t even C++ and there’s no class here. Why would we need thiscall? But it works, check out the disassembly.
Since there is no this pointer the compiler passes arg1 through ecx and arg2 on the stack. The callee is cleaning up its own arguments from the stack as we expect in thiscall functions. This craziness is part of a GCC extension which allows this to occur⁷.
Conclusion
I hope you’ve enjoyed this article on common x86 calling conventions and you can picture the C to assembly conversion happening in each. There are some exotic calling conventions we didn’t cover here like pascal, optlink, safecall, and others. To be honest, I only know about these because of Wikipedia. I don’t think I’ve ever encountered them in my career.
I’ve made it years as a professional reverse engineer without ever whiteboarding this stuff. I knew some of the general differences between x86 calling conventions but I never spent the time to really sit down and study the subject.
In today’s modern x86–64 systems however, calling conventions are quite literally something I have to think about every single working day of my life. Understanding the subtle but very important differences between the Windows and System-V ABIs is critical in the x86–64 reverse engineering process, and I’ll explain why in a future article on x86–64 calling conventions.
Footnotes & References
[1] Wikipedia, x86 calling conventions, thiscall.
[2]: Ownedcore, World of Warcraft, World of Warcraft Bots and Programs, WoW Memory Editing. GCC “thiscall” calling convention [Linux/Win32 MinGW].
[3]: Mastering Malware Analysis, Alexey Kleymenov, Amr Thabet, Packt Publishing, page 25: “In GCC compiler, this call is almost identical to cdecl calling convention and it passes the object address as a first argument. But in the Microsoft C++ compiler, it’s similar to stdcall and it passes the object address in ecx.”
[4]: GCC 4.7 Release Series Changes, New Features, and Fixes. “ Windows x86 targets are using the __thiscall calling convention for C++ class-member functions.”
[5]: Narkive Mailinglist Archive, “how to disable __thiscall on MinGW-gcc-4.7.x?”.
[6]: 6.33.34 x86 Function Attributes, thiscall.
[7]: From the GCC Documentation 6.33.34 x86 Function Attributes: “On x86–32 targets, the thiscall attribute causes the compiler to pass the first argument (if of integral type) in the register ECX. [...] As a GCC extension, this calling convention can be used for C functions and for static member methods.”
About the author
John Toterhi is a security researcher at Battelle Memorial Institute where he specializes in reverse engineering, vulnerability research, and tool development. When he’s not being a computer nerd, he’s probably picking up a new hobby or telling bad dad jokes. You can connect with John on LinkedIn, GitHub, Twitter, and Medium.
