A Comprehensive Guide
Identifying Encryption Algorithms in Assembly Code
A detailed explanation of how to recognize encryption algorithms in assembly code.
For those of you interested in reverse engineering (either as a potential career or just as a hobby), you will inevitably encounter encryption algorithms on your journey. Maybe you’re reversing some undocumented software that communicates over the internet, or maybe you’re simply solving a crackme that involves figuring out how a key is generated. Whatever your reasons might be, you’ll definitely benefit from having a thorough understanding of how common encryption algorithms work, how they’re implemented, and also how to recognize these algorithms in assembly code.
In this article series, I will cover a few common encryption algorithms, and I hope you will find this information valuable.
The first encryption algorithm: XOR Encryption
This is arguably one of the simplest encryption algorithms out there, at least one of the simplest to implement. This also makes it very easy to recognize in assembly code. Let’s first have a look at what the XOR encryption algorithm is.
A XOR cipher is an encryption algorithm that XORs some input using a key. This is extremely simple to implement, but it can also be quite trivial to break. Since a lot of other more advanced ciphers use XOR, I figured it would be a good idea to start by covering this algorithm first.
Here’s a small program that implements XOR encryption.
char* encryptString(std::string message)
{
size_t messageLength = message.length();
char key = ‘L’;
char* encrypted = new char[messageLength + 1](); for (int i = 0; i < messageLength; i++) {
encrypted[i] = message[i] ^ key;
} return encrypted;}
We’re just XOR’ing the message using a key consisting of just a single character: “L”. Usually, the key will be more than one character but I decided to keep things simple for this article.
By using a debugger, we can figure out if a program uses XOR encryption by looking at how the program handles the input from the user. Let me take you through my process of figuring this out.
Analyzing the code with a debugger
Use whatever debugger you’d like. I’ll be using Ollydbg so if I include any shortcuts, the shortcuts will apply to Ollydbg specifically and it might not work the same on other debuggers. Let’s get started by loading the program into our debugger.
The first thing we need to do is to locate the main function. One trick I like to use is to set a breakpoint for the __p_argc and __p_argv functions in the ucrtbased.dll module. This works because the program is in debug mode. It helps to have some prior knowledge of what a main function should look like, but I won’t go too much into detail on that in this article.
To set the breakpoints, press ALT + E to get a list of all the loaded modules.
If you highlight the ucrtbased.dll module, you can proceed by pressing CTRL + N to find a list of functions.
You can now locate the argv and argc function names, and set a breakpoint for each. Setting a breakpoint for argc only is also sufficient, but let’s just set a breakpoint for both.
The next thing we need to do is run the program until it reaches one of our breakpoints, and then we go through the call stack until we find the code that will call our main function.
From past experience, I know that this function is the one that will call our code’s main function. The function call that’s highlighted in the picture is the one calling our main function, while the other functions above are the argv and argc function calls.
If we enter into our main function, we immediately see a string with the value: “message here”.
I also took the liberty of adding a comment to a certain function call when I was going through this program earlier. Comments are really useful, so use them for what they’re worth. I’m of course referring to the “Encrypt func” comment here.
The encryption function
The first thing you probably noticed is that the string “message here” is being pushed onto the stack, ecx is set to contain a pointer to some stack variable, and then a function is called. You might suspect that this function call is the one that calls our encryption function, but it’s not. This is actually a function call that follows the __thiscall convention where ecx contains the pointer to the implicit this variable. If you set a breakpoint at the function call, and you take a look at the contents of the this pointer (right-click ecx register then follow in dump), you’ll see that it’s just a bunch of CC’s.
This means that the stack variable that the pointer is pointing to is unitialized. The CC’s represent unitialized memory when you compile a program in debug mode. What all of this tells us is that the function call is the constructor (more or less), and not something we’re interested in.
Further down we see more function calls, and eventually we reach the encryption function call. The way I figured this out was just to set a breakpoint for each function call, and then step over each call and see if something interesting turned up in the CPU registers. I noticed that a string literal was stored in the EAX register, and it had the same length as the “message here” string. So, that must have been the function call. Let’s enter into the function and see if we can recognize the xor encryption.
008557A0 55 PUSH EBP
008557A1 8BEC MOV EBP,ESP
008557A3 6A FF PUSH -1
008557A5 68 88B78500 PUSH Encrypti.0085B788
008557AA 64:A1 00000000 MOV EAX,DWORD PTR FS:[0]
008557B0 50 PUSH EAX
008557B1 81EC 18010000 SUB ESP,118
008557B7 53 PUSH EBX
008557B8 56 PUSH ESI
008557B9 57 PUSH EDI
008557BA 8DBD DCFEFFFF LEA EDI,DWORD PTR SS:[EBP-124]
008557C0 B9 46000000 MOV ECX,46
008557C5 B8 CCCCCCCC MOV EAX,CCCCCCCC
008557CA F3:AB REP STOS DWORD PTR ES:[EDI]
008557CC A1 08108600 MOV EAX,DWORD PTR DS:[861008]
008557D1 33C5 XOR EAX,EBP
008557D3 50 PUSH EAX
008557D4 8D45 F4 LEA EAX,DWORD PTR SS:[EBP-C]
008557D7 64:A3 00000000 MOV DWORD PTR FS:[0],EAX
008557DD C745 FC 00000000 MOV DWORD PTR SS:[EBP-4],0
008557E4 B9 27408600 MOV ECX,Encrypti.00864027
008557E9 E8 B9BBFFFF CALL Encrypti.008513A7
008557EE 8D4D 08 LEA ECX,DWORD PTR SS:[EBP+8]
008557F1 E8 27B9FFFF CALL Encrypti.0085111D
008557F6 8945 EC MOV DWORD PTR SS:[EBP-14],EAX
008557F9 C645 E3 4C MOV BYTE PTR SS:[EBP-1D],4C
008557FD 8B45 EC MOV EAX,DWORD PTR SS:[EBP-14]
00855800 83C0 01 ADD EAX,1
00855803 8985 FCFEFFFF MOV DWORD PTR SS:[EBP-104],EAX
00855809 8B8D FCFEFFFF MOV ECX,DWORD PTR SS:[EBP-104]
0085580F 51 PUSH ECX
00855810 E8 1CB9FFFF CALL Encrypti.00851131
00855815 83C4 04 ADD ESP,4
00855818 8985 F0FEFFFF MOV DWORD PTR SS:[EBP-110],EAX
0085581E 83BD F0FEFFFF 00 CMP DWORD PTR SS:[EBP-110],0
00855825 74 26 JE SHORT Encrypti.0085584D
00855827 8B95 FCFEFFFF MOV EDX,DWORD PTR SS:[EBP-104]
0085582D 52 PUSH EDX
0085582E 6A 00 PUSH 0
00855830 8B85 F0FEFFFF MOV EAX,DWORD PTR SS:[EBP-110]
00855836 50 PUSH EAX
00855837 E8 5EB9FFFF CALL Encrypti.0085119A
0085583C 83C4 0C ADD ESP,0C
0085583F 8B8D F0FEFFFF MOV ECX,DWORD PTR SS:[EBP-110]
00855845 898D DCFEFFFF MOV DWORD PTR SS:[EBP-124],ECX
0085584B EB 0A JMP SHORT Encrypti.00855857
0085584D C785 DCFEFFFF 00>MOV DWORD PTR SS:[EBP-124],0
00855857 8B95 DCFEFFFF MOV EDX,DWORD PTR SS:[EBP-124]
0085585D 8955 D4 MOV DWORD PTR SS:[EBP-2C],EDX
00855860 C745 C8 00000000 MOV DWORD PTR SS:[EBP-38],0
00855867 EB 09 JMP SHORT Encrypti.00855872
00855869 8B45 C8 MOV EAX,DWORD PTR SS:[EBP-38]
0085586C 83C0 01 ADD EAX,1
0085586F 8945 C8 MOV DWORD PTR SS:[EBP-38],EAX
00855872 8B45 C8 MOV EAX,DWORD PTR SS:[EBP-38]
00855875 3B45 EC CMP EAX,DWORD PTR SS:[EBP-14]
00855878 73 1F JNB SHORT Encrypti.00855899
0085587A 8B45 C8 MOV EAX,DWORD PTR SS:[EBP-38]
0085587D 50 PUSH EAX
0085587E 8D4D 08 LEA ECX,DWORD PTR SS:[EBP+8]
00855881 E8 7BBBFFFF CALL Encrypti.00851401
00855886 0FBE08 MOVSX ECX,BYTE PTR DS:[EAX]
00855889 0FBE55 E3 MOVSX EDX,BYTE PTR SS:[EBP-1D]
0085588D 33CA XOR ECX,EDX
0085588F 8B45 D4 MOV EAX,DWORD PTR SS:[EBP-2C]
00855892 0345 C8 ADD EAX,DWORD PTR SS:[EBP-38]
00855895 8808 MOV BYTE PTR DS:[EAX],CL
00855897 ^EB D0 JMP SHORT Encrypti.00855869
00855899 8B45 D4 MOV EAX,DWORD PTR SS:[EBP-2C]
0085589C 8985 E4FEFFFF MOV DWORD PTR SS:[EBP-11C],EAX
008558A2 C745 FC FFFFFFFF MOV DWORD PTR SS:[EBP-4],-1
008558A9 8D4D 08 LEA ECX,DWORD PTR SS:[EBP+8]
008558AC E8 B0BAFFFF CALL Encrypti.00851361
008558B1 8B85 E4FEFFFF MOV EAX,DWORD PTR SS:[EBP-11C]
008558B7 8B4D F4 MOV ECX,DWORD PTR SS:[EBP-C]
008558BA 64:890D 00000000 MOV DWORD PTR FS:[0],ECX
008558C1 59 POP ECX
008558C2 5F POP EDI
008558C3 5E POP ESI
008558C4 5B POP EBX
008558C5 81C4 24010000 ADD ESP,124
008558CB 3BEC CMP EBP,ESP
008558CD E8 F3BAFFFF CALL Encrypti.008513C5
008558D2 8BE5 MOV ESP,EBP
008558D4 5D POP EBP
008558D5 C3 RETNThe first thing I did here was to immediately scan for any XOR instructions. Lo and behold, I found two XOR instructions! Here’s the first one, and the code associated with it.
008557BA 8DBD DCFEFFFF LEA EDI,DWORD PTR SS:[EBP-124]
008557C0 B9 46000000 MOV ECX,46
008557C5 B8 CCCCCCCC MOV EAX,CCCCCCCC
008557CA F3:AB REP STOS DWORD PTR ES:[EDI]
008557CC A1 08108600 MOV EAX,DWORD PTR DS:[861008]
008557D1 33C5 XOR EAX,EBPWe can immediately discount this one because it’s a part of the debug mode code. Do you recognize the C’s? ;)
The next XOR instruction and the code associated with it looks like this:
00855869 8B45 C8 MOV EAX,DWORD PTR SS:[EBP-38]
0085586C 83C0 01 ADD EAX,1
0085586F 8945 C8 MOV DWORD PTR SS:[EBP-38],EAX
00855872 8B45 C8 MOV EAX,DWORD PTR SS:[EBP-38]
00855875 3B45 EC CMP EAX,DWORD PTR SS:[EBP-14]
00855878 73 1F JNB SHORT Encrypti.00855899
0085587A 8B45 C8 MOV EAX,DWORD PTR SS:[EBP-38]
0085587D 50 PUSH EAX
0085587E 8D4D 08 LEA ECX,DWORD PTR SS:[EBP+8]
00855881 E8 7BBBFFFF CALL Encrypti.00851401
00855886 0FBE08 MOVSX ECX,BYTE PTR DS:[EAX]
00855889 0FBE55 E3 MOVSX EDX,BYTE PTR SS:[EBP-1D]
0085588D 33CA XOR ECX,EDX
0085588F 8B45 D4 MOV EAX,DWORD PTR SS:[EBP-2C]
00855892 0345 C8 ADD EAX,DWORD PTR SS:[EBP-38]
00855895 8808 MOV BYTE PTR DS:[EAX],CL
00855897 ^EB D0 JMP SHORT Encrypti.00855869Right before the XOR instruction, a byte is stored in ECX and EDX. At first glance, this looks very promising. If we set a breakpoint at the XOR instruction, we’ll see that EAX contains the string literal: “message here”. Great! This must definitely be the part of the code that encrypts the message. The ECX register contains the first letter of the string literal, and EDX contains the letter “L”.
If we look further down, we can see that we’ll eventually reach a JMP instruction that jumps back to address 00855869. This just loads a value into EAX and adds one. Eventually, it compares EAX to some value.
It compares EAX to the value 0C which is 12 in decimal. This number happens to also be the length of our string: “message here”. Coincidence? I think not. This all looks to be just a simple for loop, which XORs every byte of the input string against the character “L”.
This should just about cover how to recognize XOR encryption. Obviously my program is very simple and it’s not always that easy to locate the correct code, but this should definitely help you recognize XOR encryption in the wild.
Because I don’t want this article to be the length of a book, I’ll be covering the next encryption algorithm in a part two of this series. I’ll leave a link to the article at the bottom of this article when I’ve finished writing it.
Thanks for reading!
