Art of Shellcoding: The MultiEncoder Shellcode (RNX2)

Shellcode Research Nipun Jaswal todayFebruary 6, 2018 3

Background
share close

Legacy research restoration. Originally published on 6 February 2018 as part of the SecurityTube Linux Assembly Expert programme under student ID SLAE-1080. The original technical sequence, screenshots, source references and publication date have been preserved from the Blogger Takeout archive.

RNX2 custom shellcode encoder combining XOR, NOT and rotation operations
The original header image from the 2018 MultiEncoder research article.

Research objective

This fourth article in the Art of Shellcoding series explores the design of a custom encoder for a compact Linux x86 /bin/sh payload. The objective was to create a reversible multi-stage byte transformation, implement its decoder in assembly, and verify every decoding step in GDB rather than treating the result as a black box.

The source payload used in the exercise was a 25-byte stack-based execve implementation. The original source remains referenced in the SLAE assignment repository.

The RNX2 encoding chain

The encoder transforms every byte through four operations:

  1. XOR with 0xAA
  2. XOR with 0xCF
  3. Apply a bitwise NOT
  4. Apply the rotation/offset transformation using decimal 131

The decoder reverses those operations in the opposite order:

  1. Subtract 131
  2. Apply NOT
  3. XOR with 0xCF
  4. XOR with 0xAA

Original Python encoder: view the archived GitHub Gist.

Terminal output showing encoded bytes generated by the RNX2 Python encoder
The encoded byte sequence generated for use by the assembly decoder stub.

Assembly decoder

The assembly decoder removes the offset, performs the NOT operation, and applies the two XOR operations to every encoded byte. After processing all 25 bytes, execution transfers to the reconstructed payload.

Original assembly decoder: view the archived GitHub Gist.

The assembled decoder and encoded payload were then placed inside a small C execution wrapper for controlled testing.

Original C wrapper: view the archived GitHub Gist.

Terminal showing successful execution of the 76-byte RNX2 encoded payload and decoder stub
The complete encoded payload and decoder stub measured 76 bytes in the original lab.

Byte-level validation in GDB

The debugger walkthrough followed the first encoded byte through each inverse transformation. The encoded buffer begins with 0x2e, which ultimately returns to the original 0x31 byte.

GDB view of the 25-byte encoded payload before RNX2 decoding begins
The 25-byte encoded payload before the decoder loop begins.

Step 1: remove the offset

Subtracting decimal 131 changes the encoded byte from 0x2e to 0xab.

GDB showing the first RNX2 decoding step changing byte 0x2e to 0xab

Step 2: apply NOT

The bitwise NOT operation changes 0xab to 0x54.

GDB showing the RNX2 NOT operation changing byte 0xab to 0x54

Steps 3 and 4: reverse both XOR operations

XOR with 0xCF produces 0x9b; XOR with 0xAA restores the original 0x31 byte.

Debugger view showing the final XOR decoding transformations

The loop repeats for the remaining 24 bytes. Immediately before transferring execution, the complete original payload is visible in memory.

Debugger view showing the fully restored 25-byte payload before execution
The original payload restored after all RNX2 decoding operations.

Engineering takeaway

The important result was not the choice of constants. The exercise demonstrated how to design a reversible transformation, implement both sides consistently, account for payload length and control flow, and prove correctness byte by byte.

It also illustrates a persistent trade-off in payload engineering: encoding may alter byte patterns and avoid selected bad characters, but the decoder increases size and introduces its own recognisable structure. Simple encoding should not be confused with reliable modern evasion.

Original source material


2026 archival context

This article documents historical 32-bit Linux exploit-development research and tooling. Modern processors, compilers and operating systems commonly introduce protections that change build and execution behaviour. The work is retained for controlled labs, defensive understanding and the historical research record.

Authorisation notice: Test only on systems you own or are explicitly authorised to assess.

Written by: Nipun Jaswal

Rate it
Previous post

Similar posts