Solving the "Custom Cipher" CTF Challenge

In this challenge, we are given a Python script that encrypts messages, an output.txt file containing the scrambled encrypted message (the "ciphertext") and a scrambled look-up table (an "SBox"), and a hint that "Every copy of my cryptosystem is personalized". Our goal is to find the hidden flag inside the encrypted message.

Step 1: Understanding the Encryption Process

First, we look at the script custom_cipher.py to see how the encryption works. It encrypts a message in chunks of 16 characters (bytes). For each 16-character chunk, it performs three operations:

1. Mixing with Key 0 (k0): It mixes the text with a secret 16-character master key. This mixing uses an operation called "XOR".
2. Substitution using an SBox: It replaces each character with another character based on a scrambled lookup table (the "SBox"). The challenge conveniently gives us this exact SBox in output.txt.
3. Mixing with Key 1 (k1): It mixes the result again with a second key (k1). This second key is generated by taking a digital fingerprint (an MD5 hash) of the master key (k0).

Step 2: Spotting the Weakness

The same keys (k0 and k1) are used for every single 16-character chunk. Because we know what the beginning of the original text is ("I have a secret, please don't share: "), we can perform what is called a "known-plaintext attack".

If we take two different chunks of the encrypted text and XOR them together, the XOR with k1 at the end mathematically cancels itself out! So, Encrypted_chunk_1 XOR Encrypted_chunk_2 will equal SBox(Plaintext_chunk_1 XOR k0) XOR SBox(Plaintext_chunk_2 XOR k0). Notice that k1 has completely disappeared from this equation.

Step 3: Finding the Master Key (k0)

Since we know the first two chunks of the original text and we have the SBox, the only unknown variable left in our equation is the master key, k0. Because the encryption operates one character at a time, we can simply guess all 256 possible values for each character of k0 independently. For each guess, we check if our equation from Step 2 holds true.

This drastically narrows down the possible values for each character of k0 to just 1 or 2 options. We then quickly write a Python script (solve.py) to test all the remaining combinations, calculate what k1 would be (its MD5 hash), and see which combination successfully encrypts the first chunk exactly as it appears in output.txt.

Step 4: Decrypting the Flag

Once our script cracks the correct k0 and k1, we just reverse the encryption process to reveal the hidden flag in the rest of the text. Reversing it simply involves:

1. Mixing (XORing) with k1.
2. Un-scrambling the substitution using an Inverse SBox (which we can create from the SBox we have).
3. Mixing (XORing) with k0.

Doing this gives us the original message and the flag!

The Flag: byuctf{if_you_used_a_llm_youre_missing_out_learning_a_really_cool_attack_!}