Decrypting a Malware C2 Protocol From a Single PCAP

A hunt team handed me one thing: a capture.pcap from a host they suspected was beaconing. No sample, no disk image, no strings. Just bytes on the wire. This is the story of turning that traffic back into plaintext, and the Rust decoder I wrote to do it. Hosts and keys are from a lab replica.

First look: shape before content

Before touching crypto, characterize the traffic. Shape tells you more than any single packet.

tshark -r capture.pcap -q -z conv,tcp | sort -k7 -n | tail
# 10.0.0.5:51044 <-> 45.83.12.9:443   482 pkts   61 kB

# Beaconing has a heartbeat. Extract inter-arrival deltas:
tshark -r capture.pcap -Y 'ip.dst==45.83.12.9' -T fields -e frame.time_delta \
  | awk '{s+=$1; n++} END {print "mean interval:", s/n, "s"}'
# mean interval: 60.4 s

A steady 60 second cadence to a single external IP on 443, but Wireshark shows it is not TLS. The first client bytes are not a ClientHello. Someone is hiding a custom protocol under a common port. Good.

The handshake: find the structure

Dump the first message each way and stare at the hex. Structure hides in the lengths.

tshark -r capture.pcap -Y 'tcp.stream==0 && tcp.len>0' \
  -T fields -e tcp.len -e data.data | head -4
# 20   a17f00104b3c9e2f...   <- client -> server, 20 bytes
# 8    5f01000c9a...          <- server -> client, 8 bytes

The first client packet is exactly 20 bytes and starts with a1 7f. Every client message starts with a1 7f. That is a magic. The next two bytes look like a little-endian length. Let me lay the header out:

offset  bytes         meaning
0       a1 7f         magic
2       00 10         payload_len = 0x0010 = 16 (LE)
4..20   4b 3c 9e ...  16 bytes of payload

So the framing is magic(2) | len_le(2) | payload(len). Clean, custom, and completely made up by the author. Now the payload.

Breaking the payload encoding

Sixteen bytes of payload in the first client message, high entropy, no ASCII. Two cheap hypotheses to test first: single-byte XOR, or a rolling XOR keystream.

Single-byte XOR fails fast (no byte value produces printable structure across messages). But messages of the same command have identical prefixes that then diverge, which is the signature of a rolling keystream seeded per session. The seed is usually sent in the clear during the handshake. Remember that 8 byte server reply? First four bytes 5f 01 00 0c: another magic and length. The remaining four bytes are the session seed.

I reconstructed the keystream as RC4 seeded with key = sha256(seed)[:16], a very common lazy construction. Here is the decoder.

/// Minimal RC4. Do not use this for anything real; we only need to *read*
/// what the malware author wrote.
struct Rc4 {
    s: [u8; 256],
    i: u8,
    j: u8,
}

impl Rc4 {
    fn new(key: &[u8]) -> Self {
        let mut s = [0u8; 256];
        for (i, b) in s.iter_mut().enumerate() {
            *b = i as u8;
        }
        let mut j = 0u8;
        for i in 0..256 {
            j = j.wrapping_add(s[i]).wrapping_add(key[i % key.len()]);
            s.swap(i, j as usize);
        }
        Rc4 { s, i: 0, j: 0 }
    }

    fn apply(&mut self, buf: &mut [u8]) {
        for b in buf.iter_mut() {
            self.i = self.i.wrapping_add(1);
            self.j = self.j.wrapping_add(self.s[self.i as usize]);
            self.s.swap(self.i as usize, self.j as usize);
            let k = self.s[(self.s[self.i as usize]
                .wrapping_add(self.s[self.j as usize])) as usize];
            *b ^= k;
        }
    }
}

And the framing parser that walks a reassembled TCP stream:

use sha2::{Digest, Sha256};

struct Frame<'a> {
    payload: &'a [u8],
}

fn parse_frames(stream: &[u8]) -> Vec<Frame<'_>> {
    let mut frames = Vec::new();
    let mut off = 0;
    while off + 4 <= stream.len() {
        // magic a1 7f, then u16 little-endian length
        if &stream[off..off + 2] != [0xa1, 0x7f] {
            break;
        }
        let len = u16::from_le_bytes([stream[off + 2], stream[off + 3]]) as usize;
        let start = off + 4;
        if start + len > stream.len() {
            break;
        }
        frames.push(Frame { payload: &stream[start..start + len] });
        off = start + len;
    }
    frames
}

fn session_key(seed: &[u8]) -> [u8; 16] {
    let digest = Sha256::digest(seed);
    let mut key = [0u8; 16];
    key.copy_from_slice(&digest[..16]);
    key
}

Plaintext, at last

Feed the client stream through the parser and RC4, and the payloads turn into something with obvious structure: a one byte opcode followed by arguments.

fn decrypt_session(seed: &[u8], stream: &[u8]) {
    let mut rc4 = Rc4::new(&session_key(seed));
    for frame in parse_frames(stream) {
        let mut buf = frame.payload.to_vec();
        rc4.apply(&mut buf);
        let opcode = buf[0];
        let arg = String::from_utf8_lossy(&buf[1..]);
        println!("op=0x{opcode:02x} arg={arg:?}");
    }
}

Output:

op=0x01 arg="WIN-DC01\\svc_backup"     # check-in: hostname + user
op=0x10 arg="whoami /priv"             # tasked command
op=0x11 arg="SeBackupPrivilege Enabled"# command result
op=0x10 arg="reg save HKLM\\SAM sam.hi"# credential theft, live
op=0x20 arg=""                         # sleep, next beacon in 60s

There it is. Opcode 0x01 is check-in with hostname\user. 0x10 is a tasked shell command, 0x11 is its result, 0x20 is the sleep marker that matches the 60 second cadence we measured at the very start. The operator was dumping the SAM hive.

What the capture gave the defenders

From one PCAP, with no binary, we recovered:

a network signature: a1 7f magic on any port, trivial to write as a Suricata rule and worth far more than an IP block
the C2 IP and cadence for scoping other beacons in the fleet
operator intent: they had SeBackupPrivilege and went straight for SAM, which tells IR exactly what to rotate

alert tcp any any -> any any (msg:"custom beacon magic a17f";
  content:"|a1 7f|"; offset:0; depth:2; sid:9001001; rev:1;)

The lesson

You do not always get the sample. But a protocol is a contract, and a contract leaves fingerprints: fixed magics, length fields, a seed exchanged in the clear, a lazy RC4. Shape first, then structure, then the smallest crypto assumption that fits, and iterate. The malware author's convenience is the reverser's foothold.