Parsing Command & Conquer: Generals replay files

This post was originally published on Tim's blog.

TL/DR: In this post we'll examine C&C Generals replay (.rep) files in detail, figure out how they work, and then look at a very early implementation of a replay viewer that I've added to OpenSAGE.

Have you ever saved a replay from [insert your favourite video game here], and wondered exactly what the replay file contains? I don't know how many people will answer "yes" to that question, but I'm one of them. For efficiency and IP-protection reasons, replay files are usually stored in a binary format, so if you open one of them in Notepad, all you'll see are a bunch of garbled characters. It's often a long road from there, to fully understanding and being able to parse one of these files.

One of the features I hope to support in OpenSAGE is being able to use replay files from the original Command & Conquer: Generals and Zero Hour games. Since no official description of Generals' replay file format has been published, I have to do it the hard way, and figure out the file format... but fortunately, I'm not the first to want to do this.

I should note here that replay files vary wildly between games, so what applies to one specific RTS game won't apply directly to other games, but the general principles should be similar, at least across RTS games.

Before we get into the details, let's start with the fun part: creating a replay file in the first place.

Creating a replay file

Let's load C&C Generals, and play a quick skirmish game. For the purposes of this blog post, I want to keep the replay file really simple, so all I'll do is start a single-player skirmish game, build one building, and exit. Here's what that looks like:

Here's the replay file that Generals saved for that game:

• simple-generals-replay.rep

First look

If we open that replay file in VSCode using the excellent hexdump extension, we'll see this:

You can immediately see some human-readable text there. That's always nice to see, when you're parsing an unknown binary format.

It was at this point, when I first looked at Generals' replay files, that I wondered whether anybody else had figured out the file format before. So I Googled and found the cnc-replayreaders repo on GitHub, which contains at least some work on parsing replay files for many SAGE games, including, happily, C&C Generals. Without this information, proceeding further would have been much harder. Here's the C++ code for a prototype version of a Generals replay parser. There's even a document with the beginnings of a spec for these replay files. This is great stuff! Many thanks to louisdx, who I believe also goes by the name R Schneider on gamereplays.org. I'll duplicate the Generals part of it here:

Generals, Zero Hour, Battle for Middle Earth, Rise of the Witch King

The replay format of the games Generals, Zero Hour, Battle for Middle Earth, Battle for Middle Earth 2, and its expansion Rise of the Witch King is considerably simpler than that of the later games. Most notably, the chunks do not have length information, so one must know their sizes by other means. Also, there is no footer, just a final chunk.

Header

The size u1 is 12 for Generals and Zero Hour, and 21 in the BfME games; u2 is 8 in CCG/ZH and 13 in the BfME games; u3 is 4 except in BfME2, where it is 6.

char             str_magic[6/8];     // == "GENREP", "BFMEREPL" or "BFME2RPL"
uint32_t         timestamp_begin;
uint32_t         timestamp_end;
byte             unknown1[u1];
tb_str           str_filename;
tb_ch            date_time[8];       // as in TW/KW
tb_str           str_version;
tb_str           str_build_date;
uint16_t         version_minor;
uint16_t         version_major;
byte             magic_hash[u2]
char             header[];           // null-terminated!
uint16_t         unknown2;
uint32_t         unkonwn3[u3];

Remarks. The two timestamps appear to indicate the real-world start and end time of the game. The header is of the same format as the later games.

The replay body

The replay body consists of a sequence of chunks. Each chunk is of the following form:

uint32_t         time_code;
uint32_t         chunk_code;
uint32_t         number;         // Player number?
byte             number_of_commands;
struct { byte cmd, byte nargs } signature[number_of_commands];
byte             arguments[];    // variable!

In the simplest case, number_of_commands is zero and signature and arguments are empty. Otherwise, there are number_of_commands many pairs of bytes {cmd,nargs}, where cmd refers to one of about ten commands and nargs is the number of arguments. The size of an argument depends on the command type. Finally, arguments consists of all the arguments of all the commands, simply one after the other. The meaning of the chunk codes is unknown in general. Only a handful of values seem to occur, all in the range 0x300 - 0x500, and 0x1B, 0x1D. Many codes only ever seem to appear with a fixed, specific signature of commands. These are the known commands and their argument sizes.

command	argument size (bytes)	Description/Notes
0x00	4	?
0x01	4	?
0x02	1	?
0x03	4	?
0x04	4	?
0x06	12	3 uint32_t's? 1 long double?
0x07	12	3 uint32_t's? 1 long double?
0x08	16	4 uint32_t's?
0x09	4/16	4 bytes in BFME2, 16 otherwise
0x0A	4	4 uint32_t's?

There is no footer. The final chunk has `number_of_commands` set to zero, and it appears to have command code 0x1B or 0x1D.

Parsing the header

Armed with that information, let's get started with parsing the header. I'll use C# here, because that's what we're using in the OpenSAGE project, but really any language that gives you the ability to read from a byte stream will work.

Before we can parse anything, we need to open a BinaryReader for the replay file:

using (var stream = File.OpenRead("simple-generals-replay.rep"))
using (var reader = new BinaryReader(stream, Encoding.Unicode, true))
{
    // TODO: Parse header
    // TODO: Parse body
}

Now we can parse the header, based on the description above. I've commented most lines with the actual values from our sample replay file.

string magic;
using (var asciiReader = new BinaryReader(stream, Encoding.ASCII, true))
{
    magic = asciiReader.ReadFixedLengthString(6); // "GENREP"
}

var origin = new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc);
DateTime readTimestamp(BinaryReader reader) => origin.AddSeconds(reader.ReadUInt32());

var timestampBegin = ReadTimestamp(reader); // "2018-03-05 1:28:17 UTC"
var timestampEnd = ReadTimestamp(reader);   // "2018-03-05 1:28:42 UTC"

var numTimecodes = reader.ReadUInt16(); // "577"

var unknown1 = reader.ReadBytes(12); // Sometimes all zeros, and sometimes not

var filename = reader.ReadNullTerminatedString(); // "Last Replay"

var dateTime = ReplayTimestamp.Parse(reader); // "2018-03-05 21:28:17.402 Monday"

var version = reader.ReadNullTerminatedString(); // "Version 1.7"
var buildDate = reader.ReadNullTerminatedString(); // "Sep  6 2012 09:43:29"

var versionMinor = reader.ReadUInt16(); // 7
var versionMajor = reader.ReadUInt16(); // 1

var magicHash = reader.ReadBytes(8); // 236, 122, 38, 27, 251, 102, 39, 119

var metadata = reader.ReadNullTerminatedAsciiString(); // "M=07maps/desert fury;MC=83506418;MS=214497;SD=625705968;C=100;S=HDESKTOP-9V9469L,0,0,TT,-1,2,-1,-1,1:CH,-1,-1,-1,-1:X:X:X:X:X:X:;"

var unknown2 = reader.ReadUInt16(); // "48"

var unknown3 = reader.ReadUInt32(); // "1"
var unknown4 = reader.ReadUInt32(); // "2"
var unknown5 = reader.ReadUInt32(); // "0"

var gameSpeed = reader.ReadUInt32(); // "30"

So, the header spec in the cnc-replayreaders project looks pretty accurate. I've worked out that one of the unknowns is the game speed. I don't know about the others, but by comparing enough replays with known game states, it should be possible to work them out.

This already gives us quite a lot of interesting data, such as the map, player details, start and end times, and even the date that the version of Generals that I'm using was compiled (September 6th 2012, apparently).

This is all very well, but what I'm really interested in is the actual replay data.

Parsing the body

There are many ways game replay data can be saved. If there aren't too many "things" in the world to keep track of, you could store the entire state of the world at every timestep. For RTS games with potentially hundreds or thousands of units, this wouldn't scale - and so these games usually only save player inputs, and use those inputs to recreate the entire simulation. (This is also why you often can't rewind replays; you'd have to start from the beginning again, and simulate every frame up to the one you're interested in, in order to get the correct state for that frame.)

Knowing that, what I'm expecting to see in the body of the replay file is a sequence of commands (or "orders", as I've chosen to call them), roughly corresponding to the actions I performed in the game. For example, "select unit" and "create power plant". Each of those orders would have arguments relevant to the order type. For example, "select unit" would have some sort of identifier to store which unit was selected.

There may be (and in fact, are) some orders that weren't triggered by me, such as an "end game" order - anything that the game can't recreate by itself needs to be saved in the replay file.

Fortunately for us, the cnc-replayreaders spec appears to match this concept, so we're on the right track. It looks like a "chunk" is probably a single order, and every order has 0 or more arguments.

The body of a replay file is composed of n chunks. There's no way to know how many chunks there are from the header - after reading the header, you just have to keep reading chunks until you get to the end of the file. Like this:

while (reader.BaseStream.Position < reader.BaseStream.Length)
{
    // TODO: Parse chunk
}

Let's look at the basic structure of a chunk. Then we can impose that structure onto our sample replay file and see what we get.

Let's use the description quoted above, and convert it into the code to parse a chunk header:

var timecode = reader.ReadUInt32(); // 0-based timecode
var orderType = reader.ReadUInt32AsEnum<OrderType>(); // See below
var number = reader.ReadUInt32(); // 0-based index of this chunk

var numUniqueArgumentTypes = reader.ReadByte(); // Number of unique argument types

// Pairs of {argument type, count}.
var argumentCounts = new (OrderArgumentType argumentType, byte count)[numUniqueArgumentTypes];
for (var i = 0; i < numUniqueArgumentTypes; i++)
{
    // Read the number of arguments for this argument type.
    argumentCounts[i] = (reader.ReadByteAsEnum<OrderArgumentType>(), reader.ReadByte());
}

// TODO: Read argument data

We don't know what orderType means yet; we'll have to look at the chunks, once we've parsed them, and make some guesses. We'll do that below.

argumentCounts stores the number of arguments for each argument type. So there might be 2 integer arguments, 1 boolean argument, and 1 float argument, for example. We can calculate the total number of arguments by summing the counts of each argument type.

Our first order of business is working out how many bytes each argument value uses, because without knowing that, we don't know where the next chunk starts.

There's nothing in the chunk header that directly tells us how big the argument data array is. We need to derive it from the number and type of arguments. I worked out the argument types from looking at a few replay files in a hex viewer and making some educated guesses. This fills in the gaps in the table of commands that I quoted above. Here's what I've found so far, although I'm sure there are more:

enum OrderArgumentType : byte
{
    Integer = 0,       // int32 (4 bytes)
    Float = 1,         // float32 (4 bytes)
    Boolean = 2,       // bool (1 byte)
    ObjectId = 3,      // uint32 (4 bytes)
    Position = 6,      // float32 * 3 (12 bytes)
    ScreenPosition = 7 // int32 * 2 (8 bytes)
}

Now we can write the code to parse order arguments. Note that, to keep things simple, I'm not doing anything with the parsed variables, but in the real code I do.

for (var i = 0; i < numUniqueArgumentTypes; i++)
{
    ref var argumentCount = ref argumentCounts[i];
    var argumentType = argumentCount.argumentType;

    for (var j = 0; j < argumentCount.count; j++)
    {
        switch (argumentType)
        {
            case OrderArgumentType.Integer:
                var i = reader.ReadInt32();
                break;

            case OrderArgumentType.Float:
                var f = reader.ReadSingle();
                break;

            case OrderArgumentType.Boolean:
                var b = reader.ReadBooleanChecked();
                break;

            case OrderArgumentType.ObjectId:
                var oi = reader.ReadUInt32();
                break;

            case OrderArgumentType.Position:
                var p = reader.ReadVector3();
                break;

            case OrderArgumentType.ScreenPosition:
                var sp = reader.ReadPoint2D();
                break;

            default:
                throw new InvalidOperationException();
        }
    }
}

Armed with all that code, we can finally parse our sample replay file. Let's do that, and write out all the replay chunks from our sample replay file. I've trimmed it down, because the replay file contains 733 lines - i.e. 733 orders - most of which are duplicate 1092 orders. I've marked the trimmed lines with "...".