Support for XCOM2 War of the Chosen

[obi0ne]

Hi,

Found out your awsome work, thank you for that.
I've been pooking around the binary in attempt to support XCOM2: Wa of the Chosen save data as well.

Figures, i'll share what I have, would appreciate if you guys could share more knowledge on how you've obtain the knowledge, and hoping togther we can support new format.
And i'll up Pull Request with required changes.

Heres what's i've come by from what i've seen, regarding header info:

1. 4B : file version, for war of chosen it's 0x16 0x00 0x00 0x00
2. 4B : not sure, might be uncompressed size (different saved files had different values, eg. 3E 02 00 00)
3. 4B: unknown
4. 4B : unknown, seems always be 00 00 00 00
5. 4B: suspecting it to be game number
6. 4B: suspecting it to be save number
7. xcom_str: save description (with futurestic date)
8. xcom_str: time
9. xcom_str: map_command
10. 4B: suspecting it to be "tactical_save" (1 - tactical, 0 - Geo)
11. 4B: unknown ?
12. 4B (bool) : Auto Save ?
13. 4B: unknwon, seems 00 00 00 00 on my save games
14. xcom_str: language ("INT" for my case)
15. 16B: unknown ?
16. xcom_str: looks like some description ("UILibrary_MissionImages.Missions_Strategy_Generic", not sure what it's representing...)
17. xcom_str: again looks like description: "AUTOSAVE: Campgain 1, Mission 3" in my case, except it has no "game futuristic date" inside it (unlike field 7)
18. 4B: number of xcom_str ahead, has the number 5 on my end, and i have 5 xcom_strings
19. arr of xcom_str: sequence of xcom_str based on previous number, contains list of DLCs
20. 4B: number of XCOM installed PACKS
21. arr of xcom_str: sequence of xcom_str based on previous number, contains list of PACKs
22. 4B: unnown ? (contains 3 on my save)
23. 4B: unknown ? contains 1, possible some boolean ?
24. 8B : unknwon contains zeros on my side.
    25): xcom_str : unknown ? has "NeutralizeFieldCommander" on my side. maybe invetory llisting ?

Any owner can shed light on how he obtained the other knowledge, i might help with this one...

cheers.

[obi0ne]

note, the 24th above might be the mission name or something.

[tracktwo]

Hi, please see #21 for details on why XCOM2 isn't supported. It's not so much a problem with understanding the format as that it can't be unambiguously parsed without also having information from the script packages and that is a big job to implement.

I am not actively working on this project other than bugfixes, so that isn't something I have any plans to take on myself. I would definitely accept PRs that add full xcom2 support, though.

[obi0ne]

Currently I have no custom mods installed, so that's fine by me.
Also I assume, have partial support is better than no support at all...
I intend to invest some time with this & see if I can help with this.

I would appreciate if you could help or point me in the right direction as to the actual header & inlined blocks structures.
How can I verify the actual structure, did you relay on some other format unreal engine reverse, or EXE reverse (IDA pro) ?
or it was more trial & error, and see what happens when you modify it ?

Also, regarding the scripts mentioned on your reply, can you elaborate further on those ?
where are those scripts & what is exactly require, should i decide to provide full support ?

[obi0ne]

Can anyone help identify the yellow fields ?

[tracktwo]

The scripts I mentioned are the unrealscript packages for the game and for any mods that have custom code. The big one is xcomgame.upk, and you can build it yourself by installing the xcom2 mod tools SDK. A big chunk of the save file is serialized versions of these unrealscript objects.

You don't necessarily need to understand all the fields in the save, as long as you know how big they are you can just read that many bytes and treat them as some unknown byte values that need to be put back in exactly the same way when regenerating the save. Even for EW many of the fields are marked 'unknown' and the values are just passed through as-is. It does make it harder to use if those bytes represent something that you might want to change in the save, though. I don't remember exactly if we even know the size of all these unknown structs to be able to treat them as opaque, it's been many years since I looked into this.

Some helpful tools for investigating unreal files are UE Explorer (https://github.com/UE-Explorer/UE-Explorer) which you can use to dump info about a script package and these can be helpful to try to interpret what the serialized form of the data will look like. There are also some useful wiki pages on nexusmods (https://wiki.nexusmods.com/index.php/Modding_XCOM:EU_2012) although that's EW specific XCOM2 uses the same engine and the formats are only slightly different.

[obi0ne]

thanks for reply with info.
I will try to "play" with it, and see how far will I get.

[obi0ne]

weird, neither UE Explorer or UE viewer is able to view xcomgame.upk file.
Keeps getting errors:

Not sure if it's because somehow compressed, or a change in serialization format made during move to XCOM2 (i've read Unreal engine is 3.5, compared to version 3 on enemy unknown).

If i try to read enemy unknown files, it's goes little bit further but also "dies"

Are you able to view XCOM2 files on your pc ?
Maybe something in my desktop environment ?

[tracktwo]

If you're using the ones from the shipping game you need to decompress them first, there's a decompress tool here: https://www.gildor.org/downloads

If you build the packages from the mod tools SDK they are uncompressed by default.

[obi0ne]

If you're using the ones from the shipping game you need to decompress them first, there's a decompress tool here: https://www.gildor.org/downloads

If you build the packages from the mod tools SDK they are uncompressed by default.

Thanks for that, you were right, it was compressed.

The tool however only seems to work on XComGame.upk file for Enemy Within & Enemy unknown & not for XCOM2 :(
they must have modified serialization format during move to UE v3.5...

Will attempt to contact Gildor author & see if he can help with new's format...

[tracktwo]

Is the decompression failing or ue explorer failing to read the decompressed file? I was able to use ue explorer on the uncompressed ones I built myself.

If it's the decompression failing it may be that xcom2 uses a different compression algorithm. I do remember seeing this but thought it was only Chimera Squad that used the different compression type, but it might be XCOM2 too. There may be another version of the decompressor that supports the newer algorithm somewhere.

[obi0ne]

Is the decompression failing or ue explorer failing to read the decompressed file? I was able to use ue explorer on the uncompressed ones I built myself.

If it's the decompression failing it may be that xcom2 uses a different compression algorithm. I do remember seeing this but thought it was only Chimera Squad that used the different compression type, but it might be XCOM2 too. There may be another version of the decompressor that supports the newer algorithm somewhere.

decompression of the original (XCOM2 War of the chosen) XcomGame.upk fails.

Searching on the tool's authors (Gildor) forum, found some 2nd approval of that: https://www.gildor.org/smf/index.php/topic,7127.new.html#new

Attempted to bump it there, will see what he answers.

Seems that the decompressor detect the file as if it's UE1 (which obviously it is not) then it fails...

[obi0ne]

found another thread there with hope: https://www.gildor.org/smf/index.php/topic,2942.0.html
try to understand if this specific build is decompressor or viewer only.

EDIT:
working partially with this build: https://drive.google.com/file/d/110USn24A_rL2wytRu9B1-E_O2kYxFI8E/view

if i filter out texture, some texture there seem to crash it, not sure why

EDIT2: now crash after complaining cannot find SDL2.dll, will debug this one, not sure why not finding it...

EDIT3: got it decompressed with original decompressor using "-game=xcom2" tag.

[obi0ne]

I need some help.

You can see working extractor on code here: https://github.com/obi0ne/xcomsave/blob/61191112d2e5dea1f5059b62b0047a695b23c74a/xcomreader.cpp#L125
(on my fork).

That's output correct "output.dat" file with actor's table.
However, i'm getting some issue's decoding actor table.

begin of file looks like:

First number is number of actors (845 on my case, so not modulu 2, maybe same as android).
But followed by two numbers: 120(dec) & 2738 (dec).
Do you have nay idea what can be those ?
Attaching zip of decoded output for my case.
output.zip

[tracktwo]

Ah, magic numbers! I took a quick look at a couple of saves I had handy and I also see the same 0x34d and 0x78 numbers in every single one so they are not likely to be an actor count. The 3rd number changes from save to save, and looks to me like it might be the actual actor count (the number tends to grow if you look at saves much later in a campaign, and the size of the table also seems to grow so that tracks). Opening up xcomgame.u in UE explorer and looking at the package I can see "Version: 845" and "Licensee version: 120", so that explains those values, although it's interesting that they get emitted as part of the actor table (then again, the actual file structure is more or less all guesswork too so they might be a header in front of the actor table, or what I call the actor table might not even be an actor table).

Just by looking at the format of the entries I can already see they are different from the ones in EU and EW but each entry does appear to have the same size other than the variable string length. My advice would be to just keep plugging away with it and recording the values without trying to figure out what they all are ahead of time. As long as you can decode the layout of the table without necessarily understanding what each field really is you may still be able to make progress.

[obi0ne]

yea, it's looks to be like:
XCOM_STR | INT(4Bytes) | FF FF FF FF | INT (4Bytes) | FF FF FF FF

Weird thing is there seems to be 108 same entires, with following:
XComGameState | 00 00 00 00 | FF FF FF FF | 2C 01 00 00 | FF FF FF FF

up till offset 0xFE0
Does this make any sense ??

108 missions ? having some place holders ?

Afterward i begin seeing some different entries, such as: XComGameStateContext_ArchiveHistory , XComGameState_Item etc.

I'll try to play with it further, thanks for your help.