This is an optimization that aims to fix issues with some titles
such as World War Z that have lots of duplicated files resulting
in a very high runtime cache requirement.
The basic idea is to group files that share lots of chunks together
so the data can be removed from the cache sooner.
For most games this has little to no effect. For some titles with heavy
duplication it can reduce the RAM usage significantly however. For
instance the RAM requirements for World War Z are reduced from 5.3 GiB
to 499 MiB.
Partially fixes#17
Also adds tsv option for list-files and fixes
no_install not being set with --exclude.
Install tags are only present in some titles, I'm not
entirely sure how EGL uses them. Perhaps to allow one
manifest to be used on different platforms? Or to only
download extra assets when the user wants to?
Either way, it's another filtering feature that may be
useful, though for now it's mostly another toy to explore
EPIC's distribution system with.
Useful to exclude unnecessary files such as redistributables.
Can be used together with --prefix to exclude files that would
still match the specified --prefix.
The refactoring didn't take into account that the raw
manifest data still needs to be accessible. Since I did
not want to modify the manifest model to hold the raw
data (waste of RAM), just make the methods return the
raw data again (at least for now)
Useful for use with the new --prefix option to search for which
prefix to use for the files you want to download.
Also supports creating a file usable with sha1sum/hashcheck to
verify a local installation manually as well as outputting the
list in csv format for other uses.
This is a change to something that was so massively stupid and
overcomplicated that I feel like I need to explain and justify myself:
After figuring out the format for manifests and spending countless
hours staring at IDA/Ghidra I kinda was sick of that, so I decided to
figure out what to do with the manifest myself by playing around with
it, which was also a lot more fun than looking through disassembly.
When looking at the chunks and files it quickly became obvious that the
way they're created is by concatenating all files into 1 MiB chunks that
can be downloaded and reassmebled (mostly) sequentially. What I did not
know was how the order of files in this "stream" was determined.
In playing around with it I came up with the old method: essentially
forming a chain of files, because each file's end generally pointed to
the start of the next file. And it worked great! At least until now...
Yesterday somebody alerted me to a game where this failed and it took me
a bit to figure out. Essentially the chaining had failed because
multiple files started at the same offset, but some of them would follow
another chain that never went back to the chunk it started at,
effectively skipping those files. This was rather annoying to deal with,
I came up with a workaround but it wasn't pretty. So I decided to jump
back into IDA/Ghidra and find out how Epic does it for real.
Well it took me a while, but thanks to symbols (yay macOS!) and a decent
decompiler in Ghidra even a noob like me was able to find it eventually.
The answer is as simple as it can be: the files are sorted alphabetically
(case-insensitive).
So really all I ever had to do was to sort files alphabetically and then
run through them to create the list of tasks.
I feel so stupid.
P.S.: I tested a few games and for the most part the resulting file
processing order is identical between the old and the new method. The
cases where it differs is when there's heavy de-duplication happening
(e.g. Diabotical's small model files) but the runtime cache size remains
the same so both methods are equally efficient, the old one just can't
handle certain cases.
Not only does this not work (path.join() will not
actually add Z: on non-Windows), it's also not necessary.
Confirmed to work with Detroit: Become Human (Demo) and
Just Cause 4.
This can be used to download Mac or 32-bit builds if desired.
For example: legendary download Corydalis --platform Mac
Will download the Slime Rancher macOS build.
Closes#12
Turns out that
1) 32 bit python is still a thing and
2) it throws an exception when trying to attach Shared Memory
that is larger than 64 MiB. While we could work around that
for cases where more than 64 MiB cache is required, for now
it makes more sense to just drop 32 bit support. Most games are
64-bit only anyway.
