Commit graph

259 commits

Author SHA1 Message Date
derrod fb7b9d4548 Update README 2020-05-06 16:06:24 +02:00
derrod 31530692ef [api/cli/core] Add extremely basic support for cloud saves
Currently only supports downloading all saves to a folder,
in the future it should support automatically extracting save
files to the proper directory (at least on Windows).
2020-05-06 15:40:04 +02:00
derrod 693ad3cefc Version 0.0.10 2020-05-05 16:27:14 +02:00
derrod b1ba25e2e0 [downloader] Improve reordering optimizations
With some titles such as Metro Exodus there is
even more duplication across files. While this
change does not manage to reduce the limit to
below the default 1 GiB limit, it does bring
it down by about 512 MiB.
2020-05-05 16:23:51 +02:00
derrod 67859fb4ac [downloader] Clean up unused parameter 2020-05-05 16:13:08 +02:00
derrod ac5856afe1 [utils] Apply reorder opts to Metro Exodus 2020-05-05 16:04:22 +02:00
derrod fefd414b8b [cli] Add --no-install alias for --download-only 2020-05-05 16:03:24 +02:00
derrod 9f7133362c [downloader] Make sure timeout is not None
Hopefully fixes #22
2020-05-05 16:03:12 +02:00
derrod 0e86791237 [downloader] Reformat manager 2020-05-05 13:21:55 +02:00
derrod 6f8da36947 [downloader] Update logging (more debug/cleanup) 2020-05-05 13:21:06 +02:00
Rodney f3afb5b393
Add PyPI link to README 2020-05-04 19:59:57 +02:00
derrod 27f11466d5 Version 0.0.9 2020-05-04 19:51:03 +02:00
derrod e54d13be37 [core] Add warning for EQU8 anti-cheat 2020-05-04 19:50:56 +02:00
derrod 441c619db0 [core] Fix disk space check 2020-05-04 19:50:44 +02:00
derrod 84bb14f252 Bump version to 0.0.8
This is to make sure downstream packages are upgraded
from the broken 0.0.7 release.
2020-05-04 16:33:24 +02:00
derrod 5330c23e68 Add missing dependencies to setup.py 2020-05-04 16:13:33 +02:00
derrod 0b99e56860 Add missing module to setup.py 2020-05-04 16:05:31 +02:00
derrod 8fd7e00683 Fix command name in README 2020-05-04 15:11:19 +02:00
derrod 8e93b4923d Version 0.0.7 2020-05-04 14:30:15 +02:00
derrod 3a08d526d5 [cli] Update CLI descriptions 2020-05-04 14:25:00 +02:00
derrod 4d0ac96bdc [cli/utils] Add aliases to download command
Eventually the "download" command will be changed to
set the --no-install flag by default and "update" will
become --update-only. For now remain backwards compatible
to give people a chance to adjust their scripts and habits.

Fixes #18
2020-05-04 14:21:25 +02:00
derrod bd3095a6be [utils] Add new helper for game specific workaround
In the future we might extend this with other features,
such as specific post-install things that some games
may require to work or preconfigured WINE settings.
2020-05-04 14:08:35 +02:00
derrod df55f18a78 [cli/core/downloader] Increase timeout and make it configurable
Fixes #16
2020-05-04 14:08:26 +02:00
derrod 69383c4788 [cli/core/downloader/utils] Add download reordering optimization
This is an optimization that aims to fix issues with some titles
such as World War Z that have lots of duplicated files resulting
in a very high runtime cache requirement.

The basic idea is to group files that share lots of chunks together
so the data can be removed from the cache sooner.

For most games this has little to no effect. For some titles with heavy
duplication it can reduce the RAM usage significantly however. For
instance the RAM requirements for World War Z are reduced from 5.3 GiB
to 499 MiB.

Partially fixes #17
2020-05-04 14:06:25 +02:00
derrod 3f27ea0296 [downloader] Fix formatting of required cache size
Scientific notation is cool and all but not
particularly useful to most people.
2020-05-02 22:02:53 +02:00
derrod c345662521 REAMDE: Fix wiki urls 2020-05-01 23:44:31 +02:00
derrod 261d0c636f [cli/core] Add some more logging 2020-05-01 19:57:42 +02:00
derrod 20a477d20f [core] Always get all base urls 2020-05-01 19:57:22 +02:00
Rodney 9f2b18fd90
Update README.md 2020-05-01 17:13:01 +02:00
derrod 5eb1b69c05 [models] Fix JSON manifest chunkinfo sha hash
This needs to be bytes() for __repr__() to work.
2020-05-01 15:15:39 +02:00
derrod e976d44fb9 [cli] Fix help metavar for install tags 2020-05-01 15:08:48 +02:00
derrod 85244f4c88 Update README and setup.py (filter for PyPI) 2020-05-01 15:08:21 +02:00
derrod 5557dc63ae [cli] only print install tags if present
Most games don't seem to use them anyway.
2020-05-01 14:36:50 +02:00
derrod 9b5620ca30 [cli/core/downloader] Support filtering by install tags
Also adds tsv option for list-files and fixes
no_install not being set with --exclude.

Install tags are only present in some titles, I'm not
entirely sure how EGL uses them. Perhaps to allow one
manifest to be used on different platforms? Or to only
download extra assets when the user wants to?

Either way, it's another filtering feature that may be
useful, though for now it's mostly another toy to explore
EPIC's distribution system with.
2020-05-01 14:34:34 +02:00
derrod 531af3f586 [cli/core/downloader] Add --exclude option to ignore files when downloading
Useful to exclude unnecessary files such as redistributables.

Can be used together with --prefix to exclude files that would
still match the specified --prefix.
2020-05-01 10:43:55 +02:00
derrod 28d7a0d2ca Update gitignore for local builds 2020-04-30 16:42:32 +02:00
derrod 0307092dc9 README/setup.py add PyPI distribution adjustments 2020-04-30 16:40:39 +02:00
derrod c287649e07 [cli] Make app name optional for "list-files"
If reading from a file/url the app name is not necessary.
2020-04-30 12:31:23 +02:00
derrod dc68617212 [cli] Change --prefix-filter to --prefix 2020-04-30 12:23:23 +02:00
derrod 5a197fe6a9 Version 0.0.6
'Residue Processing' is a fitting codename given all the cleanup...
2020-04-30 12:08:22 +02:00
derrod 4ef55c02f3 Update README with new commands/flags 2020-04-30 12:07:53 +02:00
derrod 39bfb47d75 [cli] Add csv/tsv option to list commands
Closes #6 and #13
2020-04-30 12:03:23 +02:00
derrod 31bf74b33d [cli] Set no_install flag when using --prefix options 2020-04-30 11:49:33 +02:00
derrod c9392ca22c [models] Clear raw data in manifest after it has been read
Minor optimization to save some RAM, especially for JSON manifests.
2020-04-30 11:44:43 +02:00
derrod 7dacd7ba35 [cli/core] Fix manifest saving/loading
The refactoring didn't take into account that the raw
manifest data still needs to be accessible. Since I did
not want to modify the manifest model to hold the raw
data (waste of RAM), just make the methods return the
raw data again (at least for now)
2020-04-30 11:42:09 +02:00
derrod 8b63c482c1 [cli] Add new "list-files" command to list files in a manifest
Useful for use with the new --prefix option to search for which
prefix to use for the files you want to download.

Also supports creating a file usable with sha1sum/hashcheck to
verify a local installation manually as well as outputting the
list in csv format for other uses.
2020-04-30 11:14:18 +02:00
derrod 6788b1257c [core] Refactor manifest loading 2020-04-30 11:10:34 +02:00
derrod 8f7db143a6 [downloader] Greatly simplify download task creation
This is a change to something that was so massively stupid and
overcomplicated that I feel like I need to explain and justify myself:

After figuring out the format for manifests and spending countless
hours staring at IDA/Ghidra I kinda was sick of that, so I decided to
figure out what to do with the manifest myself by playing around with
it, which was also a lot more fun than looking through disassembly.

When looking at the chunks and files it quickly became obvious that the
way they're created is by concatenating all files into 1 MiB chunks that
can be downloaded and reassmebled (mostly) sequentially. What I did not
know was how the order of files in this "stream" was determined.

In playing around with it I came up with the old method: essentially
forming a chain of files, because each file's end generally pointed to
the start of the next file. And it worked great! At least until now...

Yesterday somebody alerted me to a game where this failed and it took me
a bit to figure out. Essentially the chaining had failed because
multiple files started at the same offset, but some of them would follow
another chain that never went back to the chunk it started at,
effectively skipping those files. This was rather annoying to deal with,
I came up with a workaround but it wasn't pretty. So I decided to jump
back into IDA/Ghidra and find out how Epic does it for real.

Well it took me a while, but thanks to symbols (yay macOS!) and a decent
decompiler in Ghidra even a noob like me was able to find it eventually.
The answer is as simple as it can be: the files are sorted alphabetically
(case-insensitive).

So really all I ever had to do was to sort files alphabetically and then
run through them to create the list of tasks.

I feel so stupid.

P.S.: I tested a few games and for the most part the resulting file
processing order is identical between the old and the new method. The
cases where it differs is when there's heavy de-duplication happening
(e.g. Diabotical's small model files) but the runtime cache size remains
the same so both methods are equally efficient, the old one just can't
handle certain cases.
2020-04-30 11:05:26 +02:00
derrod ef0ea26372 [cli/core/downloader] Add support for filtering downloaded files by prefix 2020-04-29 22:08:28 +02:00
derrod d61ce7dc6d [models] Support manifests metas with build ids 2020-04-29 21:39:14 +02:00