Commit graph

72 commits

Author SHA1 Message Date
derrod c287649e07 [cli] Make app name optional for "list-files"
If reading from a file/url the app name is not necessary.
2020-04-30 12:31:23 +02:00
derrod dc68617212 [cli] Change --prefix-filter to --prefix 2020-04-30 12:23:23 +02:00
derrod 5a197fe6a9 Version 0.0.6
'Residue Processing' is a fitting codename given all the cleanup...
2020-04-30 12:08:22 +02:00
derrod 4ef55c02f3 Update README with new commands/flags 2020-04-30 12:07:53 +02:00
derrod 39bfb47d75 [cli] Add csv/tsv option to list commands
Closes #6 and #13
2020-04-30 12:03:23 +02:00
derrod 31bf74b33d [cli] Set no_install flag when using --prefix options 2020-04-30 11:49:33 +02:00
derrod c9392ca22c [models] Clear raw data in manifest after it has been read
Minor optimization to save some RAM, especially for JSON manifests.
2020-04-30 11:44:43 +02:00
derrod 7dacd7ba35 [cli/core] Fix manifest saving/loading
The refactoring didn't take into account that the raw
manifest data still needs to be accessible. Since I did
not want to modify the manifest model to hold the raw
data (waste of RAM), just make the methods return the
raw data again (at least for now)
2020-04-30 11:42:09 +02:00
derrod 8b63c482c1 [cli] Add new "list-files" command to list files in a manifest
Useful for use with the new --prefix option to search for which
prefix to use for the files you want to download.

Also supports creating a file usable with sha1sum/hashcheck to
verify a local installation manually as well as outputting the
list in csv format for other uses.
2020-04-30 11:14:18 +02:00
derrod 6788b1257c [core] Refactor manifest loading 2020-04-30 11:10:34 +02:00
derrod 8f7db143a6 [downloader] Greatly simplify download task creation
This is a change to something that was so massively stupid and
overcomplicated that I feel like I need to explain and justify myself:

After figuring out the format for manifests and spending countless
hours staring at IDA/Ghidra I kinda was sick of that, so I decided to
figure out what to do with the manifest myself by playing around with
it, which was also a lot more fun than looking through disassembly.

When looking at the chunks and files it quickly became obvious that the
way they're created is by concatenating all files into 1 MiB chunks that
can be downloaded and reassmebled (mostly) sequentially. What I did not
know was how the order of files in this "stream" was determined.

In playing around with it I came up with the old method: essentially
forming a chain of files, because each file's end generally pointed to
the start of the next file. And it worked great! At least until now...

Yesterday somebody alerted me to a game where this failed and it took me
a bit to figure out. Essentially the chaining had failed because
multiple files started at the same offset, but some of them would follow
another chain that never went back to the chunk it started at,
effectively skipping those files. This was rather annoying to deal with,
I came up with a workaround but it wasn't pretty. So I decided to jump
back into IDA/Ghidra and find out how Epic does it for real.

Well it took me a while, but thanks to symbols (yay macOS!) and a decent
decompiler in Ghidra even a noob like me was able to find it eventually.
The answer is as simple as it can be: the files are sorted alphabetically
(case-insensitive).

So really all I ever had to do was to sort files alphabetically and then
run through them to create the list of tasks.

I feel so stupid.

P.S.: I tested a few games and for the most part the resulting file
processing order is identical between the old and the new method. The
cases where it differs is when there's heavy de-duplication happening
(e.g. Diabotical's small model files) but the runtime cache size remains
the same so both methods are equally efficient, the old one just can't
handle certain cases.
2020-04-30 11:05:26 +02:00
derrod ef0ea26372 [cli/core/downloader] Add support for filtering downloaded files by prefix 2020-04-29 22:08:28 +02:00
derrod d61ce7dc6d [models] Support manifests metas with build ids 2020-04-29 21:39:14 +02:00
derrod 6e5e281c82 [core] Fix pre-installation warnings
- Make errors unique
- FNA/XNA games don't seem to really work with WINE.
2020-04-29 15:41:21 +02:00
derrod 209ba2dd81 [core] Remove unnecessary OVT path adjustments
Not only does this not work (path.join() will not
actually add Z: on non-Windows), it's also not necessary.

Confirmed to work with Detroit: Become Human (Demo) and
Just Cause 4.
2020-04-29 15:16:17 +02:00
derrod 63ff306fbf [cli] Fix offline flag 2020-04-29 14:05:30 +02:00
derrod ba7133d7d6 [cli] clean up imports 2020-04-29 10:56:47 +02:00
derrod 7c9ea631ed Update README with new flags 2020-04-28 16:18:59 +02:00
derrod eaf77abe97 Version 0.0.5 2020-04-28 15:58:34 +02:00
derrod 9613ad34ba [core] Log when manifest/base url is being overridden by the user 2020-04-28 15:56:52 +02:00
derrod 2eac83f6d1 [models] Add missing flags to FileManifests 2020-04-28 15:55:19 +02:00
derrod 8251db22d2 [cli/core] Allow overriding platform and UE filter in list-games 2020-04-28 15:32:38 +02:00
derrod e0b4769af2 [cli/core] Add option to override platform when downloading
This can be used to download Mac or 32-bit builds if desired.

For example: legendary download Corydalis --platform Mac
Will download the Slime Rancher macOS build.

Closes #12
2020-04-28 15:12:17 +02:00
derrod 79dc7d03a4 [api] Allow specifying label and platform in API requests 2020-04-28 14:51:00 +02:00
derrod f5861546b4 [core] Support manifest URLs with CDN authentication 2020-04-28 07:57:10 +02:00
derrod 5dd7cd64cc Update python requirements in README
Turns out that
1) 32 bit python is still a thing and
2) it throws an exception when trying to attach Shared Memory
that is larger than 64 MiB. While we could work around that
for cases where more than 64 MiB cache is required, for now
it makes more sense to just drop 32 bit support. Most games are
64-bit only anyway.
2020-04-28 06:55:09 +02:00
derrod e822a8f7bf [core] Fix OwnershipToken path for WINE
Fixes #2
2020-04-28 06:28:23 +02:00
derrod 89fe49aa4d [core] Add warning for BattlEye anticheat
Closes #8
2020-04-28 06:20:25 +02:00
derrod 2be1d367df [cli] Add version and release codename to help 2020-04-28 06:14:17 +02:00
derrod 0d08263afa [lfs] Return empty installed list if installed is None
Fixes #9
2020-04-28 06:13:03 +02:00
derrod e44a00d37f [downloader] Cap worker processes at 16 2020-04-28 06:12:30 +02:00
derrod 7bf517aa00 [models] Fix build id string creation 2020-04-27 14:31:12 +02:00
derrod 5e7bdca8c4 Update README with info regarding Linux packages
Closes #3
2020-04-27 12:46:22 +02:00
derrod b860034031 Update README for linux single-binary distribution 2020-04-26 21:00:00 +02:00
derrod 1da92fe341 Version 0.0.4 2020-04-26 14:58:40 +02:00
derrod 1b3ef31963 [cli/core] Fix config offline and environment overrides 2020-04-26 14:58:09 +02:00
derrod 76b7360732 Update README 2020-04-26 13:44:51 +02:00
derrod b9b459d527 [cli] Fix missing space breaking URL in log message 2020-04-26 13:37:35 +02:00
derrod 6895f25fbb Version 0.0.3 2020-04-26 13:24:15 +02:00
derrod f4b1605b92 [core] Minor improvements for DLC/logging 2020-04-26 13:20:54 +02:00
derrod 50bd35683f [cli] Minor adjustments to parser setup and readability 2020-04-26 13:20:11 +02:00
derrod 0485a728e3 [cli/downloader] Migrate to Queue based logging for subprocesses 2020-04-26 13:19:28 +02:00
derrod 035e23b964 [core/lfs] Clean up local manifest saving/loading 2020-04-25 23:06:46 +02:00
derrod 8f23770e43 [core] Add warning for games that are not marked for offline use 2020-04-25 22:54:18 +02:00
derrod ec97cef264 [core] Add additional warning for Ownership Token requirement 2020-04-25 12:20:55 +02:00
derrod c20619d6e7 [cli] Rework CLI
- Class instead of giant main() function
- Uses subparsers for commands (cleaner)
- Will make future enhancements easier
2020-04-25 12:20:14 +02:00
derrod 96602d1890 [downloader/models] Minor code cleanups 2020-04-25 10:18:45 +02:00
derrod f1885639ae [downloader] Remove unnecessary imports 2020-04-25 09:25:12 +02:00
derrod 0e81b87fac [cli] Simplify "is game installed" check 2020-04-24 06:38:11 +02:00
derrod d8b629a66d [cli] Simplify manual login procedure 2020-04-24 06:37:40 +02:00