derrod 8f7db143a6 [downloader] Greatly simplify download task creation ... This is a change to something that was so massively stupid and overcomplicated that I feel like I need to explain and justify myself: After figuring out the format for manifests and spending countless hours staring at IDA/Ghidra I kinda was sick of that, so I decided to figure out what to do with the manifest myself by playing around with it, which was also a lot more fun than looking through disassembly. When looking at the chunks and files it quickly became obvious that the way they're created is by concatenating all files into 1 MiB chunks that can be downloaded and reassmebled (mostly) sequentially. What I did not know was how the order of files in this "stream" was determined. In playing around with it I came up with the old method: essentially forming a chain of files, because each file's end generally pointed to the start of the next file. And it worked great! At least until now... Yesterday somebody alerted me to a game where this failed and it took me a bit to figure out. Essentially the chaining had failed because multiple files started at the same offset, but some of them would follow another chain that never went back to the chunk it started at, effectively skipping those files. This was rather annoying to deal with, I came up with a workaround but it wasn't pretty. So I decided to jump back into IDA/Ghidra and find out how Epic does it for real. Well it took me a while, but thanks to symbols (yay macOS!) and a decent decompiler in Ghidra even a noob like me was able to find it eventually. The answer is as simple as it can be: the files are sorted alphabetically (case-insensitive). So really all I ever had to do was to sort files alphabetically and then run through them to create the list of tasks. I feel so stupid. P.S.: I tested a few games and for the most part the resulting file processing order is identical between the old and the new method. The cases where it differs is when there's heavy de-duplication happening (e.g. Diabotical's small model files) but the runtime cache size remains the same so both methods are equally efficient, the old one just can't handle certain cases.		2020-04-30 11:05:26 +02:00
..
__init__.py	First public test version of legendary	2020-04-14 15:40:41 +02:00
manager.py	[downloader] Greatly simplify download task creation	2020-04-30 11:05:26 +02:00
workers.py	[cli/downloader] Migrate to Queue based logging for subprocesses	2020-04-26 13:19:28 +02:00