Witold Baryluk
9131f32c22
[downloader] Avoid buffer copies in worker ( #621 )
...
This increases peek download speed from about 850MB/s to 960MB/s on my computer.
https://github.com/derrod/legendary/issues/620
2023-11-16 01:40:44 +01:00
derrod
450784283d
[cli/core/downloader] Add option to bind to IP(s)
2023-10-14 14:20:17 +02:00
derrod
96b155800a
[downloader] Check if files exist before running analysis
...
This allows additional SDL tags to be installed without going through a repair.
It will also now redownload deleted files if there's an update rather than just
trusting what the old manifest says should be installed locally.
2023-06-18 00:11:59 +02:00
derrod
0e23b8e4f0
Cherry-pick some Sourcery refactoring suggestions
2022-10-25 15:38:55 +02:00
derrod
202f07973a
[downloader/models] Calculate maximum disk space delta during installation
2022-01-02 13:06:59 +01:00
derrod
fc66f9f372
[downloader/mp] Also kill workers on interrupt if they don't exit themselves
2021-12-30 14:34:02 +01:00
derrod
21d62dcd76
[downloader/mp] Fix status message formatting for odd cache usage
2021-12-28 17:39:10 +01:00
derrod
d0d37c40e7
[models/downloader] Add task flag to make file executable
2021-12-01 20:45:17 +01:00
derrod
75cc194424
[downloader.mp] Fix exponential backoff calculation and log as info
2021-10-07 11:16:39 +02:00
derrod
2d366a7704
[downloader.mp] Add exponential backoff to retries
...
Fixes #347
2021-10-07 10:46:17 +02:00
derrod
d70d5a6521
[downloader] Fix ETA calculation for runtimes >= 1 hour
...
Closes #345
2021-10-02 19:00:09 +02:00
derrod
b319cb505c
[cli/downloader.mp] Explicitly set UTF-8 encoding for resume/repair file
...
Fixes #297
2021-09-28 02:42:17 +02:00
derrod
6746ce85a1
[downloader] Fix wrong property being used
...
This caused re-using old files to fail.
2021-09-01 02:38:58 +02:00
derrod
7ff16aad44
[core/downloader] Move downloader to new sub-package
...
Preparing to add a new AIO based downloader
2021-08-14 05:12:10 +02:00
derrod
9e01ae1c5b
[downloader] Update to use new dataclasses
2021-08-14 05:10:31 +02:00
derrod
dc381cacb0
[downloader] Adjust for changes in new dataclass attribute names
2021-08-11 09:21:19 +02:00
derrod
ee3b3fb9fe
Fix various typos and some style complaints
...
Closes #175
2021-01-02 06:53:21 +01:00
derrod
fd004db4b9
[downloader] Move optional file deletion tasks to the end
2020-12-16 12:32:19 +01:00
derrod
4d138c1a22
[downloader] Silently attempt to delete files not selected for install
...
This is to clean up when changes are made to the selected install tags
2020-12-16 12:29:26 +01:00
derrod
08c2008281
[downloader/models] Add silent deletion task flag
2020-12-16 12:28:37 +01:00
derrod
bece6ef5de
[downloader] Fix skipping unneeded chunks
...
This was erroneously removed in daeee2e
2020-12-16 12:14:31 +01:00
derrod
5e1896cf2c
[downloader] Make insufficient memory message error more helpful
2020-12-14 08:42:22 +01:00
derrod
daeee2eb8d
[downloader] Rework order optimizer to be significantly faster
...
Also allow for a larger amount of files to be optimized.
2020-12-14 08:04:28 +01:00
derrod
9c87f8ab4f
[downloader] Log optimization time and increase upper file limit
2020-12-14 06:28:03 +01:00
derrod
b7db0ac721
[api/core/downloader] Update User-Agents
2020-12-13 03:14:55 +01:00
derrod
1640a47d6a
[downloader] Correctly support empty install tag
2020-12-08 06:12:47 +01:00
derrod
7046b06f14
[api/core/downloader] User-Agent update
2020-11-01 14:34:36 +01:00
derrod
36d02fa5ce
[downloader] Fix original file offset when using delta manifests
2020-09-09 10:28:28 +02:00
derrod
e898fe03fc
[downloader] Remove delta manifest workarounds
2020-09-09 10:13:21 +02:00
derrod
a55f75d5e8
[core/downloader] Prevent file deletion when using delta manifest
...
This is technically not how we should do this. In theory we should
"overlay" the delta manifest over the proper one and simply add/replace
the chunk/file list entries with the one from the delta manifest.
However simply not deleting files also works for the time being since files
are rarely deleted anyways.
2020-09-06 05:57:28 +02:00
derrod
bd66d6c5dc
[api/core/downloader] Update User-Agents
2020-09-06 03:16:06 +02:00
derrod
e1fc3df180
[manager] Improve chunk reuse algorithm
...
This will mostly be important for delta manifests,
which are yet to be implemented.
2020-09-05 04:51:20 +02:00
derrod
5b4a6e6d9f
[downloader] Fix temporary file name in result handler
2020-08-06 09:28:25 +02:00
derrod
211f65c4c6
[core/downloader/api] Update User-Agents
2020-05-31 02:37:37 +02:00
derrod
6b88d93576
[downloader] Clean up progress and other logging
2020-05-22 14:31:49 +02:00
derrod
8ab63f0665
[downloader] Make resume file version-agnostic
...
This may result in less-optimal situations since
it effectively also disables patching. But it does
allow us to re-use the resume file mechanisms for
repairing broken installs.
2020-05-20 12:05:26 +02:00
derrod
724ef4848c
[downloader] Check if files in resume file actually exist
...
Also improves logging to make it obvious to users that a
resume file is being used.
2020-05-19 16:14:01 +02:00
derrod
b89afeaefd
[downloader] Check for success before adding to resume file
2020-05-19 14:46:59 +02:00
derrod
0a41a043ac
[api/downloader] Update user-agent
2020-05-15 10:25:39 +02:00
derrod
d6abbcc694
Cleanup! (Remove shebangs, unused files)
2020-05-15 07:01:30 +02:00
derrod
0f01e404a4
[cli/core/downloader] Support multiple filter arguments
...
Fixes #20
2020-05-14 16:32:08 +02:00
derrod
b1ba25e2e0
[downloader] Improve reordering optimizations
...
With some titles such as Metro Exodus there is
even more duplication across files. While this
change does not manage to reduce the limit to
below the default 1 GiB limit, it does bring
it down by about 512 MiB.
2020-05-05 16:23:51 +02:00
derrod
67859fb4ac
[downloader] Clean up unused parameter
2020-05-05 16:13:08 +02:00
derrod
9f7133362c
[downloader] Make sure timeout is not None
...
Hopefully fixes #22
2020-05-05 16:03:12 +02:00
derrod
0e86791237
[downloader] Reformat manager
2020-05-05 13:21:55 +02:00
derrod
6f8da36947
[downloader] Update logging (more debug/cleanup)
2020-05-05 13:21:06 +02:00
derrod
df55f18a78
[cli/core/downloader] Increase timeout and make it configurable
...
Fixes #16
2020-05-04 14:08:26 +02:00
derrod
69383c4788
[cli/core/downloader/utils] Add download reordering optimization
...
This is an optimization that aims to fix issues with some titles
such as World War Z that have lots of duplicated files resulting
in a very high runtime cache requirement.
The basic idea is to group files that share lots of chunks together
so the data can be removed from the cache sooner.
For most games this has little to no effect. For some titles with heavy
duplication it can reduce the RAM usage significantly however. For
instance the RAM requirements for World War Z are reduced from 5.3 GiB
to 499 MiB.
Partially fixes #17
2020-05-04 14:06:25 +02:00
derrod
3f27ea0296
[downloader] Fix formatting of required cache size
...
Scientific notation is cool and all but not
particularly useful to most people.
2020-05-02 22:02:53 +02:00
derrod
9b5620ca30
[cli/core/downloader] Support filtering by install tags
...
Also adds tsv option for list-files and fixes
no_install not being set with --exclude.
Install tags are only present in some titles, I'm not
entirely sure how EGL uses them. Perhaps to allow one
manifest to be used on different platforms? Or to only
download extra assets when the user wants to?
Either way, it's another filtering feature that may be
useful, though for now it's mostly another toy to explore
EPIC's distribution system with.
2020-05-01 14:34:34 +02:00
