Nick Sweeting
b5311d2c57
Merge branch 'dev' into plugins-browsertrix
2024-03-26 15:29:02 -07:00
Nick Sweeting
e48159b8a0
cleanup docker-compose by storing crontabs in data dir
2024-03-26 15:24:05 -07:00
Nick Sweeting
ac73fb5129
merge fixes
2024-03-26 15:22:40 -07:00
Nick Sweeting
b4c3aa5097
Merge branch 'main' into dev
2024-03-26 15:01:36 -07:00
Nick Sweeting
a4453b6f87
fix PERSONAS PERSONAS_DIR typo
2024-03-26 14:19:25 -07:00
jim winstead
8b1b01e508
Update to Django 4.2.x, now in LTS until April 2026
2024-03-25 17:46:01 -07:00
Nick Sweeting
c22df0b63a
Merge branch 'dev' into plugins-browsertrix
2024-03-18 14:43:06 -07:00
Nick Sweeting
c0b5dbcecb
create new data/personas dir to hold cookies and chrome profiles
2024-03-18 14:41:39 -07:00
Nick Sweeting
c5bb99dce1
explicitly use Default profile inside user data dir
2024-03-18 14:40:40 -07:00
Nick Sweeting
1fc5d7c5c8
add USER_AGENT config option to set all USER_AGENTs at once
2024-03-18 14:39:09 -07:00
Nick Sweeting
11b067a1ae
Merge branch 'dev' into plugins-browsertrix
2024-03-15 15:45:56 -07:00
jim winstead
5478d13d52
Add generic_jsonl parser
...
Resolves #1369
2024-03-14 15:42:29 -07:00
Nick Sweeting
ca2c484a8e
Add _EXTRA_ARGS for various extractors ( #1360 )
2024-03-14 01:55:09 -07:00
Nick Sweeting
48f4b12ae2
Use COOKIES_FILE to fetch page titles ( #1364 )
2024-03-14 01:52:44 -07:00
Nick Sweeting
099f7d00fe
Use feedparser for RSS parsing ( #1362 )
...
Fixes #1171
Fixes #870 (probably, would need to test against a Wallabag Atom file to
Fixes #135
Fixes #123
Fixes #106
2024-03-14 01:51:45 -07:00
Nick Sweeting
3512dc7e60
Disable searching for existing chrome user profiles by default
2024-03-14 00:58:45 -07:00
Ben Muthalaly
f4deb97f59
Add ARGS and EXTRA_ARGS for Mercury extractor
2024-03-05 21:15:38 -06:00
Ben Muthalaly
d8cf09c21e
Remove unnecessary variable length args for dedupe
2024-03-05 21:13:45 -06:00
Ben Muthalaly
5082d61613
Merge branch 'title-cookies-file' of https://github.com/benmuth/ArchiveBox into title-cookies-file
2024-03-05 02:03:03 -06:00
Ben Muthalaly
4686da91e6
Fix cookies being set incorrectly
2024-03-05 01:48:35 -06:00
Naomi Phillips
a729480b75
Add COOKIES_FILE support for singlefile extractor
2024-03-03 02:32:46 -05:00
Ben Muthalaly
d74ddd42ae
Flip dedupe precedence order
2024-03-01 14:50:32 -06:00
jim winstead
9f462a87a8
Use feedparser for RSS parsing in generic_rss and pinboard_rss parsers
...
The feedparser packages has 20 years of history and is very good at parsing
RSS and Atom, so use that instead of ad-hoc regex and XML parsing.
The medium_rss and shaarli_rss parsers weren't touched because they are
probably unnecessary. (The special parse for pinboard is just needing because
of how tags work.)
Doesn't include tests because I haven't figured out how to run them in the
docker development setup.
Fixes #1171
2024-03-01 11:25:45 -08:00
Nick Sweeting
a577d1ed23
Merge branch 'dev' into title-cookies-file
2024-02-29 21:29:36 -08:00
Nick Sweeting
fe11e1c2f4
check if COOKIE_FILE is file
2024-02-28 18:19:44 -08:00
jim winstead
ccabda4c7d
Handle list of tags in JSON, and be more clever about comma vs. space
2024-02-28 17:38:49 -08:00
jim winstead
178e676e0f
Fix JSON parser by not always mangling the input
...
Rather than by assuming the JSON file we are parsing has junk at the beginning
(which maybe only used to happen?), try parsing it as-is first, and then fall
back to trying again after skipping the first line
Fixes #1347
2024-02-27 14:48:19 -08:00
Ben Muthalaly
68326a60ee
Add cookies file to http request in download_url
2024-02-27 15:30:31 -06:00
Ben Muthalaly
4d9c5a7b4b
Add CHROME_EXTRA_ARGS
...
Also fix `YOUTUBEDL_EXTRA_ARGS`.
2024-02-23 18:40:03 -06:00
Ben Muthalaly
ab8f395e0a
Add YOUTUBEDL_EXTRA_ARGS
2024-02-23 15:40:31 -06:00
Ben Muthalaly
4e69d2c9e1
Add EXTRA_*_ARGS for wget, curl, and singlefile
2024-02-22 23:04:11 -06:00
Nick Sweeting
1ea7ac168a
Merge branch 'dev' into plugins-browsertrix
2024-02-22 04:51:31 -08:00
Nick Sweeting
8b9bc3dec8
minor fixes
2024-02-22 04:50:22 -08:00
Nick Sweeting
6a4e568d1b
new archivebox update speed improvements
2024-02-22 04:50:22 -08:00
Nick Sweeting
edabc47828
Merge branch 'dev' into plugins-browsertrix
2024-02-12 21:28:58 -08:00
Nick Sweeting
903c72fa88
fix typing errors
2024-02-12 21:28:52 -08:00
Nick Sweeting
fbb3c84be2
Merge branch 'dev' into plugins-browsertrix
2024-02-12 21:26:44 -08:00
Nick Sweeting
91c4641199
skip dir size calculation when path is too long
2024-02-12 21:26:34 -08:00
Nick Sweeting
19aefc85e6
fix get_system_user failing on uid 999 in k3s
2024-02-08 18:58:12 -08:00
Nick Sweeting
777694eb1a
add type hints to plugin config models
2024-02-08 05:55:02 -08:00
Nick Sweeting
97b185987d
add TODO to support archive.org-style urls
2024-02-08 05:54:34 -08:00
Nick Sweeting
b56bfe57f1
add CUSTOM_TEMPLATES=/data/templates default config in Docker
2024-02-08 05:54:18 -08:00
Nick Sweeting
c6faa9ab76
add extra information to headers extractor output
2024-02-06 22:22:03 -08:00
Nick Sweeting
a4bd441077
Update config.py fix trim-filenames
2024-01-31 01:59:43 -08:00
Nick Sweeting
308b493469
add pudb and type hints
2024-01-28 02:57:51 -08:00
Nick Sweeting
9861a4fb48
more mypy fixes
2024-01-28 02:57:51 -08:00
Nick Sweeting
eaa4a9c5ea
fix django core auth
2024-01-28 02:57:51 -08:00
Nick Sweeting
beb83f292d
mypy fixes
2024-01-28 02:57:51 -08:00
Nick Sweeting
4aee3b590e
Merge branch 'dev' into plugins-browsertrix
2024-01-24 12:34:41 -08:00
Nick Sweeting
e8772513ca
Update archivebox/core/auth_ldap.py
...
add log line
2024-01-24 11:00:42 -08:00
