Nick Sweeting
166573d417
autodetect qemu and warn about broken chrome in emulated containers
2023-10-31 00:23:19 -07:00
Nick Sweeting
62e077a5bc
Merge pull request #1241 from overhacked/fts5-search
2023-10-30 18:49:52 -07:00
Nick Sweeting
851f59b4ba
Merge pull request #1254 from c01o/dev
2023-10-30 18:44:55 -07:00
mAAdhaTTah
18e89bee3d
Fix readwise token
...
Accidentally shipped this hard-coded. Token is long-size revoked,
but obviously hard to use this way!
2023-10-29 17:27:04 -04:00
Nick Sweeting
a680724367
Merge branch 'dev' into search_index_extract_html_text
2023-10-27 23:09:28 -07:00
c01o
8c25a81009
Remove playwright from armv7
2023-10-27 05:59:25 +09:00
Ross Williams
310b4d1242
Add htmltotext extractor
...
Saves HTML text nodes and selected element attributes in
`htmltotext.txt` for each Snapshot. Primarily intended to be used
for search indexing.
2023-10-23 21:42:32 -04:00
Nick Sweeting
63ad43f46c
Merge branch 'dev' into method_allow_deny
2023-10-20 04:25:44 -07:00
Nick Sweeting
53355be46a
Merge branch 'dev' into fts5-search
2023-10-20 04:23:50 -07:00
Nick Sweeting
82d8662c74
add more readability error output
2023-10-20 04:14:28 -07:00
Nick Sweeting
76f9b91ed3
dockerfile fixes
2023-10-20 04:08:38 -07:00
Nick Sweeting
86c662421b
show FS_USER in version output debug string
2023-10-20 02:47:23 -07:00
Nick Sweeting
7910a5b34b
fix pretty_path printing logic
2023-10-20 02:47:03 -07:00
Nick Sweeting
d4ca2d1154
update mercury bin path to postlight parser
2023-10-20 02:46:41 -07:00
Nick Sweeting
63c276a93d
redirect add page back to snapshots list automatically
2023-10-19 18:24:19 -07:00
Nick Sweeting
22bcffe5ee
remove accidental duplicate template code
2023-10-19 18:24:05 -07:00
Nick Sweeting
85f8583d62
catch ldap configuration and packaging errors and make them non fatal
2023-10-19 18:05:59 -07:00
Nick Sweeting
d286dca925
better to log before doing a thing than after
2023-10-18 11:47:55 -07:00
Ben Muthalaly
9e6a87114b
Fix formatting, logging, logic issues
2023-10-18 12:07:36 -05:00
Ben Muthalaly
521ea70e0c
Add check for existing user, change varable names
2023-10-18 03:07:54 -05:00
Ross Williams
9b85f35b63
sqlite search: check SQLite version when indexing
...
If creating the FTS5 tables fails due to a known version
incompatiblity, report the required version to the user.
2023-10-16 14:50:48 -04:00
Ross Williams
1e604a1352
sqlite search: clean up errors and type-checking
...
Clean up error handling, and report a better error message
on search and flush if FTS5 tables haven't yet been created.
Add some mypy comments to clean up type-checking errors.
2023-10-16 14:31:52 -04:00
Ross Williams
adb9f0ecc9
sqlite search: Rename connection
to get_connection
...
`connection` could cause confusion with `django.db.connection` and
`get_connection` is a better callable name.
2023-10-16 13:23:26 -04:00
Ross Williams
e0f8eeeaa7
Improve search.backends.sqlite retry logic
...
Retry with table creation should fail if it is attempted for a second
time.
2023-10-16 13:23:26 -04:00
Ross Williams
8fe5faf4d0
Introduce SQLite FTS5-powered search backend
...
Use SQLite's FTS5 extension to power full-text search without any
additional dependencies. FTS5 was introduced in SQLite 3.9.0,
[released][1] in 2015 so should be available on most SQLite
installations at this point in time.
[1]: https://www.sqlite.org/changes.html#version_3_9_0
2023-10-16 13:23:26 -04:00
Ross Williams
c53ec45a29
WIP: add sqlite search backend boilerplate
2023-10-16 13:23:26 -04:00
Ross Williams
6555719489
Add space after tags when extracting text
...
Add space after any close tag to ensure that
tokens that would be rendered separate in HTML
get extracted as separate tokens in text.
Example:
`<p>First</p><p>Second</p>` --> `First Second`
NOT `FirstSecond`
2023-10-16 09:59:08 -04:00
Ben Muthalaly
44a94157be
Remove logging of configured username and password
2023-10-15 23:36:47 -05:00
Ben Muthalaly
11d473e536
Add config options to add admin user on first run
2023-10-14 00:38:04 -05:00
Ross Williams
d8aa84ac98
Make extracting text for indexing optional
...
Add a configuration option to enable/disable HTML text extraction
for indexing
2023-10-12 13:14:39 -04:00
Ross Williams
b6a20c962a
Extract text from singlefile.html when indexing
...
singlefile.html contains a lot of large strings in the form of `data:`
URLs, which can be unnecessarily stored in full-text indices. Also,
large chunks of JavaScript shouldn't be indexed, either, as they pollute
search results for searches about JS functions, etc.
This commit takes a blanket approach of parsing singlefile.html as it is
read and only outputting text and selected textual attributes (like
`alt`) for indexing.
2023-10-12 13:06:35 -04:00
Ben Muthalaly
77917e9b55
Fix HTML title parsing bugs.
...
This slightly modifies the HTML_TITLE_REGEX to fix two parsing errors.
The first occurred when title tags were empty (e.g. "<title></title>")
which was parsed as "</title". The second occurred when titles were a
single character (e.g. "<title>A</title>") which was not matched by the
regex, and so would fall back to link.base_url.
Now when tags are empty, it falls back to link.base_url, and single
character titles are parsed correctly.
The way the regex works now is still a bit wonky for some edge cases.
I couldn't find any cases of incorrect behavior, but it still might be
worth reworking more completely for robustness.
2023-10-09 02:00:01 -05:00
Nick Sweeting
5c1a14e4f2
ignore errors while getting system user name
2023-09-14 03:39:44 -07:00
Nick Sweeting
ffe2968e4f
improve some comments
2023-09-14 02:41:27 -07:00
Nick Sweeting
f809efce4d
Merge pull request #996 from barthalion/dev
2023-09-03 21:40:49 -07:00
Nick Sweeting
aaca74f6a8
only start parsing json after the first open brace
2023-09-03 21:40:12 -07:00
Nick Sweeting
cd9f228b2f
Merge pull request #1214 from DanielBatteryStapler/DanielBatteryStapler-patch-1
2023-09-03 21:25:12 -07:00
Nick Sweeting
16d278fbdb
Merge pull request #1168 from mAAdhaTTah/add-readwise-reader
2023-09-03 21:24:49 -07:00
Nick Sweeting
110a22ee32
Merge branch 'dev' into DanielBatteryStapler-patch-1
2023-08-31 15:20:46 -07:00
Nick Sweeting
73a5f74d38
update default YOUTUBEDL_ARGS to fix subs and filesize
2023-08-31 15:17:45 -07:00
Nick Sweeting
86366d5640
Update logging_util.py to fix generator subscripting error
2023-08-31 15:12:43 -07:00
spresse1
603ce7ec10
After a timeout, chrome will leave behind a SingletonLock, which prevents future instances of chrome from starting. When an extractor fails due to a timeout, remove this file.
2023-08-28 17:27:03 +02:00
root
23f086aa40
add LDAP support
2023-08-17 19:51:02 -05:00
DanielBatteryStapler
94dacc49c7
Fix archive_org icon "exists"
2023-08-15 23:49:54 -04:00
Ross Williams
c039ef05b3
Fix hyphen placement in util.URL_REGEX
...
Incorrect hyphen placement in `URL_REGEX` was allowing it to match more
characters than intended. In a regex character class, a literal hyphen
can only appear as the first character in the class, or it will be
interpreted as the delimiter of a range of characters.
The issue fixed here caused the range of characters from `[$-_]`
be treated as valid URL characters, instead of the intended set of three
characters `[-_$]`. The incorrect range interpretation inadvertantly
included most ASCII punctuation, most importantly the angle brackets,
square brackets, and single quote that the expression uses
to mark the end of a match.
This causes the expression to match a URL that has a "hostname" portion
beginning with one of the intended "stop parsing" characters. For
example:
```
https://<b>www</b>.example.com/ # MATCHES but should not
https://[for example] # MATCHES but should not
scheme='https://' # MATCHES, including final quote, but should not
```
Some test cases have been added to the `URL_REGEX` assert in
archivebox.parsers to cover this possibility.
2023-08-08 15:24:16 -04:00
Ross Williams
2076474252
Drop use of TypeAlias to maintain Python 3.9 compat
...
TypeAlias annotation was introduced in Python 3.10, and is not strictly
necessary. Drop use of it to maintain Python 3.9 compatibility.
2023-08-02 10:56:48 -04:00
Ross Williams
b44f7e68b1
Add URL-specific method allow/deny lists
...
Allows enabling only allow-listed extractors or disabling specific
deny-listed extractors for a regular expression matched against an added
site's URL.
2023-08-02 09:36:40 -04:00
Ross Williams
46e80dd509
Rename URL_(WHITE|BLACK)LIST to URL_(ALLOW|DENY)LIST
...
Retain aliases for old configuration files
2023-08-02 09:31:48 -04:00
Nick Sweeting
b773041952
Merge pull request #1199 from overhacked/chrome_version_detection_fix
2023-08-01 10:14:18 -07:00
Ross Williams
d0e65eba7f
More reliably detect Google Chrome version number
...
Previous method was splitting on the first whitespace, and missing the
version number when it appeared as `"Google Chrome 115.0.234.2342"`
instead of, i.e. `"Chromium 115.0.234.8283"`.
This commit changes the version detection to regex search for
whitespace, then one or more digits followed by a period, then at least
one more digit. Only the first sequence of digits is captured. Unless
Chrome radically changes their version numbering, this should capture
the first group of digits after the reported browser name, which would
be the major version.
2023-07-31 15:34:58 -04:00
Ross Williams
9d9872d325
bin_version means to modify, not replace environ
...
the `bin_version` function means to modify the environment,
not replace it entirely. Fixes bugs that occur when it wipes out the
PATH environment variable, such as when running in a virtual
environment.
2023-07-31 11:36:34 -04:00
mAAdhaTTah
181501fd36
Add Readwise Reader API parser
...
Implemented similar to the Pocket API.
2023-07-02 11:20:58 -04:00
Sascha Ißbrücker
7bf4f40da0
just use out_dir
2023-05-29 10:03:49 +02:00
Sascha Ißbrücker
40c122515a
fix: make oneshot command return successful exist code
2023-05-29 10:01:27 +02:00
Micah R Ledbetter
1e50ca243e
Add FAVICON_PROVIDER option for custom favicon service
2023-05-05 20:42:36 -05:00
David Calano
f48e48e6da
Fix for Issue #1008
...
- Added missing decode() when setting pkg_path variable
2023-03-29 01:48:12 -04:00
Tom Ryder
53af810ff8
Add missing closing quote to style attribute
2023-03-27 10:54:04 +13:00
ふぁ
44a5a5ed7e
add explicitly specify --headless=new
...
Signed-off-by: ふぁ <yuki@yuki0311.com>
2023-03-17 19:30:14 +09:00
Nick Sweeting
9f42a3bf29
fix whitespace
2023-03-15 16:01:02 -07:00
ふぁ
d77c770c47
add CHROME_TIMEOUT args
...
Signed-off-by: ふぁ <yuki@yuki0311.com>
2023-03-14 20:29:41 +09:00
Nick Sweeting
606fa397a4
disable passing timeout arg to chrome because v111 is crashing when passed
2023-03-13 10:50:18 +00:00
Nick Sweeting
1f1c70a8b1
remove --single-process from chrome args and add some rendering optimization args
2023-03-13 10:49:57 +00:00
Nick Sweeting
9599845b56
ensure DOM HTML dump is non-zero length file when retrying
2023-03-13 10:49:26 +00:00
Nick Sweeting
dca69933eb
Update archivebox/config.py
...
Co-authored-by: dugite-code <dugite-code@users.noreply.github.com>
2023-01-09 18:22:01 -08:00
Nick Sweeting
2538b170c7
Merge branch 'dev' into feat/reverse-proxy-auth
2023-01-09 18:20:45 -08:00
Nick Sweeting
0cbeeb4346
Merge pull request #1021 from renaisun/dev
2023-01-09 18:17:39 -08:00
Joseph Turian
07de4a79a1
Merge branch 'dev' into feature/kludge-984-UTF8-bug
2022-12-20 11:39:01 +01:00
Nick Sweeting
e114b1f6dc
Merge pull request #1027 from turian/feature/migrations-0021_auto_20220914_0934.py
2022-11-27 19:28:55 -08:00
SnZ
2db830c6a8
Method typo?
...
Fixes '[Errno 2] No such file or directory' error during add
2022-11-20 01:51:16 +01:00
Joseph Turian
a26a91d09f
Merge branch 'feature/migrations-0021_auto_20220914_0934.py' into feature/kludge-984-UTF8-bug
2022-09-14 09:44:55 +00:00
Joseph Turian
22d8e57637
Add missing migration 0021
2022-09-14 09:36:17 +00:00
Joseph Turian
30947aeb07
yt-dlp flag cleanup
2022-09-14 06:29:57 +02:00
Joseph Turian
f729bbe122
yt-dlp fixes
2022-09-14 06:27:58 +02:00
Joseph Turian
081a12b079
Add ts
2022-09-12 21:32:47 +00:00
Joseph Turian
daef48e59b
flake8
2022-09-12 21:31:33 +00:00
Joseph Turian
983f485cc0
flake8
2022-09-12 21:29:43 +00:00
Joseph Turian
b864c38d9e
Don't be strict on unicode errors
2022-09-12 20:40:45 +00:00
Joseph Turian
dba423a568
A few more youtube-dl tweaks
2022-09-12 20:36:23 +00:00
Joseph Turian
f5f7aff3b4
Added yt-dlp everywhere
2022-09-12 20:34:02 +00:00
renaisun
0ea955b3ed
add a missing comma
2022-09-12 09:08:28 +08:00
notevenaperson
40659b5e9d
singlefile.py: Code to ensure options are deduplicated
2022-09-12 09:08:28 +08:00
Joseph Turian
2b58cce43f
Attempted to warn on #984 and #1014
2022-09-11 12:19:16 +02:00
Bartłomiej Piotrowski
eb97fd427b
Skip first line of the "JSON" file
...
ArchiveBox moves the file to parse to the sources directory and adds the
original filename at the top, making the file invalid.
2022-07-05 10:56:40 +02:00
Nick Sweeting
03eb7e5875
Update config.py
2022-06-09 01:04:55 -07:00
renaisun
8899fe0b92
Add SINGLEFILE_ARGS to control single-file arguments
2022-06-09 14:35:48 +08:00
Nick Sweeting
d586a8babc
show mount paths with at symbol in version output
2022-06-08 20:22:58 -07:00
Nick Sweeting
319ea481b8
Update config.py
2022-06-08 20:17:38 -07:00
Nick Sweeting
01555dfe34
Update main.py
2022-06-08 20:17:31 -07:00
Nick Sweeting
2bbc742017
typo fix
2022-06-08 20:16:08 -07:00
Nick Sweeting
e2fa68dba6
resolve config paths before using
2022-06-08 20:15:22 -07:00
Nick Sweeting
ccce4a6a2f
use new is_mount and COMMIT_HASH config options
2022-06-08 20:13:22 -07:00
Nick Sweeting
9f90a2d60d
disable unused sqlite3 stuff
2022-06-08 20:12:55 -07:00
Nick Sweeting
c78a2edc42
add is_mount and COMMIT_HASH to config.py
2022-06-08 20:04:01 -07:00
Nick Sweeting
375ba9d135
Update settings.py
2022-06-08 20:00:29 -07:00
Nick Sweeting
ae5c8f2bf8
fix newline included in commit hash
2022-06-08 19:57:38 -07:00
Nick Sweeting
cb3ebbe69a
fix git commit hash location
2022-06-08 19:52:48 -07:00
Nick Sweeting
413aa2ef04
fix commit hash detection
2022-06-08 19:51:46 -07:00
Nick Sweeting
33ec2117e9
Update main.py
2022-06-08 19:50:45 -07:00
Nick Sweeting
dd29e1bf78
clean up first line of CLI version output for easier downstream parsing
2022-06-08 19:46:09 -07:00
Nick Sweeting
0c6d4c82c3
Update config.py
2022-06-08 19:11:02 -07:00
Nick Sweeting
f9c5808940
Update config.py
2022-06-08 19:09:11 -07:00
Nick Sweeting
5509b5cd8b
Update main.py
2022-06-08 19:08:33 -07:00
Nick Sweeting
19b88d30b2
fix missing brace
2022-06-08 19:06:03 -07:00
Nick Sweeting
31d5fbbf17
Update config.py
2022-06-08 19:04:06 -07:00
Nick Sweeting
6b019da3e9
Update config.py
2022-06-08 19:01:55 -07:00
Nick Sweeting
c752c7053d
Update main.py
2022-06-08 18:59:08 -07:00
Nick Sweeting
f9c82841ad
fix sqlite option detection
2022-06-08 18:58:15 -07:00
Nick Sweeting
1fd5830f58
enforce UTC timezone on server
2022-06-08 18:41:22 -07:00
Nick Sweeting
3e3c011f86
enforce UTC timezone on server
2022-06-08 18:40:48 -07:00
Nick Sweeting
e06717419c
fix sqlite3 version detection
2022-06-08 18:35:31 -07:00
Nick Sweeting
d0f129295f
move sqlite3 checks up a level
2022-06-08 18:29:53 -07:00
Nick Sweeting
0c7d7deb32
add missing brace
2022-06-08 18:26:42 -07:00
Nick Sweeting
ca16c88a3d
show PUID, PGID, ENFORCE_ATOMIC_WRITES, and OUTPUT_PERMISSIONS in version output header
2022-06-08 18:24:58 -07:00
Nick Sweeting
89175ccb22
check SQLite3 version and enabled extensions on startup
2022-06-08 18:24:17 -07:00
Nick Sweeting
c245d36e44
add PUID and PGID as config options in archivebox
2022-06-08 17:42:52 -07:00
Nick Sweeting
c5fc3e1e65
--ammend
2022-05-09 23:59:27 -07:00
Nick Sweeting
0b4df768ba
hack to check for generator type cause too lazy to import
2022-05-09 23:50:56 -07:00
Nick Sweeting
5e4ddbbf25
fix mercury bin parsing back
2022-05-09 21:58:17 -07:00
Nick Sweeting
e96c1bcf13
bump mercury parser to git head version
2022-05-09 21:48:41 -07:00
Nick Sweeting
d581a5081f
correctly handle bytes strings in hints
2022-05-09 21:29:37 -07:00
Nick Sweeting
a6767671fb
append content of referenced files to imports
2022-05-09 21:21:39 -07:00
Nick Sweeting
f6d6a06c78
always show all totals in log output
2022-05-09 21:21:26 -07:00
Nick Sweeting
d05510f844
fix version string parsing on macOS in some cases where LANG C is not supported
2022-05-09 21:21:08 -07:00
Nick Sweeting
4b8b17e788
add update flag support to archivebox schedule
2022-05-09 20:18:43 -07:00
Nick Sweeting
8cfe6f4afb
cleanup update flag handling and show better logging to clarify when its working
2022-05-09 20:15:55 -07:00
Nick Sweeting
38e54b93fe
allow parsing to continue even when fetching URL contents fails
2022-05-09 19:56:24 -07:00
Nick Sweeting
ecbcb6a1b3
fix bracing in template tag for PREVIEW_ORIGINALS
2022-05-09 19:56:08 -07:00
Nick Sweeting
8ebf3e2f93
add config option PREVIEW_ORIGINALS to hide original iframes in snapshot detail pages
2022-05-09 19:31:41 -07:00
Nick Sweeting
acd53c854d
handle new wallabag export format with newlines mid-tag attributes
2022-05-09 19:07:48 -07:00
Nick Sweeting
950b5cbbb6
Merge pull request #924 from prnake/dev
...
improve title extractor
2022-05-09 18:38:12 -07:00
Nick Sweeting
6e66863871
add max 5s writing delay for concurrent writers and flush WAL slower
2022-05-09 18:36:40 -07:00
Nick Sweeting
57df65f28f
use yt-dlp for media archiving instead of youtube-dl
2022-04-21 07:11:35 -07:00
Nick Sweeting
eb81d41f84
bump Dockerfile base image version and install yt-dlp
2022-04-21 07:11:35 -07:00
Ross
c63822a5e5
Fix missing input redirection in a hint text
2022-04-19 22:25:49 +01:00
Igor Rzegocki
d4f534e612
add LOGOUT_REDIRECT_URL
2022-03-31 21:40:14 +02:00
Pellaeon Lin
5e9d05483e
Fix bin_version: set LANG=C when calling executables to avoid parsing
...
localized output.
2022-02-24 17:01:00 +08:00
prnake
011bd104cb
remove unused import
2022-02-09 10:48:51 +08:00
papersnake
de8e22efb7
improve title extractor
2022-02-08 23:17:52 +08:00
Nick Sweeting
666ab20df5
Update archivebox/config.py
2022-01-10 20:42:09 -05:00
hannah98
fc3d2bb4dc
rename TAG_SEPARATORS to TAG_SEPARATOR_PATTERN
2022-01-06 14:14:41 +00:00
hannah98
049f88def9
Added TAG_SEPARATORS option to supply a regex of characters to use when splitting tags
2021-12-30 20:19:48 +00:00
Nick Sweeting
d7f01922f3
fix direct assignment of tags to many-to-many set
2021-12-23 12:29:17 -05:00
Nick Sweeting
b1b7ee2b85
Update sql.py
2021-12-23 12:17:55 -05:00
hannah98
4b8962b60b
Fix #725 - correctly parse tags on json import
2021-12-20 08:58:58 -06:00
Mika Tuupola
f14a861605
Change logfile open to write mode only
2021-12-19 23:17:33 +02:00
TheCakeIsNaOH
decab91ea2
( #847 ) Decode error output hints to string if needed
2021-12-16 16:46:12 -06:00
Nick Sweeting
44f5338470
fix typo in pocket_api articl variable name
2021-11-12 19:23:47 -05:00
Nick Sweeting
8878dcc5e8
Merge pull request #843 from bltavares/patch-1
2021-11-12 15:57:19 -08:00
Nick Sweeting
d5240f1a1d
Merge pull request #885 from adamwolf/safari-admin-actions
2021-11-12 08:56:31 -08:00
Adam Wolf
18e1fb0d96
Fixes Add button behavior on Safari
...
Previously, when you clicked the Add button, the page wouldn't change.
It looked like it wasn't doing anything, as noted by @rcarmo
(https://github.com/ArchiveBox/ArchiveBox/issues/658#issuecomment-948300055 )
I didn't track it down the exact reason why. It may be that Safari
didn't like the two opening <h3>s, but I was able to find a bunch of
people complaining about Safari being very finicky with innerHTML
and actually repainting the page, enough that I decided to try just
extending the block hide/show behavior already done with the
delay-warning, and it works for me now in both Chrome and Safari.
For #658 .
2021-10-28 22:31:54 -05:00