CodeMirror/ArchiveBox
1
0
Fork 0
mirror of synced 2024-06-29 03:20:58 +12:00
Code Issues Projects Releases Wiki Activity
3190 commits 5 branches 31 tags 9.4 MiB
Commit graph

1290 commits

Author SHA1 Message Date
Nick Sweeting 31d5fbbf17
Update config.py 2022-06-08 19:04:06 -07:00
Nick Sweeting 6b019da3e9
Update config.py 2022-06-08 19:01:55 -07:00
Nick Sweeting c752c7053d
Update main.py 2022-06-08 18:59:08 -07:00
Nick Sweeting f9c82841ad
fix sqlite option detection 2022-06-08 18:58:15 -07:00
Nick Sweeting 1fd5830f58
enforce UTC timezone on server 2022-06-08 18:41:22 -07:00
Nick Sweeting 3e3c011f86
enforce UTC timezone on server 2022-06-08 18:40:48 -07:00
Nick Sweeting e06717419c
fix sqlite3 version detection 2022-06-08 18:35:31 -07:00
Nick Sweeting d0f129295f
move sqlite3 checks up a level 2022-06-08 18:29:53 -07:00
Nick Sweeting 0c7d7deb32
add missing brace 2022-06-08 18:26:42 -07:00
Nick Sweeting ca16c88a3d
show PUID, PGID, ENFORCE_ATOMIC_WRITES, and OUTPUT_PERMISSIONS in version output header 2022-06-08 18:24:58 -07:00
Nick Sweeting 89175ccb22
check SQLite3 version and enabled extensions on startup 2022-06-08 18:24:17 -07:00
Nick Sweeting c245d36e44
add PUID and PGID as config options in archivebox 2022-06-08 17:42:52 -07:00
Nick Sweeting c5fc3e1e65 --ammend 2022-05-09 23:59:27 -07:00
Nick Sweeting 0b4df768ba hack to check for generator type cause too lazy to import 2022-05-09 23:50:56 -07:00
Nick Sweeting 5e4ddbbf25 fix mercury bin parsing back 2022-05-09 21:58:17 -07:00
Nick Sweeting e96c1bcf13 bump mercury parser to git head version 2022-05-09 21:48:41 -07:00
Nick Sweeting d581a5081f correctly handle bytes strings in hints 2022-05-09 21:29:37 -07:00
Nick Sweeting a6767671fb append content of referenced files to imports 2022-05-09 21:21:39 -07:00
Nick Sweeting f6d6a06c78 always show all totals in log output 2022-05-09 21:21:26 -07:00
Nick Sweeting d05510f844 fix version string parsing on macOS in some cases where LANG C is not supported 2022-05-09 21:21:08 -07:00
Nick Sweeting 4b8b17e788 add update flag support to archivebox schedule 2022-05-09 20:18:43 -07:00
Nick Sweeting 8cfe6f4afb cleanup update flag handling and show better logging to clarify when its working 2022-05-09 20:15:55 -07:00
Nick Sweeting 38e54b93fe allow parsing to continue even when fetching URL contents fails 2022-05-09 19:56:24 -07:00
Nick Sweeting ecbcb6a1b3 fix bracing in template tag for PREVIEW_ORIGINALS 2022-05-09 19:56:08 -07:00
Nick Sweeting 8ebf3e2f93 add config option PREVIEW_ORIGINALS to hide original iframes in snapshot detail pages 2022-05-09 19:31:41 -07:00
Nick Sweeting acd53c854d handle new wallabag export format with newlines mid-tag attributes 2022-05-09 19:07:48 -07:00
Nick Sweeting 950b5cbbb6
Merge pull request #924 from prnake/dev 
improve title extractor
 2022-05-09 18:38:12 -07:00
Nick Sweeting 6e66863871 add max 5s writing delay for concurrent writers and flush WAL slower 2022-05-09 18:36:40 -07:00
Nick Sweeting 57df65f28f use yt-dlp for media archiving instead of youtube-dl 2022-04-21 07:11:35 -07:00
Nick Sweeting eb81d41f84 bump Dockerfile base image version and install yt-dlp 2022-04-21 07:11:35 -07:00
Ross c63822a5e5
Fix missing input redirection in a hint text 2022-04-19 22:25:49 +01:00
Igor Rzegocki d4f534e612
add LOGOUT_REDIRECT_URL 2022-03-31 21:40:14 +02:00
Pellaeon Lin 5e9d05483e Fix bin_version: set LANG=C when calling executables to avoid parsing 
localized output.
 2022-02-24 17:01:00 +08:00
prnake 011bd104cb
remove unused import 2022-02-09 10:48:51 +08:00
papersnake de8e22efb7 improve title extractor 2022-02-08 23:17:52 +08:00
Nick Sweeting 666ab20df5
Update archivebox/config.py 2022-01-10 20:42:09 -05:00
hannah98 fc3d2bb4dc rename TAG_SEPARATORS to TAG_SEPARATOR_PATTERN 2022-01-06 14:14:41 +00:00
hannah98 049f88def9 Added TAG_SEPARATORS option to supply a regex of characters to use when splitting tags 2021-12-30 20:19:48 +00:00
Nick Sweeting d7f01922f3
fix direct assignment of tags to many-to-many set 2021-12-23 12:29:17 -05:00
Nick Sweeting b1b7ee2b85
Update sql.py 2021-12-23 12:17:55 -05:00
hannah98 4b8962b60b Fix #725 - correctly parse tags on json import 2021-12-20 08:58:58 -06:00
Mika Tuupola f14a861605 Change logfile open to write mode only 2021-12-19 23:17:33 +02:00
TheCakeIsNaOH decab91ea2
(#847) Decode error output hints to string if needed 2021-12-16 16:46:12 -06:00
Nick Sweeting 44f5338470
fix typo in pocket_api articl variable name 2021-11-12 19:23:47 -05:00
Nick Sweeting 8878dcc5e8
Merge pull request #843 from bltavares/patch-1 2021-11-12 15:57:19 -08:00
Nick Sweeting d5240f1a1d
Merge pull request #885 from adamwolf/safari-admin-actions 2021-11-12 08:56:31 -08:00
Adam Wolf 18e1fb0d96 Fixes Add button behavior on Safari 
Previously, when you clicked the Add button, the page wouldn't change.

It looked like it wasn't doing anything, as noted by @rcarmo
(https://github.com/ArchiveBox/ArchiveBox/issues/658#issuecomment-948300055)

I didn't track it down the exact reason why. It may be that Safari
didn't like the two opening <h3>s, but I was able to find a bunch of
people complaining about Safari being very finicky with innerHTML
and actually repainting the page, enough that I decided to try just
extending the block hide/show behavior already done with the
delay-warning, and it works for me now in both Chrome and Safari.

For #658.
 2021-10-28 22:31:54 -05:00
Adam Wolf 83731f5a68 Tweak JS so Safari can choose admin actions 
I noticed that Safari was submitting both the empty option and the
selected options back to the server.

Digging into it, I was able to get Safari to deselect the ---------
option by using '[selected]' as the selector.

For #658
 2021-10-28 22:22:46 -05:00
Igor Rzegocki 05de1c9fe6
healthcheck endpoint 2021-10-03 19:12:03 +02:00
Igor Rzegocki 95cf85f8cf
Support for Reverse Proxy authentication backends (like authelia) 2021-09-30 17:40:13 +02:00
Bruno Tavares bb2a2e758a
Avoid KeyError on Pocket API parser 
When trying to import my pocket library I got a lot of ` KeyError`  on Python. Pocket API has a few idiosyncrasies, such as sometimes returning the keys on json, sometimes not.

` ` ` sh
archivebox add --parser pocket_api pocket://my_username
` ` ` 

Gave me this errors
` ` ` 
  File "/app/archivebox/parsers/pocket_api.py", line 54, in link_from_article
    title = article['resolved_title'] or article['given_title'] or url
KeyError: 'resolved_title'
` ` ` 

This commit are the patches I've changed to successfully import my library
 2021-09-07 21:53:36 -03:00
Ross Williams f6cf35a45d Fix Pinboard RSS parsing valid links as None 
`item.find(p)` returns either an `ElementTree.Element` or `None`.  The
[lambda on line 24][lambda] coerces the return value to a bool, which is
`False` if the `<link>` element has no children (see
[`ElementTree.py` line 207][etbooldef]), so the lambda returns `None`.

Further, returning a `Link` with `url=None` violates
[an assertion in `index/schema.py`][assertion], which crashes
the `archivebox add` command.

[lambda]: 3d54b1321b/archivebox/parsers/pinboard_rss.py (L24)
[etbooldef]: 3d8993a744/Lib/xml/etree/ElementTree.py (L207)
[assertion]: 3d54b1321b/archivebox/index/schema.py (L165)
 2021-08-04 10:13:37 -04:00
Inndy 0e81a0722e Discard Referer header from iframe and link to original URL 2021-07-19 21:48:01 +08:00
Nick Sweeting 5a2c78e14b add proper support for URL_WHITELIST instead of using negation regexes 2021-07-06 23:42:00 -04:00
Nick Sweeting e4974d3536 support negation patterns by checking both re.search and re.match 2021-07-06 23:17:05 -04:00
TJ Horner cdcfb7fa44 Exempt /add route from CSRF 2021-07-01 20:55:51 -04:00
Nick Sweeting e0a2b2e252 ominous warnings 2021-06-01 03:03:42 -04:00
Nick Sweeting aa53fe653c fix use of uneeded perms arg 2021-06-01 02:58:36 -04:00
Nick Sweeting c2d1a57581 fix umask dir permissions 2021-06-01 00:50:18 -04:00
Nick Sweeting 4715ace7dd ignore BaseException lgtm errors 2021-05-31 20:59:05 -04:00
Nick Sweeting 9f1470cf03 fix output permissions tests 2021-05-31 20:57:46 -04:00
Nick Sweeting 8230f88d80 change default OUTPUT_PERMISSIONS to disallow execution except on dirs 2021-05-31 19:31:51 -04:00
Nick Sweeting 1112526543 add option ENFORCE_ATOMIC_WRITES to allow disabling forced FSYNC writes on network drives 2021-05-31 19:31:51 -04:00
Nick Sweeting 49faec8f6d
add no-zygote and single-process args to try and prevent orphan chrome processes after exit 2021-05-13 05:04:23 -04:00
Nick Sweeting eb4d3bca9d
Update readability.py 2021-05-13 00:13:32 -04:00
Nick Sweeting c3d009e4ec fix python file execution checking 2021-04-24 04:43:02 -04:00
Nick Sweeting 79e19ecd47 shield pwd import for windows 2021-04-24 03:51:38 -04:00
Nick Sweeting 3db77fd1a2 fix lint errors 2021-04-24 03:49:01 -04:00
Nick Sweeting 3992e0fee3 auto prepend python binary to args when running system command with python file as first argument 2021-04-24 03:29:22 -04:00
Nick Sweeting fa84f749ff run archivebox version using python binary 2021-04-24 03:29:22 -04:00
Nick Sweeting 226e26852c dont try to autodetect whether node, chrome, etc are needed during setup 2021-04-24 03:29:22 -04:00
Nick Sweeting f129b9f443 allow executing main 2021-04-24 03:11:06 -04:00
Nick Sweeting 208f866fc4 fix user detection on freebsd always returning root 2021-04-24 02:52:09 -04:00
Nick Sweeting 06f6084e3b ignore stdin when passed instead of throwing an error 2021-04-24 00:09:52 -04:00
Nick Sweeting eb80dc26a1 cleanup config files 2021-04-23 22:58:44 -04:00
Nick Sweeting 4f91af0ce9 TODO create admin user from env vars 2021-04-23 15:50:34 -04:00
Nick Sweeting 9563e74f7c enforce minimum django version 2021-04-23 15:02:26 -04:00
Kunwoo Ahn b6ee7d291e
fix bookmarklet script 
Fix '&' bug in parameter
 2021-04-16 21:24:51 +09:00
Nick Sweeting 63552be0e3 whoops 2021-04-13 04:03:47 -04:00
Nick Sweeting 7d1f8cb496 no longer recommend snap chromium, just use playwright 2021-04-13 03:59:17 -04:00
Nick Sweeting 2c6f0a96bf fix extra arg 2021-04-13 02:21:51 -04:00
Nick Sweeting 54cab113f3 allow static_index.json in dir root from old static exports 2021-04-12 17:06:32 -04:00
Nick Sweeting 50b341baab bail out if old index.json is found during init but doesnt contain links 2021-04-12 16:51:45 -04:00
Nick Sweeting 217952c8d0 sigh linters 2021-04-11 12:51:50 -04:00
Nick Sweeting bfefe042bf fix noqa 2021-04-11 03:57:04 -04:00
Nick Sweeting 335e4658c8 keep grid header on one line 2021-04-11 00:49:33 -04:00
Nick Sweeting 52429b8d15 dont prevent search when backend throws an exception or times out 2021-04-10 08:18:13 -04:00
Nick Sweeting 9f05cf8283 virtual-time-budget doesnt work with some chrome stuff 2021-04-10 08:04:59 -04:00
Nick Sweeting 62078a77f8 show run duration after each archived link in cli output 2021-04-10 07:52:01 -04:00
Nick Sweeting 6a6ef63775 enforce utf8 on more file reads 2021-04-10 07:22:32 -04:00
Nick Sweeting 193df5c8d3 add video subtitles and description to full-text index 2021-04-10 07:22:20 -04:00
Nick Sweeting 1e3c1e3c83 also write autogenerated subtitles 2021-04-10 06:56:19 -04:00
Nick Sweeting e7473c4e3e add some more padding to public index columns 2021-04-10 06:38:33 -04:00
Nick Sweeting 6949803395 enforce new models to use uuid keys 2021-04-10 06:32:45 -04:00
Nick Sweeting 1b67dd5c45 fix issues with django 3.2 2021-04-10 06:22:07 -04:00
Nick Sweeting 0c321a06d0 hide scrollbars in screenshots 2021-04-10 05:45:19 -04:00
Nick Sweeting c5a9b77f4f update user agents to mimic latest chrome 2021-04-10 05:45:02 -04:00
Nick Sweeting bf0da664be fix favicons on public index 2021-04-10 05:23:02 -04:00
Nick Sweeting 3093057675 bail out on sonic indexing after 5 errors 2021-04-10 05:18:03 -04:00
Nick Sweeting 89158d5dc7 allow full-text search from public index 2021-04-10 05:14:02 -04:00
Nick Sweeting d1051e40bd remove tzlocal 2021-04-10 04:42:39 -04:00
Nick Sweeting a9986f1f05 add timezone support, tons of CSS and layout improvements, more detailed snapshot admin form info, ability to sort by recently updated, better grid view styling, better table layouts, better dark mode support 2021-04-10 04:21:36 -04:00
Nick Sweeting cf7d7e4990 add new timezone autosetting and cache header setting middlewares 2021-04-10 04:21:36 -04:00
Nick Sweeting 1977ae8962 add more helper props to snapshot for getting latest headers, favicon, etc 2021-04-10 04:21:36 -04:00
Nick Sweeting 8d68f1744e tweak columns in private snapshots list and shorten action names 2021-04-10 04:21:36 -04:00
Nick Sweeting 722f530c80 support adding and removing multiple tags at once using autocomplete multiselect field 2021-04-10 04:21:36 -04:00
Nick Sweeting 06c58ac42a fix unreachable config backup clear 2021-04-09 12:15:47 -04:00
Nick Sweeting 32764347ce add new SEARCH_BACKEND_TIMEOUT config option defaulted to 90sec 2021-04-09 00:27:12 -04:00
Nick Sweeting 2e57df917e handle BaseExceptions properly 2021-04-08 06:08:17 -04:00
Nick Sweeting 2c07d8f1f3 fix lgtm alerts 2021-04-06 13:47:26 -04:00
Nick Sweeting 21bb84f627 fix lint errors 2021-04-06 02:50:51 -04:00
Nick Sweeting e92db03f5f fallback to using bin_path to find archivebox binary if sys.argv is empty 2021-04-06 02:45:57 -04:00
Nick Sweeting c98053a8ca allow filtering by archiveresult status 2021-04-06 02:07:22 -04:00
Nick Sweeting f67a5a215a fix readability indexing process and implement a max total character length on indexed content 2021-04-06 02:01:38 -04:00
Nick Sweeting b3a89172ab add original src url to dosctring 2021-04-06 01:38:24 -04:00
Nick Sweeting e7c7a8f71c force kill forked child processes after each extractor command 2021-04-06 01:35:21 -04:00
Nick Sweeting af6561b527 remove CDN jquery and bootstrap by inlining them instead 2021-04-06 01:13:31 -04:00
Nick Sweeting e4823cc5d7 add back CUSTOM_TEMPLATES_DIR config option 2021-04-06 00:57:20 -04:00
Nick Sweeting 4b098bea15 fix navbar showing user info even when not logged in 2021-04-06 00:44:01 -04:00
Nick Sweeting 68a506e1a7 fix Snapshot admin actions buttons submitting on Safari 2021-04-06 00:43:49 -04:00
Nick Sweeting 5a9f27204a dont use chrome when its not available on windows systems 2021-04-05 23:33:08 -04:00
Nick Sweeting 1485e4eb90 use older docker compose version for more features 2021-04-05 23:22:05 -04:00
Nick Sweeting cadac4838e fix searching by url contents 2021-04-05 23:21:07 -04:00
Nick Sweeting 2fc9f1cfa3 dont depend on README in Dockerfile install step 2021-04-05 23:17:07 -04:00
Nick Sweeting 8b4f84959a new setup flag on init command to autosetup on init 2021-04-05 21:15:32 -04:00
Nick Sweeting 5c181532b5 only install dependencies in setup if they are missing 2021-04-05 21:15:10 -04:00
Nick Sweeting 21820acecd import User model into core models for convenience 2021-04-05 21:14:22 -04:00
Nick Sweeting bbebc33b7a better autodetection of node modules dependencies 2021-04-05 21:14:06 -04:00
Nick Sweeting 211bf86f4a fix bug checking value in subcommand_args when None 2021-04-05 21:13:49 -04:00
Nick Sweeting 79fb60a47a dont allow caching when installing packages 2021-04-05 20:08:48 -04:00
Nick Sweeting 05c17af4dc fix lint errors 2021-04-05 20:00:16 -04:00
Nick Sweeting ef76f7c490 require running setup in existing data dir 2021-04-05 19:59:33 -04:00
Nick Sweeting 8c1d12020f fix lint errors 2021-04-05 19:53:42 -04:00
Nick Sweeting 3449bac5bc add new setup command to autoinstall dependencies 2021-04-05 19:51:57 -04:00
Nick Sweeting 24f7dd9c05 dont raise systemexit on succesful config 2021-04-05 19:51:40 -04:00
Nick Sweeting ffd119c66c fix linter 2021-04-03 15:51:27 -04:00
Nick Sweeting 7162649b03 fix search fields on admin snapshots list 2021-04-01 14:00:07 -04:00
Nick Sweeting 5492c204b8 tweak add page copy 2021-04-01 14:00:07 -04:00
Nick Sweeting f59b6d4189 only add url-list lines that are real urls 2021-04-01 14:00:07 -04:00
Nick Sweeting d73f7d7d96 add db_index on url field 2021-04-01 14:00:07 -04:00
Nick Sweeting c438e49871 fix clicking on tags on snapshot list 2021-04-01 14:00:07 -04:00
Nick Sweeting 0e39a2098d split migrations and data dir check 2021-04-01 14:00:07 -04:00
Nick Sweeting 5d7ad9b1b4 add form parser option 2021-04-01 02:34:16 -04:00
Nick Sweeting 59d5423483 fix snapshot icon caching and ordering 2021-04-01 02:22:15 -04:00
Nick Sweeting fc5d99b4b4 fix missing package.json repo 2021-04-01 01:40:02 -04:00
Nick Sweeting ff9445648b fix navbar and tags display 2021-04-01 01:39:34 -04:00
Nick Sweeting 7ee7a94e67 fix str split using old overwrite pipe instead of append pipe 2021-03-31 11:30:15 -04:00
Nick Sweeting 005da31f8f add overwrite flag option to schedule cmd 2021-03-31 11:29:51 -04:00
Nick Sweeting 534ae60ab0 dont overwrite the log on each run, and use dedicated log for scheduled archiving 2021-03-31 11:20:42 -04:00
Nick Sweeting 5d3a03b299 use stderr and hint in case of parser returning no urls instead of bare exception 2021-03-31 01:39:01 -04:00