Bruno Tavares
bb2a2e758a
Avoid KeyError on Pocket API parser
...
When trying to import my pocket library I got a lot of ` KeyError` on Python. Pocket API has a few idiosyncrasies, such as sometimes returning the keys on json, sometimes not.
` ` ` sh
archivebox add --parser pocket_api pocket://my_username
` ` `
Gave me this errors
` ` `
File "/app/archivebox/parsers/pocket_api.py", line 54, in link_from_article
title = article['resolved_title'] or article['given_title'] or url
KeyError: 'resolved_title'
` ` `
This commit are the patches I've changed to successfully import my library
2021-09-07 21:53:36 -03:00
Ross Williams
f6cf35a45d
Fix Pinboard RSS parsing valid links as None
...
`item.find(p)` returns either an `ElementTree.Element` or `None`. The
[lambda on line 24][lambda] coerces the return value to a bool, which is
`False` if the `<link>` element has no children (see
[`ElementTree.py` line 207][etbooldef]), so the lambda returns `None`.
Further, returning a `Link` with `url=None` violates
[an assertion in `index/schema.py`][assertion], which crashes
the `archivebox add` command.
[lambda]: 3d54b1321b/archivebox/parsers/pinboard_rss.py (L24)3d8993a744/Lib/xml/etree/ElementTree.py (L207)3d54b1321b/archivebox/index/schema.py (L165)
2021-08-04 10:13:37 -04:00
Inndy
0e81a0722e
Discard Referer header from iframe and link to original URL
2021-07-19 21:48:01 +08:00
Nick Sweeting
5a2c78e14b
add proper support for URL_WHITELIST instead of using negation regexes
2021-07-06 23:42:00 -04:00
Nick Sweeting
e4974d3536
support negation patterns by checking both re.search and re.match
2021-07-06 23:17:05 -04:00
TJ Horner
cdcfb7fa44
Exempt /add route from CSRF
2021-07-01 20:55:51 -04:00
Nick Sweeting
e0a2b2e252
ominous warnings
2021-06-01 03:03:42 -04:00
Nick Sweeting
aa53fe653c
fix use of uneeded perms arg
2021-06-01 02:58:36 -04:00
Nick Sweeting
c2d1a57581
fix umask dir permissions
2021-06-01 00:50:18 -04:00
Nick Sweeting
4715ace7dd
ignore BaseException lgtm errors
2021-05-31 20:59:05 -04:00
Nick Sweeting
9f1470cf03
fix output permissions tests
2021-05-31 20:57:46 -04:00
Nick Sweeting
8230f88d80
change default OUTPUT_PERMISSIONS to disallow execution except on dirs
2021-05-31 19:31:51 -04:00
Nick Sweeting
1112526543
add option ENFORCE_ATOMIC_WRITES to allow disabling forced FSYNC writes on network drives
2021-05-31 19:31:51 -04:00
Nick Sweeting
49faec8f6d
add no-zygote and single-process args to try and prevent orphan chrome processes after exit
2021-05-13 05:04:23 -04:00
Nick Sweeting
eb4d3bca9d
Update readability.py
2021-05-13 00:13:32 -04:00
Nick Sweeting
c3d009e4ec
fix python file execution checking
2021-04-24 04:43:02 -04:00
Nick Sweeting
79e19ecd47
shield pwd import for windows
2021-04-24 03:51:38 -04:00
Nick Sweeting
3db77fd1a2
fix lint errors
2021-04-24 03:49:01 -04:00
Nick Sweeting
3992e0fee3
auto prepend python binary to args when running system command with python file as first argument
2021-04-24 03:29:22 -04:00
Nick Sweeting
fa84f749ff
run archivebox version using python binary
2021-04-24 03:29:22 -04:00
Nick Sweeting
226e26852c
dont try to autodetect whether node, chrome, etc are needed during setup
2021-04-24 03:29:22 -04:00
Nick Sweeting
f129b9f443
allow executing main
2021-04-24 03:11:06 -04:00
Nick Sweeting
208f866fc4
fix user detection on freebsd always returning root
2021-04-24 02:52:09 -04:00
Nick Sweeting
06f6084e3b
ignore stdin when passed instead of throwing an error
2021-04-24 00:09:52 -04:00
Nick Sweeting
eb80dc26a1
cleanup config files
2021-04-23 22:58:44 -04:00
Nick Sweeting
4f91af0ce9
TODO create admin user from env vars
2021-04-23 15:50:34 -04:00
Nick Sweeting
9563e74f7c
enforce minimum django version
2021-04-23 15:02:26 -04:00
Kunwoo Ahn
b6ee7d291e
fix bookmarklet script
...
Fix '&' bug in parameter
2021-04-16 21:24:51 +09:00
Nick Sweeting
63552be0e3
whoops
2021-04-13 04:03:47 -04:00
Nick Sweeting
7d1f8cb496
no longer recommend snap chromium, just use playwright
2021-04-13 03:59:17 -04:00
Nick Sweeting
2c6f0a96bf
fix extra arg
2021-04-13 02:21:51 -04:00
Nick Sweeting
54cab113f3
allow static_index.json in dir root from old static exports
2021-04-12 17:06:32 -04:00
Nick Sweeting
50b341baab
bail out if old index.json is found during init but doesnt contain links
2021-04-12 16:51:45 -04:00
Nick Sweeting
217952c8d0
sigh linters
2021-04-11 12:51:50 -04:00
Nick Sweeting
bfefe042bf
fix noqa
2021-04-11 03:57:04 -04:00
Nick Sweeting
335e4658c8
keep grid header on one line
2021-04-11 00:49:33 -04:00
Nick Sweeting
52429b8d15
dont prevent search when backend throws an exception or times out
2021-04-10 08:18:13 -04:00
Nick Sweeting
9f05cf8283
virtual-time-budget doesnt work with some chrome stuff
2021-04-10 08:04:59 -04:00
Nick Sweeting
62078a77f8
show run duration after each archived link in cli output
2021-04-10 07:52:01 -04:00
Nick Sweeting
6a6ef63775
enforce utf8 on more file reads
2021-04-10 07:22:32 -04:00
Nick Sweeting
193df5c8d3
add video subtitles and description to full-text index
2021-04-10 07:22:20 -04:00
Nick Sweeting
1e3c1e3c83
also write autogenerated subtitles
2021-04-10 06:56:19 -04:00
Nick Sweeting
e7473c4e3e
add some more padding to public index columns
2021-04-10 06:38:33 -04:00
Nick Sweeting
6949803395
enforce new models to use uuid keys
2021-04-10 06:32:45 -04:00
Nick Sweeting
1b67dd5c45
fix issues with django 3.2
2021-04-10 06:22:07 -04:00
Nick Sweeting
0c321a06d0
hide scrollbars in screenshots
2021-04-10 05:45:19 -04:00
Nick Sweeting
c5a9b77f4f
update user agents to mimic latest chrome
2021-04-10 05:45:02 -04:00
Nick Sweeting
bf0da664be
fix favicons on public index
2021-04-10 05:23:02 -04:00
Nick Sweeting
3093057675
bail out on sonic indexing after 5 errors
2021-04-10 05:18:03 -04:00
Nick Sweeting
89158d5dc7
allow full-text search from public index
2021-04-10 05:14:02 -04:00
Nick Sweeting
d1051e40bd
remove tzlocal
2021-04-10 04:42:39 -04:00
Nick Sweeting
a9986f1f05
add timezone support, tons of CSS and layout improvements, more detailed snapshot admin form info, ability to sort by recently updated, better grid view styling, better table layouts, better dark mode support
2021-04-10 04:21:36 -04:00
Nick Sweeting
cf7d7e4990
add new timezone autosetting and cache header setting middlewares
2021-04-10 04:21:36 -04:00
Nick Sweeting
1977ae8962
add more helper props to snapshot for getting latest headers, favicon, etc
2021-04-10 04:21:36 -04:00
Nick Sweeting
8d68f1744e
tweak columns in private snapshots list and shorten action names
2021-04-10 04:21:36 -04:00
Nick Sweeting
722f530c80
support adding and removing multiple tags at once using autocomplete multiselect field
2021-04-10 04:21:36 -04:00
Nick Sweeting
06c58ac42a
fix unreachable config backup clear
2021-04-09 12:15:47 -04:00
Nick Sweeting
32764347ce
add new SEARCH_BACKEND_TIMEOUT config option defaulted to 90sec
2021-04-09 00:27:12 -04:00
Nick Sweeting
2e57df917e
handle BaseExceptions properly
2021-04-08 06:08:17 -04:00
Nick Sweeting
2c07d8f1f3
fix lgtm alerts
2021-04-06 13:47:26 -04:00
Nick Sweeting
21bb84f627
fix lint errors
2021-04-06 02:50:51 -04:00
Nick Sweeting
e92db03f5f
fallback to using bin_path to find archivebox binary if sys.argv is empty
2021-04-06 02:45:57 -04:00
Nick Sweeting
c98053a8ca
allow filtering by archiveresult status
2021-04-06 02:07:22 -04:00
Nick Sweeting
f67a5a215a
fix readability indexing process and implement a max total character length on indexed content
2021-04-06 02:01:38 -04:00
Nick Sweeting
b3a89172ab
add original src url to dosctring
2021-04-06 01:38:24 -04:00
Nick Sweeting
e7c7a8f71c
force kill forked child processes after each extractor command
2021-04-06 01:35:21 -04:00
Nick Sweeting
af6561b527
remove CDN jquery and bootstrap by inlining them instead
2021-04-06 01:13:31 -04:00
Nick Sweeting
e4823cc5d7
add back CUSTOM_TEMPLATES_DIR config option
2021-04-06 00:57:20 -04:00
Nick Sweeting
4b098bea15
fix navbar showing user info even when not logged in
2021-04-06 00:44:01 -04:00
Nick Sweeting
68a506e1a7
fix Snapshot admin actions buttons submitting on Safari
2021-04-06 00:43:49 -04:00
Nick Sweeting
5a9f27204a
dont use chrome when its not available on windows systems
2021-04-05 23:33:08 -04:00
Nick Sweeting
1485e4eb90
use older docker compose version for more features
2021-04-05 23:22:05 -04:00
Nick Sweeting
cadac4838e
fix searching by url contents
2021-04-05 23:21:07 -04:00
Nick Sweeting
2fc9f1cfa3
dont depend on README in Dockerfile install step
2021-04-05 23:17:07 -04:00
Nick Sweeting
8b4f84959a
new setup flag on init command to autosetup on init
2021-04-05 21:15:32 -04:00
Nick Sweeting
5c181532b5
only install dependencies in setup if they are missing
2021-04-05 21:15:10 -04:00
Nick Sweeting
21820acecd
import User model into core models for convenience
2021-04-05 21:14:22 -04:00
Nick Sweeting
bbebc33b7a
better autodetection of node modules dependencies
2021-04-05 21:14:06 -04:00
Nick Sweeting
211bf86f4a
fix bug checking value in subcommand_args when None
2021-04-05 21:13:49 -04:00
Nick Sweeting
79fb60a47a
dont allow caching when installing packages
2021-04-05 20:08:48 -04:00
Nick Sweeting
05c17af4dc
fix lint errors
2021-04-05 20:00:16 -04:00
Nick Sweeting
ef76f7c490
require running setup in existing data dir
2021-04-05 19:59:33 -04:00
Nick Sweeting
8c1d12020f
fix lint errors
2021-04-05 19:53:42 -04:00
Nick Sweeting
3449bac5bc
add new setup command to autoinstall dependencies
2021-04-05 19:51:57 -04:00
Nick Sweeting
24f7dd9c05
dont raise systemexit on succesful config
2021-04-05 19:51:40 -04:00
Nick Sweeting
ffd119c66c
fix linter
2021-04-03 15:51:27 -04:00
Nick Sweeting
7162649b03
fix search fields on admin snapshots list
2021-04-01 14:00:07 -04:00
Nick Sweeting
5492c204b8
tweak add page copy
2021-04-01 14:00:07 -04:00
Nick Sweeting
f59b6d4189
only add url-list lines that are real urls
2021-04-01 14:00:07 -04:00
Nick Sweeting
d73f7d7d96
add db_index on url field
2021-04-01 14:00:07 -04:00
Nick Sweeting
c438e49871
fix clicking on tags on snapshot list
2021-04-01 14:00:07 -04:00
Nick Sweeting
0e39a2098d
split migrations and data dir check
2021-04-01 14:00:07 -04:00
Nick Sweeting
5d7ad9b1b4
add form parser option
2021-04-01 02:34:16 -04:00
Nick Sweeting
59d5423483
fix snapshot icon caching and ordering
2021-04-01 02:22:15 -04:00
Nick Sweeting
fc5d99b4b4
fix missing package.json repo
2021-04-01 01:40:02 -04:00
Nick Sweeting
ff9445648b
fix navbar and tags display
2021-04-01 01:39:34 -04:00
Nick Sweeting
7ee7a94e67
fix str split using old overwrite pipe instead of append pipe
2021-03-31 11:30:15 -04:00
Nick Sweeting
005da31f8f
add overwrite flag option to schedule cmd
2021-03-31 11:29:51 -04:00
Nick Sweeting
534ae60ab0
dont overwrite the log on each run, and use dedicated log for scheduled archiving
2021-03-31 11:20:42 -04:00
Nick Sweeting
5d3a03b299
use stderr and hint in case of parser returning no urls instead of bare exception
2021-03-31 01:39:01 -04:00
Nick Sweeting
8ce93ff787
use KEY, NAME, and PARSER to define parsers instead of hardcoding in init
2021-03-31 01:05:49 -04:00
Nick Sweeting
36f0646501
Merge pull request #669 from FliegendeWurst/fix-issue-235
...
add command: --parser option (fixes #235 )
2021-03-31 00:53:47 -04:00
Nick Sweeting
2656e59215
change list style
2021-03-31 00:47:42 -04:00
Nick Sweeting
bd8a01e677
sigh vendor yet another dependency
2021-03-30 14:49:42 -04:00
FliegendeWurst
60bd9a902e
add command: --parser option
2021-03-28 10:09:11 +02:00
Nick Sweeting
a58ad5b272
allow larger tags
2021-03-27 05:52:42 -04:00
Nick Sweeting
7e7c134ec0
use exc
2021-03-27 05:32:07 -04:00
Nick Sweeting
5568b3b612
fix adding and removing tags via admin snapshot list
2021-03-27 05:27:02 -04:00
Nick Sweeting
49b77b91bb
fix unused var
2021-03-27 05:12:54 -04:00
Nick Sweeting
3e26ae4a66
support finding multiple urls as substrings in text
2021-03-27 04:30:40 -04:00
Nick Sweeting
f3a3d76439
support adding urls with tags directly via CLI and add page
2021-03-27 04:30:15 -04:00
Nick Sweeting
5fb9ca389f
check more url parsing invariants on startup
2021-03-27 03:57:22 -04:00
Nick Sweeting
fea0b89dbe
add tag cli option
2021-03-27 03:57:05 -04:00
Nick Sweeting
d486f8c162
use tempfile when logs dir is not available
2021-03-27 02:17:12 -04:00
Nick Sweeting
3c2f4c95cf
fix unused import
2021-03-27 01:50:26 -04:00
Nick Sweeting
56c6b11b14
add comment
2021-03-27 01:19:32 -04:00
Nick Sweeting
bd6d9c165b
enforce utf8 on literally all file operations because windows sucks
2021-03-27 01:16:29 -04:00
Nick Sweeting
185d2f9f9b
check for JSON1 extesnion in Sqlite on startup
2021-03-27 00:27:58 -04:00
Nick Sweeting
177d22da50
fix clicking add tag when no tag is selected throwing 500
2021-03-01 01:39:06 -05:00
Nick Sweeting
76ff789164
fix tags page rendering when snapshot has no updated timestamp
2021-03-01 00:40:41 -05:00
Nick Sweeting
0036e9cce2
add migration
2021-02-28 22:55:12 -05:00
Nick Sweeting
9fc9f52cfb
increase sqlite timeout and dont check for same thread condition
2021-02-28 22:55:00 -05:00
Nick Sweeting
1cabde3ccd
remove atomic transactions
2021-02-28 22:54:40 -05:00
Nick Sweeting
844b5c5e20
minor db fix
2021-02-28 22:54:04 -05:00
Nick Sweeting
62089fdb22
close sqlite3 connections if unused
2021-02-28 22:53:34 -05:00
Nick Sweeting
6aef1dd98d
dont check db if init is pending
2021-02-28 22:53:23 -05:00
Nick Sweeting
acbce25201
missing migrations
2021-02-18 08:05:05 -05:00
Nick Sweeting
9ead76f9f8
cross link snapshot index, admin, and actions pages using uuids
2021-02-18 08:04:50 -05:00
Nick Sweeting
c257ae240e
only enable debug toolbar if nothreading is on
2021-02-18 08:04:26 -05:00
Nick Sweeting
ca2bb673ea
improve inline admin forms and add tags autocomplete
2021-02-18 08:04:14 -05:00
Nick Sweeting
33df9c1ebe
fix after and before in remove tests
2021-02-18 06:21:44 -05:00
Nick Sweeting
9c07fbdc0b
unused import fix
2021-02-18 05:56:00 -05:00
Nick Sweeting
106f6adc59
fix lgtm alerts
2021-02-18 04:29:24 -05:00
Nick Sweeting
46a4197514
fix tests
2021-02-18 04:26:56 -05:00
Nick Sweeting
13c2b7c8eb
fix url used by gridview
2021-02-18 02:38:30 -05:00
Nick Sweeting
e3b69c450c
move db checks under check_db if statement
2021-02-18 02:37:33 -05:00
Nick Sweeting
74a9dd8880
add missing migrations
2021-02-18 02:36:21 -05:00
Nick Sweeting
7282f4a42a
add favicon and robots to internal static dir
2021-02-18 02:35:51 -05:00
Nick Sweeting
dcc812b9f5
make output link easier to see on archiveresult admin list
2021-02-18 02:35:33 -05:00
Nick Sweeting
df47486d31
make admin link easier to find on snapshot detail page
2021-02-18 02:35:11 -05:00
Nick Sweeting
8369ca0b87
fix snapshot detail page titles garbled due to double escaping
2021-02-18 02:34:56 -05:00
Nick Sweeting
ba2afa0672
allow cancelling init partway through and show hints to resume
2021-02-18 02:34:42 -05:00
Nick Sweeting
75e1bfd0a9
create_or_update ArchiveResults from history instead of get_or_create
2021-02-18 02:34:20 -05:00
Nick Sweeting
69e0302c2a
only add https to url if needed when prompting to save new snapshot
2021-02-18 02:34:03 -05:00
Nick Sweeting
9918bddb7e
remove emptystring tags before saving
2021-02-18 02:33:42 -05:00
Nick Sweeting
f1823381d0
improve latest title logic to take longest title
2021-02-18 02:33:28 -05:00
Nick Sweeting
ad4bd49163
increase max title length to 512
2021-02-18 02:33:08 -05:00
Nick Sweeting
4e5671dda9
add ability to add and remove tags directly from snapshot list
2021-02-18 02:32:58 -05:00
Nick Sweeting
c877d67e0c
check if pragma is already wal before setting
2021-02-18 02:31:42 -05:00
Nick Sweeting
7b7aa239fd
autocreate db cache table when needed and reenable WAL mode
2021-02-17 18:26:04 -05:00
