1
0
Fork 0
mirror of synced 2024-06-28 19:10:33 +12:00
Commit graph

1240 commits

Author SHA1 Message Date
Nick Sweeting d1051e40bd remove tzlocal 2021-04-10 04:42:39 -04:00
Nick Sweeting a9986f1f05 add timezone support, tons of CSS and layout improvements, more detailed snapshot admin form info, ability to sort by recently updated, better grid view styling, better table layouts, better dark mode support 2021-04-10 04:21:36 -04:00
Nick Sweeting cf7d7e4990 add new timezone autosetting and cache header setting middlewares 2021-04-10 04:21:36 -04:00
Nick Sweeting 1977ae8962 add more helper props to snapshot for getting latest headers, favicon, etc 2021-04-10 04:21:36 -04:00
Nick Sweeting 8d68f1744e tweak columns in private snapshots list and shorten action names 2021-04-10 04:21:36 -04:00
Nick Sweeting 722f530c80 support adding and removing multiple tags at once using autocomplete multiselect field 2021-04-10 04:21:36 -04:00
Nick Sweeting 06c58ac42a fix unreachable config backup clear 2021-04-09 12:15:47 -04:00
Nick Sweeting 32764347ce add new SEARCH_BACKEND_TIMEOUT config option defaulted to 90sec 2021-04-09 00:27:12 -04:00
Nick Sweeting 2e57df917e handle BaseExceptions properly 2021-04-08 06:08:17 -04:00
Nick Sweeting 2c07d8f1f3 fix lgtm alerts 2021-04-06 13:47:26 -04:00
Nick Sweeting 21bb84f627 fix lint errors 2021-04-06 02:50:51 -04:00
Nick Sweeting e92db03f5f fallback to using bin_path to find archivebox binary if sys.argv is empty 2021-04-06 02:45:57 -04:00
Nick Sweeting c98053a8ca allow filtering by archiveresult status 2021-04-06 02:07:22 -04:00
Nick Sweeting f67a5a215a fix readability indexing process and implement a max total character length on indexed content 2021-04-06 02:01:38 -04:00
Nick Sweeting b3a89172ab add original src url to dosctring 2021-04-06 01:38:24 -04:00
Nick Sweeting e7c7a8f71c force kill forked child processes after each extractor command 2021-04-06 01:35:21 -04:00
Nick Sweeting af6561b527 remove CDN jquery and bootstrap by inlining them instead 2021-04-06 01:13:31 -04:00
Nick Sweeting e4823cc5d7 add back CUSTOM_TEMPLATES_DIR config option 2021-04-06 00:57:20 -04:00
Nick Sweeting 4b098bea15 fix navbar showing user info even when not logged in 2021-04-06 00:44:01 -04:00
Nick Sweeting 68a506e1a7 fix Snapshot admin actions buttons submitting on Safari 2021-04-06 00:43:49 -04:00
Nick Sweeting 5a9f27204a dont use chrome when its not available on windows systems 2021-04-05 23:33:08 -04:00
Nick Sweeting 1485e4eb90 use older docker compose version for more features 2021-04-05 23:22:05 -04:00
Nick Sweeting cadac4838e fix searching by url contents 2021-04-05 23:21:07 -04:00
Nick Sweeting 2fc9f1cfa3 dont depend on README in Dockerfile install step 2021-04-05 23:17:07 -04:00
Nick Sweeting 8b4f84959a new setup flag on init command to autosetup on init 2021-04-05 21:15:32 -04:00
Nick Sweeting 5c181532b5 only install dependencies in setup if they are missing 2021-04-05 21:15:10 -04:00
Nick Sweeting 21820acecd import User model into core models for convenience 2021-04-05 21:14:22 -04:00
Nick Sweeting bbebc33b7a better autodetection of node modules dependencies 2021-04-05 21:14:06 -04:00
Nick Sweeting 211bf86f4a fix bug checking value in subcommand_args when None 2021-04-05 21:13:49 -04:00
Nick Sweeting 79fb60a47a dont allow caching when installing packages 2021-04-05 20:08:48 -04:00
Nick Sweeting 05c17af4dc fix lint errors 2021-04-05 20:00:16 -04:00
Nick Sweeting ef76f7c490 require running setup in existing data dir 2021-04-05 19:59:33 -04:00
Nick Sweeting 8c1d12020f fix lint errors 2021-04-05 19:53:42 -04:00
Nick Sweeting 3449bac5bc add new setup command to autoinstall dependencies 2021-04-05 19:51:57 -04:00
Nick Sweeting 24f7dd9c05 dont raise systemexit on succesful config 2021-04-05 19:51:40 -04:00
Nick Sweeting ffd119c66c fix linter 2021-04-03 15:51:27 -04:00
Nick Sweeting 7162649b03 fix search fields on admin snapshots list 2021-04-01 14:00:07 -04:00
Nick Sweeting 5492c204b8 tweak add page copy 2021-04-01 14:00:07 -04:00
Nick Sweeting f59b6d4189 only add url-list lines that are real urls 2021-04-01 14:00:07 -04:00
Nick Sweeting d73f7d7d96 add db_index on url field 2021-04-01 14:00:07 -04:00
Nick Sweeting c438e49871 fix clicking on tags on snapshot list 2021-04-01 14:00:07 -04:00
Nick Sweeting 0e39a2098d split migrations and data dir check 2021-04-01 14:00:07 -04:00
Nick Sweeting 5d7ad9b1b4 add form parser option 2021-04-01 02:34:16 -04:00
Nick Sweeting 59d5423483 fix snapshot icon caching and ordering 2021-04-01 02:22:15 -04:00
Nick Sweeting fc5d99b4b4 fix missing package.json repo 2021-04-01 01:40:02 -04:00
Nick Sweeting ff9445648b fix navbar and tags display 2021-04-01 01:39:34 -04:00
Nick Sweeting 7ee7a94e67 fix str split using old overwrite pipe instead of append pipe 2021-03-31 11:30:15 -04:00
Nick Sweeting 005da31f8f add overwrite flag option to schedule cmd 2021-03-31 11:29:51 -04:00
Nick Sweeting 534ae60ab0 dont overwrite the log on each run, and use dedicated log for scheduled archiving 2021-03-31 11:20:42 -04:00
Nick Sweeting 5d3a03b299 use stderr and hint in case of parser returning no urls instead of bare exception 2021-03-31 01:39:01 -04:00
Nick Sweeting 8ce93ff787 use KEY, NAME, and PARSER to define parsers instead of hardcoding in init 2021-03-31 01:05:49 -04:00
Nick Sweeting 36f0646501
Merge pull request #669 from FliegendeWurst/fix-issue-235
add command: --parser option (fixes #235)
2021-03-31 00:53:47 -04:00
Nick Sweeting 2656e59215
change list style 2021-03-31 00:47:42 -04:00
Nick Sweeting bd8a01e677 sigh vendor yet another dependency 2021-03-30 14:49:42 -04:00
FliegendeWurst 60bd9a902e add command: --parser option 2021-03-28 10:09:11 +02:00
Nick Sweeting a58ad5b272 allow larger tags 2021-03-27 05:52:42 -04:00
Nick Sweeting 7e7c134ec0 use exc 2021-03-27 05:32:07 -04:00
Nick Sweeting 5568b3b612 fix adding and removing tags via admin snapshot list 2021-03-27 05:27:02 -04:00
Nick Sweeting 49b77b91bb fix unused var 2021-03-27 05:12:54 -04:00
Nick Sweeting 3e26ae4a66 support finding multiple urls as substrings in text 2021-03-27 04:30:40 -04:00
Nick Sweeting f3a3d76439 support adding urls with tags directly via CLI and add page 2021-03-27 04:30:15 -04:00
Nick Sweeting 5fb9ca389f check more url parsing invariants on startup 2021-03-27 03:57:22 -04:00
Nick Sweeting fea0b89dbe add tag cli option 2021-03-27 03:57:05 -04:00
Nick Sweeting d486f8c162 use tempfile when logs dir is not available 2021-03-27 02:17:12 -04:00
Nick Sweeting 3c2f4c95cf fix unused import 2021-03-27 01:50:26 -04:00
Nick Sweeting 56c6b11b14 add comment 2021-03-27 01:19:32 -04:00
Nick Sweeting bd6d9c165b enforce utf8 on literally all file operations because windows sucks 2021-03-27 01:16:29 -04:00
Nick Sweeting 185d2f9f9b check for JSON1 extesnion in Sqlite on startup 2021-03-27 00:27:58 -04:00
Nick Sweeting 177d22da50 fix clicking add tag when no tag is selected throwing 500 2021-03-01 01:39:06 -05:00
Nick Sweeting 76ff789164 fix tags page rendering when snapshot has no updated timestamp 2021-03-01 00:40:41 -05:00
Nick Sweeting 0036e9cce2 add migration 2021-02-28 22:55:12 -05:00
Nick Sweeting 9fc9f52cfb increase sqlite timeout and dont check for same thread condition 2021-02-28 22:55:00 -05:00
Nick Sweeting 1cabde3ccd remove atomic transactions 2021-02-28 22:54:40 -05:00
Nick Sweeting 844b5c5e20 minor db fix 2021-02-28 22:54:04 -05:00
Nick Sweeting 62089fdb22 close sqlite3 connections if unused 2021-02-28 22:53:34 -05:00
Nick Sweeting 6aef1dd98d dont check db if init is pending 2021-02-28 22:53:23 -05:00
Nick Sweeting acbce25201 missing migrations 2021-02-18 08:05:05 -05:00
Nick Sweeting 9ead76f9f8 cross link snapshot index, admin, and actions pages using uuids 2021-02-18 08:04:50 -05:00
Nick Sweeting c257ae240e only enable debug toolbar if nothreading is on 2021-02-18 08:04:26 -05:00
Nick Sweeting ca2bb673ea improve inline admin forms and add tags autocomplete 2021-02-18 08:04:14 -05:00
Nick Sweeting 33df9c1ebe fix after and before in remove tests 2021-02-18 06:21:44 -05:00
Nick Sweeting 9c07fbdc0b unused import fix 2021-02-18 05:56:00 -05:00
Nick Sweeting 106f6adc59 fix lgtm alerts 2021-02-18 04:29:24 -05:00
Nick Sweeting 46a4197514 fix tests 2021-02-18 04:26:56 -05:00
Nick Sweeting 13c2b7c8eb fix url used by gridview 2021-02-18 02:38:30 -05:00
Nick Sweeting e3b69c450c move db checks under check_db if statement 2021-02-18 02:37:33 -05:00
Nick Sweeting 74a9dd8880 add missing migrations 2021-02-18 02:36:21 -05:00
Nick Sweeting 7282f4a42a add favicon and robots to internal static dir 2021-02-18 02:35:51 -05:00
Nick Sweeting dcc812b9f5 make output link easier to see on archiveresult admin list 2021-02-18 02:35:33 -05:00
Nick Sweeting df47486d31 make admin link easier to find on snapshot detail page 2021-02-18 02:35:11 -05:00
Nick Sweeting 8369ca0b87 fix snapshot detail page titles garbled due to double escaping 2021-02-18 02:34:56 -05:00
Nick Sweeting ba2afa0672 allow cancelling init partway through and show hints to resume 2021-02-18 02:34:42 -05:00
Nick Sweeting 75e1bfd0a9 create_or_update ArchiveResults from history instead of get_or_create 2021-02-18 02:34:20 -05:00
Nick Sweeting 69e0302c2a only add https to url if needed when prompting to save new snapshot 2021-02-18 02:34:03 -05:00
Nick Sweeting 9918bddb7e remove emptystring tags before saving 2021-02-18 02:33:42 -05:00
Nick Sweeting f1823381d0 improve latest title logic to take longest title 2021-02-18 02:33:28 -05:00
Nick Sweeting ad4bd49163 increase max title length to 512 2021-02-18 02:33:08 -05:00
Nick Sweeting 4e5671dda9 add ability to add and remove tags directly from snapshot list 2021-02-18 02:32:58 -05:00
Nick Sweeting c877d67e0c check if pragma is already wal before setting 2021-02-18 02:31:42 -05:00
Nick Sweeting 7b7aa239fd autocreate db cache table when needed and reenable WAL mode 2021-02-17 18:26:04 -05:00
Nick Sweeting d60d73754d fix favicon.ico and robots.txt served by runserver 2021-02-17 18:25:47 -05:00
Nick Sweeting a8a6752b06 add CACHE_BACKEND options to settings for easier dev 2021-02-17 18:25:23 -05:00
Nick Sweeting 3457773988 fix ArchiveResult extractor showing up on top of dropdown in admin inline form 2021-02-17 18:25:01 -05:00
Nick Sweeting 084cf7ff51 add more explanation about snapshot.save timestamp bump 2021-02-17 13:34:46 -05:00
Nick Sweeting 265bcc0264 fix lint errors2 2021-02-16 16:29:41 -05:00
Nick Sweeting 6f0eec92eb fix lint errors 2021-02-16 16:26:48 -05:00
Nick Sweeting d89034dcde disable debug_toolbar by default 2021-02-16 16:23:09 -05:00
Nick Sweeting 71cf8d5224 add migrations 2021-02-16 15:57:13 -05:00
Nick Sweeting 118bddb41a fix admin title and url truncation and add edit/admin link and ID to Snapshot detail page 2021-02-16 15:55:07 -05:00
Nick Sweeting 05e891632c add snapshot_id to Link and uuid to ArchiveResult 2021-02-16 15:54:27 -05:00
Nick Sweeting 0b8bdd8cc5 show dir size after each snapshot archiving process in cli 2021-02-16 15:53:52 -05:00
Nick Sweeting acb932ba12 improve readability and mercury error handling and fix output path to be relative 2021-02-16 15:53:11 -05:00
Nick Sweeting c95698e608 bump Snapshot.updated time after each extractor, change extractor order 2021-02-16 15:52:18 -05:00
Nick Sweeting 28e50c5e49 tweak snapshot asset serving logic to show multiple choices in case of conflict 2021-02-16 15:51:56 -05:00
Nick Sweeting a6c64f2560 only get archive_size if index.html is present within dir 2021-02-16 15:51:17 -05:00
Nick Sweeting c24e4bf11d tweak django debug toolbar panels and add djdt_flamegraph 2021-02-16 15:51:02 -05:00
Nick Sweeting 1e2188517b speed up latest_title by preferring db title if present 2021-02-16 15:49:53 -05:00
Nick Sweeting 8b236b9367 cache dir size, snapshot icons, tags str, and title in django cache 2021-02-16 15:49:29 -05:00
Nick Sweeting 51440ede3a show extra fields on ArchiveResult admin list and detail page 2021-02-16 15:47:49 -05:00
Nick Sweeting 74c882ef16 add nothreading arg to archivebox server for perf profiling 2021-02-16 15:46:58 -05:00
Nick Sweeting 8cde6bc676 add gist.github.com to git domains 2021-02-16 07:36:09 -05:00
Nick Sweeting bdf1b102be load ArchiveResults from orphaned links history during init 2021-02-16 06:20:05 -05:00
Nick Sweeting d0f8a5e710 change mercury atomic_write output order 2021-02-16 06:19:16 -05:00
Nick Sweeting e62ecffeb0 disable django caches explicitly 2021-02-16 06:18:39 -05:00
Nick Sweeting 988a10a9f6 fix warc path in snapshot_icons 2021-02-16 06:18:05 -05:00
Nick Sweeting 48eddadc9b inline archive_size and archive_path snapshot methods 2021-02-16 06:16:02 -05:00
Nick Sweeting 46a7b972ee disable WAL and cache table for now 2021-02-16 06:13:50 -05:00
Nick Sweeting 24ae01cdbe add ArchiveResult admin 2021-02-16 06:13:27 -05:00
Nick Sweeting 87069dd41e fix wsgi.py 2021-02-16 04:16:51 -05:00
Nick Sweeting 8c2d81c041 always serve static files 2021-02-16 04:16:24 -05:00
Nick Sweeting fb8e6cabcb fix error log location 2021-02-16 04:15:09 -05:00
Nick Sweeting 26fa63749d add debug toolbar 2021-02-16 02:50:05 -05:00
Nick Sweeting fad2620c62 minor tweaks 2021-02-16 02:49:31 -05:00
Nick Sweeting ae43d35e65 fix color option in docker 2021-02-16 01:26:26 -05:00
Nick Sweeting 4eac9d0ec1 show extra system setup info in version output 2021-02-16 01:23:58 -05:00
Nick Sweeting 82de67db34 fix missing/outdated template variables 2021-02-16 01:23:31 -05:00
Nick Sweeting 22da885148 log every archivebox command run to the errors.log 2021-02-16 01:23:01 -05:00
Nick Sweeting 19f7c907e0 exec archivebox from docker entrypoint script to avoid nesting pid under bash 2021-02-16 01:22:36 -05:00
Nick Sweeting 10d687c55c add new MEDIA_MAX_SIZE=750m size option for limiting youtubedl downloads 2021-02-16 01:21:19 -05:00
Nick Sweeting 49939f3eaa only accept stdin if args are not passed, fix stdin hang in docker 2021-02-16 01:20:47 -05:00
Nick Sweeting de1a939df4 fix public index pagination links and total count 2021-02-15 21:07:12 -05:00
Nick Sweeting 108cb100b3 hotlink direclty to specific archivebox release from footer info 2021-02-15 21:06:56 -05:00
Nick Sweeting 56ec19fb29 allow hotlinking directly to a specific extractor output using hash on Snapshot detail page 2021-02-15 20:55:22 -05:00
Nick Sweeting a089136fea fix missing stdout template var on /add when not rendering success page 2021-02-15 20:54:47 -05:00
Nick Sweeting b6d7c74680 speed up the Snapshot handling view and show index page when extractor output is missing or multiple snapshots returned 2021-02-15 20:52:08 -05:00
Nick Sweeting 0375853683 log error tracebacks to logs/errors.log file and filter noisy 404s and 200s from log output 2021-02-15 20:51:23 -05:00
Nick Sweeting 0ec9bfb971 fix dead missing template variables 2021-02-15 20:50:12 -05:00
Nick Sweeting b3a50a2c10 fix server quick-init param not being passed properly to subcommand 2021-02-15 20:49:40 -05:00
Nick Sweeting b06e256ad9 fix add command not updating snapshot detail index pages when passed index-only and overwrite flags together 2021-02-15 20:49:23 -05:00
Nick Sweeting 8e98cef7ad fix after and before args flipped when filtering 2021-02-15 20:48:51 -05:00
Nick Sweeting 33d180afe7 allow filtering snapshots by timestamp in list, update, and remove cmds 2021-02-15 20:48:35 -05:00
Nick Sweeting 0c9db1c554 remove symbols from random secret key for easier copy pastin 2021-02-15 20:45:42 -05:00
Nick Sweeting 4faef03ba3 compute snapshot properties directly without loading whole Link 2021-02-15 20:44:08 -05:00
Nick Sweeting 9ce3bd5bdc use index.LINK_FILTERS to validate filter-type args instead of hardocding them twice 2021-02-15 20:43:36 -05:00
Nick Sweeting c28ad8bd1b fix AddLinkForm widget complaining about missing template var class 2021-02-15 20:42:59 -05:00
Nick Sweeting 78463c243a remove unused GIT_SHA config option 2021-02-15 20:42:33 -05:00
Nick Sweeting 9cd4ba38f0 add new SNAPSHOTS_PER_PAGE pagination limit config 2021-02-15 20:42:00 -05:00
Nick Sweeting 00ae1f15a7 ignore shm db file and config files in archivebox data dir on init 2021-02-15 14:52:37 -05:00
Nick Sweeting 3c3bae02d2 add quick-init option to skip reimporting all snapshot dirs on init 2021-02-15 14:52:10 -05:00
Nick Sweeting e61e12c889 use setup.py to determine dependencies in Dockerfile instead of egg-info requires.txt 2021-02-15 14:51:32 -05:00
Nick Sweeting 0407d03b6b add cli tests file back 2021-02-15 13:39:49 -05:00
Nick Sweeting 611216765d switch sqlite to use WAL mode by default to prevent database locked errors 2021-02-15 13:39:03 -05:00
Nick Sweeting 683a08772b change wording of db not found error 2021-02-08 23:27:46 -05:00
Nick Sweeting 6705354e57 fix assertion 2021-02-08 23:24:48 -05:00
Nick Sweeting a49884ade8 fix emptystrings in cmd_version causing exception 2021-02-08 23:22:02 -05:00
Nick Sweeting 171bbeb69b catch exception on import of old index.json into ArchiveResult 2021-02-01 16:31:29 -05:00
Nick Sweeting 0aea5ed3e8 fix handling of skipped ArchiveResult entries with null output 2021-02-01 14:37:34 -05:00
Nick Sweeting c4b02be24d remove dead tests code 2021-02-01 05:14:43 -05:00
Nick Sweeting 783f597955 minor build fixes 2021-02-01 05:13:46 -05:00
Nick Sweeting aa84a7ff2b fix migration creating conflicting tags based on slug 2021-02-01 05:13:23 -05:00
Nick Sweeting 7d0f5653c3 fix lgtm alerts 2021-02-01 02:27:24 -05:00
Nick Sweeting 04c951cdd5 fix alerts 2021-02-01 02:22:02 -05:00
Nick Sweeting 534ead2440 use the db exclusively for icons instead of hammering filesystem 2021-02-01 02:18:13 -05:00
Nick Sweeting 923f517a8f minor fixes 2021-02-01 02:17:54 -05:00
Nick Sweeting 560d3103a8 cleanup snapshot detail page UI 2021-01-30 22:04:24 -05:00
Nick Sweeting 54c5331693 check for output existance when rendering files icons 2021-01-30 22:04:14 -05:00
Nick Sweeting 15e87353bd only show archive.org if enabled 2021-01-30 22:03:59 -05:00
Nick Sweeting 846c966c4d use globbing to find wget output path 2021-01-30 22:02:39 -05:00
Nick Sweeting e6fa16e13a only chmod wget output if it exists 2021-01-30 22:02:11 -05:00
Nick Sweeting 385daf9af8 save the url as title for staticfiles or non html files 2021-01-30 22:01:49 -05:00
Nick Sweeting 24e24934f7 add headers.json and fix relative singlefile path resolving for sonic 2021-01-30 21:59:34 -05:00
Nick Sweeting c089501073 add response status code to headers.json 2021-01-30 20:44:49 -05:00
Nick Sweeting b9b1c3d9e8 fix singlefile output path not relative 2021-01-30 20:44:49 -05:00
Nick Sweeting d072f1d413 hide ssl warnings when checking SSL is disabled 2021-01-30 20:44:49 -05:00
Nick Sweeting 9d24bfd0dc disable progress bars on mac again 2021-01-30 20:44:49 -05:00
Nick Sweeting 326ce78496 simplify debug 2021-01-30 06:09:26 -05:00
Nick Sweeting d6de04a83a fix lgtm errors 2021-01-30 06:07:35 -05:00
Nick Sweeting cc80ceb0a2 fix icons in public index 2021-01-30 05:47:55 -05:00
Nick Sweeting 1ce0eca217 add trailing slashes to canonical paths 2021-01-30 05:47:55 -05:00
Nick Sweeting 6edae6a17f add future api spec design 2021-01-30 05:47:55 -05:00
Nick Sweeting a98298103d cleanup templates and views 2021-01-30 05:47:55 -05:00
Nick Sweeting ed13ec7655 remove active theme 2021-01-30 05:47:55 -05:00
Nick Sweeting c2aaa41c76 fix missing str path 2021-01-30 01:25:08 -05:00
Nick Sweeting ff7d2ffa09 fix version in legacy footer 2021-01-29 09:18:38 -05:00
Nick Sweeting 6e84890abd improve loading snapshots tooltips 2021-01-29 09:09:23 -05:00
Nick Sweeting 8a4edb45e7 also search url, timestamp, tags on public index 2021-01-29 09:08:03 -05:00
Nick Sweeting f6c3683ab8 fix snapshot favicon loading spinner height 2021-01-29 00:15:32 -05:00
Nick Sweeting 3227f54b52 limit youtubedl download size to 750m and stop splitting out audio files 2021-01-29 00:15:32 -05:00
Nick Sweeting d7df9e58ea hide footer on add page 2021-01-28 23:15:05 -05:00
Nick Sweeting 5c54bcc1f3 fix files icons greying out on public index 2021-01-28 22:57:12 -05:00