Angel Rey
|
a8a8fd14ac
|
Fixed indent headers.json
|
2020-09-23 11:07:00 -05:00 |
|
|
Angel Rey
|
852e3c9cff
|
Added headers extractor
|
2020-09-23 11:07:00 -05:00 |
|
Cristian
|
eb34a6af62
|
lint: Fix mercury extractor lint issues
|
2020-09-23 10:35:39 -05:00 |
|
|
Cristian
|
46b9e3d536
|
fix: Fix mercury extractor test
|
2020-09-23 10:34:05 -05:00 |
|
ttimasdf
|
2bf496e7e9
|
feat: Add mercury-parsed content to summary page
|
2020-09-22 18:44:12 -05:00 |
|
|
ttimasdf
|
357b677363
|
fix: add mercury-parser to extractors list
|
2020-09-22 18:44:12 -05:00 |
|
|
ttimasdf
|
706bd895e0
|
feat: Add mercury-parser
|
2020-09-22 18:44:12 -05:00 |
|
|
Cristian
|
b18bbf8874
|
test: Fix tests post-rebase
|
2020-09-17 09:09:52 -05:00 |
|
apkallum
|
422664079a
|
fix test type casting for folder['path']
|
2020-09-17 09:09:52 -05:00 |
|
|
apkallum
|
0144f19227
|
fix github action folder listing
|
2020-09-17 09:09:52 -05:00 |
|
|
apkallum
|
1aa7bac85b
|
fix oneshot command type signature
|
2020-09-17 09:09:52 -05:00 |
|
|
apkallum
|
95157427c2
|
update stubs file
|
2020-09-17 09:09:52 -05:00 |
|
|
apkallum
|
008769d296
|
add support for Paths in json encoder
|
2020-09-17 09:09:52 -05:00 |
|
|
apkallum
|
abf68e5437
|
no home() in Paths
|
2020-09-17 09:09:52 -05:00 |
|
|
apkallum
|
b99784b919
|
pathlib with / syntax for config, index
|
2020-09-17 09:09:52 -05:00 |
|
|
apkallum
|
594d9e49ce
|
first attempt to migrate to Pathlib
|
2020-09-17 09:09:52 -05:00 |
|
|
Cristian
|
b2ed96c35a
|
feat: Redirect old add view to the main one
|
2020-09-17 09:08:20 -05:00 |
|
|
Cristian
|
b3ec170e39
|
fix: Remove unused imports
|
2020-09-16 08:50:56 -05:00 |
|
|
Cristian
|
bc116c25f8
|
refactor: Change View to FormView
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
a06bd715a9
|
remove reference to old home
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
1cdaad00a8
|
no more oldhome, cbvs uniform across views
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
94a590b31a
|
factor out a base.html template
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
5e8c115f3f
|
unify public archive view
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
3288f8579b
|
add public add view + toggle setting
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
6f7cc2b3ef
|
ensure results have icons
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
3048c0f6dc
|
add icons to new public view
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
c50af04cce
|
search view inherits from modified public view
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
948b2469f6
|
no files count in public view
|
2020-09-16 08:50:56 -05:00 |
|
|
apkallum
|
5c4ac3cf3d
|
new public view derived from django
|
2020-09-16 08:50:56 -05:00 |
|
|
Cristian
|
50f3f16203
|
lint: Remove unused import
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
0a83392cbf
|
fix: Replace any typing with Union[Iterable[Link], QuerySet] in archive_links
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
779a446085
|
feat: Make title and tags editable in admin
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
5348f4735a
|
fix: Change check to avoid issues with empty querysets
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
cf18130f85
|
feat: Add deprecation warning for index.json
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
018bd91745
|
refactor: Remove get_iter lambda from archive_links
|
2020-09-15 08:05:46 -05:00 |
|
Cristian Vargas
|
5e9b3099c6
|
Update fix_duplicate_links_in_index docstring
Co-authored-by: Nick Sweeting <git@sweeting.me>
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
01fb44fd40
|
refactor: Change archive_links check to focus on queryset, so it allows other iterables and not just lists
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
fa622d3e14
|
refactor: Replace --index with --with-headers in the list command to make it more explicit. Change it so it affects the csv output too.
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
2aa8d69b72
|
fix: Save history in main index (to mimic previous behaviour)
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
7e9d195d13
|
feat: Update list command to sort using sqlite
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
f55153eab3
|
feat: Update update command to work with querysets
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
fe9604a772
|
feat: Add tests for remove command
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
a8ed72501d
|
feat: Refactor remove command to use querysets
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
be520d137a
|
feat: Refactor add method to use querysets
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
6a2e6aad2f
|
fix: status command was failing on empty archives
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
be0dff8126
|
feat: Add tests to refactored init command
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
404f333e17
|
feat: Refactor get_invalid_folders to work with a queryset instead of a list of links
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
dae606de6e
|
feat: Update init to take advantage of querysets to reduce memory consumption
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
6b4b7127b4
|
feat: Remove unused imports
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
b8585dd92e
|
feat: load_main_index returns a queryset now
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
a77d6dc235
|
feat: list command fails when --index is used without --json or --html
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
885ff50449
|
feat: Add html export to list command
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
aab8f96520
|
feat: Add flag to list command to support index like output
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
be57db1369
|
feat: Save static indexes at the end of init
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
c16fdf1b47
|
feat: Update data folder check
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
874403e667
|
feat: Remove patch_main_index
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
31343c1367
|
feat: Update extractors and add command to use sql index as source of truth
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
e9caee6b10
|
feat: Update status command to consider sql as the main index
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
02f36b2096
|
feat: Replace index.json with index.sql as the main index in init
|
2020-09-15 08:05:46 -05:00 |
|
|
Cristian
|
bd3c824d45
|
fix: Escape JSON output on command failure so the user can run the command manually
|
2020-09-04 10:23:41 -05:00 |
|
Nick Sweeting
|
a645f36b87
|
add comment about fake cmd
|
2020-09-01 19:42:22 -04:00 |
|
|
Cristian
|
66037535fd
|
feat: Add curl command on readability as default command to debug
|
2020-09-01 10:16:24 -05:00 |
|
|
Cristian
|
bf3ea42141
|
fix: Add a default cmd value to handle case where the html cannot be retrieved
|
2020-08-27 09:51:33 -05:00 |
|
|
Nick Sweeting
|
d179264cb7
|
dont warn about chrome twice
|
2020-08-18 20:08:04 -04:00 |
|
|
Nick Sweeting
|
a2c158e43e
|
catch OSErrors due to missing path
|
2020-08-18 19:09:45 -04:00 |
|
|
Nick Sweeting
|
4428134073
|
fix version parsing bug
|
2020-08-18 19:09:45 -04:00 |
|
|
Nick Sweeting
|
75deaf293d
|
Merge branch 'master' into node_config
|
2020-08-18 19:04:50 -04:00 |
|
|
Nick Sweeting
|
104b6e2a7b
|
ignore font assets by default
|
2020-08-18 18:58:05 -04:00 |
|
|
Nick Sweeting
|
d803481bd8
|
better folder list column output
|
2020-08-18 18:46:21 -04:00 |
|
|
Nick Sweeting
|
7144e0bdce
|
search for node dependencies in output dir first
|
2020-08-18 18:40:19 -04:00 |
|
|
Nick Sweeting
|
4566904db2
|
better colors and hints in version
|
2020-08-18 18:38:14 -04:00 |
|
|
Nick Sweeting
|
31a6318582
|
better colors, hints, and progress bars in docker
|
2020-08-18 18:32:11 -04:00 |
|
|
Nick Sweeting
|
5cb13ff1dd
|
ignore node cruft in output folder during init
|
2020-08-18 18:20:26 -04:00 |
|
|
Nick Sweeting
|
364388424c
|
update config stubs
|
2020-08-18 18:15:45 -04:00 |
|
|
Nick Sweeting
|
f6484d8ddc
|
add node modules to path
|
2020-08-18 18:14:56 -04:00 |
|
|
Nick Sweeting
|
87b79fe5e3
|
cleanup sdist and bdist build process
|
2020-08-18 17:12:49 -04:00 |
|
|
Nick Sweeting
|
73408fb035
|
fix version parsing and attempt to npm install during pip post_install
|
2020-08-18 15:58:05 -04:00 |
|
|
Nick Sweeting
|
71788cfd26
|
single-file supports version cli flag now
|
2020-08-18 15:00:12 -04:00 |
|
|
Nick Sweeting
|
8b427c9d79
|
get VERSION from package.json instead of VERSION to avoid duplication
|
2020-08-18 15:00:00 -04:00 |
|
|
Cristian
|
cc0fa747ce
|
feat: Add options to ease management of node related extractors
|
2020-08-18 10:34:28 -05:00 |
|
|
Nick Sweeting
|
73fad928df
|
0.4.17 release
|
2020-08-18 09:47:32 -04:00 |
|
|
Nick Sweeting
|
e87f1d57a3
|
fix linters
|
2020-08-18 09:22:12 -04:00 |
|
|
Nick Sweeting
|
61ab952dab
|
fix parser docstring
|
2020-08-18 09:20:05 -04:00 |
|
|
Nick Sweeting
|
e29bfba053
|
change iframe panel size and shape on details index
|
2020-08-18 09:17:56 -04:00 |
|
|
Nick Sweeting
|
430d51756b
|
show archive dir size in link details page
|
2020-08-18 09:17:37 -04:00 |
|
|
Nick Sweeting
|
273588c75a
|
change main link to point to link details index to reduce confusion
|
2020-08-18 09:17:21 -04:00 |
|
|
Nick Sweeting
|
6087e30d38
|
dont allow ui editing of db fields for now
|
2020-08-18 09:17:01 -04:00 |
|
|
Nick Sweeting
|
c9b3bab84d
|
fix pull title not working
|
2020-08-18 08:49:26 -04:00 |
|
|
Nick Sweeting
|
f18d92570e
|
wip attempt to fix timestamp unique constraint errors
|
2020-08-18 08:30:09 -04:00 |
|
|
Nick Sweeting
|
b0c0a676f8
|
re-enable readability and singlefile by default now that its less noisy
|
2020-08-18 08:29:46 -04:00 |
|
|
Nick Sweeting
|
15efb2d5ed
|
new generic_html parser for extracting hrefs
|
2020-08-18 08:29:05 -04:00 |
|
|
Nick Sweeting
|
a682a9c478
|
make all parsers accept arbitrary meta kwargs
|
2020-08-18 08:27:47 -04:00 |
|
|
Nick Sweeting
|
2e2b4f8150
|
fix url is too long to be a path error
|
2020-08-18 08:23:57 -04:00 |
|
|
Nick Sweeting
|
e3ac4c2405
|
htmldecode downloaded sources before parsing for links
|
2020-08-18 08:23:20 -04:00 |
|
|
Nick Sweeting
|
fc0d064ddf
|
add hints
|
2020-08-18 08:22:34 -04:00 |
|
|
Nick Sweeting
|
9d093a34f8
|
better dependency version check
|
2020-08-18 08:21:55 -04:00 |
|
|
Nick Sweeting
|
2f282402c7
|
add hint and prefix stdout utils
|
2020-08-18 08:21:36 -04:00 |
|
|
Nick Sweeting
|
2557a0da04
|
tweak cli output coloring
|
2020-08-18 08:16:18 -04:00 |
|
|
Nick Sweeting
|
d7d53cfb12
|
dont show skipped extractors to reduce visual noise
|
2020-08-18 08:13:35 -04:00 |
|
|
Nick Sweeting
|
5f84a7bc6e
|
better handle the case where json index lags behind sql index
|
2020-08-18 08:13:13 -04:00 |
|
|
Nick Sweeting
|
77d2f08a5c
|
show more info in merge conflict error message
|
2020-08-18 08:12:35 -04:00 |
|
|
Nick Sweeting
|
718d39e242
|
add common code extensions to default blacklist
|
2020-08-18 08:12:10 -04:00 |
|
|
Nick Sweeting
|
3658153cf8
|
fix url parsing through quotes
|
2020-08-18 08:04:57 -04:00 |
|
|
Nick Sweeting
|
a3155b88a1
|
better error message when nothing left to resume
|
2020-08-18 04:42:39 -04:00 |
|
|
Nick Sweeting
|
f8eb245519
|
better archivebox schedule CLI UX
|
2020-08-18 04:39:58 -04:00 |
|
|
Nick Sweeting
|
ec5609a09a
|
fix resume flag on archivebox update
|
2020-08-18 04:39:39 -04:00 |
|
|
Nick Sweeting
|
12f3977da5
|
fix source filenames missing domain
|
2020-08-18 04:39:29 -04:00 |
|
|
Nick Sweeting
|
ec1cce283e
|
include failed link updates in total count stats
|
2020-08-18 04:39:19 -04:00 |
|
|
Nick Sweeting
|
6e142e3531
|
match existing snapshots on url only, ignoring timestamp
|
2020-08-18 04:39:04 -04:00 |
|
|
Nick Sweeting
|
f371032b71
|
show warning when killing archivebox during index writing
|
2020-08-18 04:38:29 -04:00 |
|
|
Nick Sweeting
|
92de20af15
|
better detect missing dependencies on startup
|
2020-08-18 04:38:13 -04:00 |
|
|
Nick Sweeting
|
b681a477ae
|
add overwrite flag to add command to force re-archiving
|
2020-08-18 04:37:54 -04:00 |
|
|
Nick Sweeting
|
da671532a4
|
fix schedule cmd defaulting to day and remove week
|
2020-08-18 04:35:59 -04:00 |
|
|
Nick Sweeting
|
84a8b0d713
|
hide progress bars on mac
|
2020-08-18 03:27:52 -04:00 |
|
|
Nick Sweeting
|
29c8da83d6
|
0.4.16 release
|
2020-08-18 02:08:52 -04:00 |
|
|
Nick Sweeting
|
7c16944a44
|
Merge pull request #446 from cdvv7788/hotfix/#445
|
2020-08-18 02:06:32 -04:00 |
|
|
Nick Sweeting
|
7638dc45ea
|
0.4.15 release
|
2020-08-18 01:59:50 -04:00 |
|
|
Nick Sweeting
|
235eb20dbd
|
support cron in docker
|
2020-08-18 01:59:04 -04:00 |
|
|
Nick Sweeting
|
494be09bc2
|
add depth flag to schedule cmd
|
2020-08-18 01:58:54 -04:00 |
|
|
Cristian
|
05c71fc302
|
fix: Organize readability extractor so a timeout does not break the whole process
|
2020-08-17 08:34:40 -05:00 |
|
|
Nick Sweeting
|
225b63b732
|
skip invalid urls at all stages
|
2020-08-17 03:12:17 -04:00 |
|
|
Nick Sweeting
|
429f39dec1
|
0.4.14 release
|
2020-08-14 13:13:50 -04:00 |
|
|
Nick Sweeting
|
58e928520a
|
tweak log output for skipped methods
|
2020-08-14 13:12:50 -04:00 |
|
|
Nick Sweeting
|
03b73bfe77
|
Update archivebox/extractors/readability.py
|
2020-08-14 12:55:22 -04:00 |
|
|
Nick Sweeting
|
050b717bb9
|
Merge branch 'master' into readability-extractor
|
2020-08-14 12:35:35 -04:00 |
|
|
Nick Sweeting
|
0ef2b17678
|
only show data locations in version output when in a data dir
|
2020-08-13 23:21:57 -04:00 |
|
|
Nick Sweeting
|
a0901ba474
|
use BIND_ADDR config default for runserver
|
2020-08-13 23:21:37 -04:00 |
|
|
Cristian
|
b7aa3df8d2
|
feat: Disable singlefile and readability by default
|
2020-08-12 14:42:21 -05:00 |
|
|
Cristian
|
eb3528fa9f
|
feat: Add readability output to legacy index.html
|
2020-08-11 12:14:13 -05:00 |
|
|
Cristian
|
5dc7e63792
|
feat: Update dockerfile to support readability
|
2020-08-11 11:52:43 -05:00 |
|
|
Cristian
|
2a68af1b94
|
tests: Add readability tests
|
2020-08-11 11:15:15 -05:00 |
|
|
Cristian
|
8aa7b34de7
|
tests: Add readability to ignored methods in tests
|
2020-08-11 08:58:49 -05:00 |
|
|
Cristian
|
dc87d8b68c
|
tests: Update failing tests
|
2020-08-11 08:48:13 -05:00 |
|
|
Cristian
|
0ec747f64e
|
feat: Look in wget, singlefile or dom outputs before attempting to download the information again
|
2020-08-11 08:37:12 -05:00 |
|
|
Cristian
|
a14762640e
|
feat: Avoid running readability when the target is a file
|
2020-08-11 08:37:12 -05:00 |
|
|
Cristian
|
61e08a7c43
|
docs: Update docs link
|
2020-08-11 08:37:12 -05:00 |
|
|
Cristian
|
b33c66a9f7
|
feat: Split output of readability into multiple files
|
2020-08-11 08:37:12 -05:00 |
|
|
Cristian
|
7e2b249388
|
feat: Initial version of readability extractor
|
2020-08-11 08:37:12 -05:00 |
|
|
apkallum
|
50069d1eb3
|
set tz variable globally as UTC
|
2020-08-10 23:21:02 -04:00 |
|
|
apkallum
|
e9bd0b122e
|
fix: utc timing for initial command log as well
|
2020-08-10 19:17:17 -04:00 |
|
|
Nick Sweeting
|
cd09d1b077
|
0.4.13 release
|
2020-08-10 14:39:06 -04:00 |
|
|
Nick Sweeting
|
fcbc61917e
|
0.4.12 release
|
2020-08-10 14:26:32 -04:00 |
|
|
Nick Sweeting
|
33ab7fd4ec
|
autodetect when running inside docker and provide hints
|
2020-08-10 14:18:04 -04:00 |
|
|
Nick Sweeting
|
f24cb3dcbe
|
add docker help text
|
2020-08-10 13:42:31 -04:00 |
|
|
Nick Sweeting
|
430be7bc68
|
add missing staticfile check to singlefile
|
2020-08-10 13:42:20 -04:00 |
|
|
Cristian
|
76846d18a0
|
docs: Improve message for missing singlefile binary
|
2020-08-10 09:00:10 -05:00 |
|
|
Cristian
|
e358634f81
|
fix: Add missing configuration that breaks on edge case where only single file is being used
|
2020-08-08 09:12:14 -05:00 |
|
|
Nick Sweeting
|
87ba82ad39
|
0.4.11 release
|
2020-08-06 23:10:59 -04:00 |
|
|
Nick Sweeting
|
5b8abb2dce
|
bump version
|
2020-08-06 23:10:37 -04:00 |
|
|
Nick Sweeting
|
19aa5c3e94
|
fix SAVE_SINGLEFILE setting to depend on chrome
|
2020-08-06 23:07:25 -04:00 |
|