CodeMirror/ArchiveBox
1
0
Fork 0
mirror of synced 2024-07-03 13:30:28 +12:00
Code Issues Projects Releases Wiki Activity
839 commits 5 branches 31 tags 9.4 MiB
Commit graph

49 commits

Author SHA1 Message Date
Nick Sweeting e6bd1f8ca8 major codebase-wide code cleanups 2019-03-21 01:28:12 -04:00
Nick Sweeting c806068683 fix missing import 2019-03-20 21:12:43 -04:00
Nick Sweeting c79e1df8b2 use hints system to pass stdout and errors up to the traceback 2019-03-20 21:10:09 -04:00
Nick Sweeting eb5cc8078a better UX before titles have been fetched during archiving progress 2019-03-19 18:10:11 -04:00
noncetonic e230e27929
Changes HEADLESS_USER_AGENT to CHROME_USER_AGENT 2019-03-19 08:13:27 -07:00
noncetonic a13f22d15a
Adds support for HEADLESS_USER_AGENT for Chrome 2019-03-19 05:32:48 -07:00
Nick Sweeting 1c1bc76ac1 add chrome headless option and improve default data dir finding 2019-03-12 17:50:10 -04:00
Nick Sweeting 10bb970d66
Update archive_methods.py 2019-03-12 12:45:33 -04:00
Nick Sweeting c474bb7992
fix settings not being applied 2019-03-11 03:13:59 -04:00
Nick Sweeting b2ccb7dbcb refactor error hint printing to be DRYer 2019-03-08 16:25:42 -05:00
Nick Sweeting 552734241b put git clones in a git folder to avoid retries 2019-02-27 15:55:39 -05:00
Nick Sweeting 3eaa76267e fix keyerror domain bug 2019-02-27 15:42:53 -05:00
Nick Sweeting b03e9fade8 better link corruption guards, remove title prefetching, save index after run 2019-02-21 17:45:28 -05:00
Nick Sweeting c95632883e fix without_hash to without_fragment 2019-02-21 16:03:19 -05:00
Nick Sweeting d689264365 add new config and dependency options 2019-02-21 15:47:15 -05:00
Nick Sweeting d52c9c5304 allow passing COOKIES_FILE to wget 2019-02-21 12:58:51 -05:00
Nick Sweeting 5a7d00a639 fetch page title during archiving process 2019-02-19 01:44:54 -05:00
Nick Sweeting e6d5cd4432 ignore robots.txt when using wget 2019-02-06 22:06:36 -08:00
Nick Sweeting 56d382235f better progress output 2019-02-06 22:06:36 -08:00
Nick Sweeting 522b6288c1 hide frequent youtubedl incompatibility 2019-02-04 21:18:17 -08:00
Nick Sweeting 78ffa57c06 ignore hsts as well 2019-02-04 20:37:19 -08:00
Nick Sweeting 637f63dc80 chmod media after downloading 2019-02-04 20:15:34 -08:00
Nick Sweeting 756f31a50b hide common youtubedl failures 2019-02-04 20:08:54 -08:00
Nick Sweeting 794c043fa4 fix chrome data dir in wrong folder 2019-02-04 20:03:16 -08:00
Nick Sweeting 9f5441c4f3 quieter media downloads 2019-02-04 20:03:06 -08:00
Nick Sweeting 4e5b466bb2 fix broken wget user agents 2019-02-04 20:02:54 -08:00
Nick Sweeting d8c7c98c01 fix exception format string 2019-02-04 20:02:40 -08:00
Nick Sweeting 705abda1a5 send ArchiveBox user agent to archive.org 2019-02-04 20:02:33 -08:00
Nick Sweeting 8a9b4d6975 dont git clone urls with hashes 2019-02-04 19:21:08 -08:00
Nick Sweeting b8e5e1d5bd skip media when folder exists, even if empty 2019-02-04 19:21:00 -08:00
Nick Sweeting 901666bae6 dont show wget errors unless all files failed to download 2019-02-04 19:20:36 -08:00
Nick Sweeting ad7038e031 catch json parse errors in link archiving 2019-02-04 08:00:08 -08:00
Nick Sweeting 071b39b944 nicer error formats 2019-01-20 14:08:33 -05:00
Nick Sweeting de6eb649e0 disable wget timestamping if using warcs 2019-01-20 14:08:33 -05:00
Nick Sweeting ce07a6c738 patch subprocess.run to have better timeout handling 2019-01-20 14:08:33 -05:00
Nick Sweeting 7ea36c4adb bump timeouts and improve curl archive method 2019-01-20 14:08:33 -05:00
Nick Sweeting c42fcd42d7 fetch warc file inline with wget instead of as separate step 2019-01-14 22:43:14 -05:00
Nick Sweeting 321cdeda11 add mirror to git clone 2019-01-12 20:13:13 -05:00
Nick Sweeting 6189b1a8af fix invalid old config vars 2019-01-12 03:13:55 -05:00
Nick Sweeting f83750c545 cleanup options and make cli flags better for chrome headless timeouts 2019-01-11 22:38:50 -05:00
Nick Sweeting 911b84a79b make wget cli options easier to read 2019-01-11 22:13:51 -05:00
Nick Sweeting b77476cb83 fix user agent quoting 2019-01-11 19:54:43 -05:00
Nick Sweeting e8808b0a1f add WARC downloading 2019-01-11 07:02:49 -05:00
Nick Sweeting 74189966bf add link to media 2019-01-11 06:50:42 -05:00
Nick Sweeting a15a331798 fix media download with longer timeout 2019-01-11 06:33:35 -05:00
Nick Sweeting c33f7ba91c add ability to fetch media 2019-01-11 05:52:29 -05:00
Nick Sweeting 0df098717a make git domains configurable 2019-01-11 05:27:25 -05:00
Nick Sweeting 827e15b31a add git downloading 2019-01-11 05:18:49 -05:00
Nick Sweeting 57d42339a4 rename pip dir archive to archivebox 2018-12-31 20:53:01 -05:00
Renamed from archiver/archive_methods.py (Browse further)