+**🖥 Supported OSs:** Linux/BSD, macOS, Windows (Docker) **👾 CPUs:** `amd64` (`x86_64`), `arm64`, `arm7` (raspi>=3)
Note: On `arm7` the `playwright` package is not available, so `chromium` must be installed manually if needed.
@@ -432,7 +434,7 @@ For more discussion on managed and paid hosting options see here: (depending on how you chose to install it) ```bash mkdir -p ~/archivebox/data # create a new data dir anywhere @@ -601,15 +604,20 @@ docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://exampl ## Input Formats: How to pass URLs into ArchiveBox for saving --
-
@@ -677,7 +685,7 @@ It uses all available methods out-of-the-box, but you can disable extractors and
-ArchiveBox can be configured via environment variables, by using the `archivebox config` CLI, or by editing `./ArchiveBox.conf` directly.
+ArchiveBox can be configured via environment variables, by using the `archivebox config` CLI, or by editing `./ArchiveBox.conf`.
@@ -958,13 +969,14 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
-- Set [`CHROME_USER_AGENT`, `WGET_USER_AGENT`, `CURL_USER_AGENT`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent) to impersonate a real browser (instead of an ArchiveBox bot) -- Set up a logged-in browser session for archiving using [`CHROME_DATA_DIR` & `COOKIES_FILE`](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile) -- Rewrite your URLs before archiving to swap in an alternative frontend thats more bot-friendly e.g. - `reddit.com/some/url` -> `teddit.net/some/url`: https://github.com/mendel5/alternative-front-ends +
@@ -983,11 +995,10 @@ ArchiveBox appends a hash with the current date `https://example.com#2020-10-24` Because ArchiveBox uniquely identifies snapshots by URL, it must use a workaround to take multiple snapshots of the same URL (otherwise they would show up as a single Snapshot entry). It makes the URLs of repeated snapshots unique by adding a hash with the archive date at the end: -```bash -archivebox add 'https://example.com#2020-10-24' +
button in the Admin UI is a shortcut for this hash-date multi-snapshotting workaround.
@@ -1164,7 +1175,7 @@ ArchiveBox is neither the highest fidelity nor the simplest tool available for s
Expand to see the full list of ways ArchiveBox saves each page...
+Expand to see the full list of ways it saves each page...
./archive/{Snapshot.id}/@@ -677,7 +685,7 @@ It uses all available methods out-of-the-box, but you can disable extractors and
Expand to see examples...
@@ -722,8 +730,9 @@ CURL_USER_AGENT="Mozilla/5.0 ..." To achieve high-fidelity archives in as many situations as possible, ArchiveBox depends on a variety of 3rd-party libraries and tools that specialize in extracting different types of content. -> Under-the-hood, ArchiveBox uses [Django](https://www.djangoproject.com/start/overview/) to power its [Web UI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage) and [SQlite](https://www.sqlite.org/locrsf.html) + the filesystem to provide [fast & durable metadata storage](https://www.sqlite.org/locrsf.html) w/ [determinisitc upgrades](https://stackoverflow.com/a/39976321/2156113). ArchiveBox bundles industry-standard tools like [Google Chrome](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install), [`wget`, `yt-dlp`, `readability`, etc.](#dependencies) internally, and its operation can be [tuned, secured, and extended](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) as-needed for many different applications. +> Under-the-hood, ArchiveBox uses [Django](https://www.djangoproject.com/start/overview/) to power its [Web UI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage) and [SQlite](https://www.sqlite.org/locrsf.html) + the filesystem to provide [fast & durable metadata storage](https://www.sqlite.org/locrsf.html) w/ [determinisitc upgrades](https://stackoverflow.com/a/39976321/2156113). +For the actual archiving, ArchiveBox bundles industry-standard tools like [Google Chrome](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install), [`wget`, `yt-dlp`, `readability`, etc.](#dependencies) internally, and its operation can be [tuned, secured, and extended](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) as-needed for many different applications.
@@ -774,7 +783,7 @@ Installing directly on **Windows without Docker or WSL/WSL2/Cygwin is not offici
## Archive Layout
-All of ArchiveBox's state (SQLite DB, archived assets, config, logs, etc.) is stored in a single folder called the "ArchiveBox Data Folder".
+All of ArchiveBox's state (SQLite DB, archived assets, config, logs, etc.) is stored in a single folder (`data/`).
@@ -921,28 +933,27 @@ Be aware that malicious archived JS can access the contents of other pages in yo
@@ -823,17 +832,17 @@ Each snapshot subfolder
+
@@ -876,8 +886,7 @@ If you're importing pages with private content or URLs containing secret tokens
./archive/TIMESTAMP/ includes a static Expand to learn how to export your ArchiveBox collection...
+
+-> *NOTE: These exports are not paginated, exporting many URLs or the entire archive at once may be slow. Use the filtering CLI flags on the `archivebox list` command to export specific Snapshots or ranges.* - -```bash| -# do a one-off single URL archive wihout needing a data dir initialized +NOTE: These exports are not paginated, exporting many URLs or the entire archive at once may be slow. Use the filtering CLI flags on the
+archivebox listcommand to export specific Snapshots or ranges.
# do a one-off single URL archive wihout needing a data dir initialized
archivebox oneshot 'https://example.com'
# archivebox list --help
@@ -843,16 +852,17 @@ archivebox list --csv=timestamp,url,title > index.csv # export to csv spreadshe
# (if using Docker Compose, add the -T flag when piping)
# docker compose run -T archivebox list --html 'https://example.com' > index.json
-```
+Learn More
+@@ -876,8 +886,7 @@ If you're importing pages with private content or URLs containing secret tokens
Expand to learn about privacy, permissions, and user accounts...
-```bash -# don't save private content to ArchiveBox, e.g.: +# don't save private content to ArchiveBox, e.g.:
archivebox add 'https://docs.google.com/document/d/12345somePrivateDocument'
archivebox add 'https://vimeo.com/somePrivateVideo'
@@ -893,19 +902,22 @@ archivebox manage createsuperuser
# if extra paranoid or anti-Google:
archivebox config --set SAVE_FAVICON=False # disable favicon fetching (it calls a Google API passing the URL's domain part only)
archivebox config --set CHROME_BINARY=chromium # ensure it's using Chromium instead of Chrome
-```
++-#### Learn More - -- https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive -- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview -- https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile -- https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir -- https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#cookies_file +CAUTION: Assume anyone viewing your archives will be able to see any cookies, session tokens, or private URLs passed to ArchiveBox during archiving. +Make sure to secure your ArchiveBox data and don't share snapshots with others without stripping out sensitive headers and content first.
+
Learn More
+-
+
- https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive +
- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview +
- https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile +
- https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir +
- https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#cookies_file +
@@ -921,28 +933,27 @@ Be aware that malicious archived JS can access the contents of other pages in yo
Expand to see risks and mitigations...
-```bash -# visiting an archived page with malicious JS: +# visiting an archived page with malicious JS:
https://127.0.0.1:8000/archive/1602401954/example.com/index.html
# example.com/index.js can now make a request to read everything from:
https://127.0.0.1:8000/index.html
https://127.0.0.1:8000/archive/*
# then example.com/index.js can send it off to some evil server
-```
+++NOTE: Only the
+wget&domextractor methods execute archived JS when viewing snapshots, all other archive methods produce static output that does not execute JS on viewing. +If you are worried about these issues ^ you should disable these extractors usingarchivebox config --set SAVE_WGET=False SAVE_DOM=False.
Learn More
+@@ -958,13 +969,14 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
-- Set [`CHROME_USER_AGENT`, `WGET_USER_AGENT`, `CURL_USER_AGENT`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent) to impersonate a real browser (instead of an ArchiveBox bot) -- Set up a logged-in browser session for archiving using [`CHROME_DATA_DIR` & `COOKIES_FILE`](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile) -- Rewrite your URLs before archiving to swap in an alternative frontend thats more bot-friendly e.g. - `reddit.com/some/url` -> `teddit.net/some/url`: https://github.com/mendel5/alternative-front-ends +
-
+
- Set
CHROME_USER_AGENT,WGET_USER_AGENT,CURL_USER_AGENTto impersonate a real browser (instead of an ArchiveBox bot)
+ - Set up a logged-in browser session for archiving using
CHROME_DATA_DIR&COOKIES_FILE
+ - Rewrite your URLs before archiving to swap in an alternative frontend thats more bot-friendly e.g.
+reddit.com/some/url->teddit.net/some/url: https://github.com/mendel5/alternative-front-ends
+
@@ -983,11 +995,10 @@ ArchiveBox appends a hash with the current date `https://example.com#2020-10-24` Because ArchiveBox uniquely identifies snapshots by URL, it must use a workaround to take multiple snapshots of the same URL (otherwise they would show up as a single Snapshot entry). It makes the URLs of repeated snapshots unique by adding a hash with the archive date at the end: -```bash -archivebox add 'https://example.com#2020-10-24' +
archivebox add 'https://example.com#2020-10-24'
...
archivebox add 'https://example.com#2020-10-25'
-```
+
-
- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
Our Community Wiki strives to be a comprehensive index of the broader web archiving community...
+Our Community Wiki strives to be a comprehensive index of the web archiving industry...
- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)