Update README.md to fix trailing words
This commit is contained in:
Nick Sweeting
GitHub
parent
bd19b794e5
commit
d936b9eb8a
149
README.md
149
README.md
|
|
@ -25,23 +25,25 @@ Without active preservation effort, everything on the internet eventually dissap
|
|||
|
||||
*ArchiveBox is an open source tool that helps organizations and individuals archive web content and retain control over their data: save copies of browser bookmarks, preserve evidence for legal cases, backup photos from FB / Insta / Flickr, download your media from YT / Soundcloud / etc., snapshot research papers & academic citations, and more...*
|
||||
|
||||
> ➡️ *Use ArchiveBox on [Linux](#quickstart)/[macOS](#quickstart)/[Windows](#quickstart)/[Docker](#quickstart) as a [CLI tool](#usage), [self-hosted Web App](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive), [`pip` library](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage), or [one-off command](#static-archive-exporting).*
|
||||
> ➡️ *ArchiveBox is available on [Linux](#quickstart)/[macOS](#quickstart)/[Windows](#quickstart)/[Docker](#quickstart) as a [CLI tool](#usage), [self-hosted Web App](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive), [`pip` library](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage), or [one-off command](#static-archive-exporting).*
|
||||
|
||||
<hr/>
|
||||
|
||||
📥 **You can feed ArchiveBox URLs one at a time, or schedule regular imports** from your bookmarks or history, social media feeds or RSS, link-saving services like Pocket/Pinboard, our [Browser Extension](https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj), and more. See <a href="#input-formats">Input Formats</a> for a full list.
|
||||
📥 **You can feed ArchiveBox URLs one at a time, or schedule regular imports** from your bookmarks or history, social media feeds or RSS, link-saving services like Pocket/Pinboard, our [Browser Extension](https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj), and more.
|
||||
See <a href="#input-formats">Input Formats</a> for a full list...
|
||||
|
||||
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/90f1ce3c-75bb-401d-88ed-6297694b76ae" alt="snapshot detail page" align="right" width="190px" style="float: right"/>
|
||||
|
||||
**It saves snapshots of the URLs you feed it in several redundant formats.**
|
||||
It also detects any content featured *inside* each webpage & extracts it out into a folder:
|
||||
It also detects any content featured *inside* pages & extracts it out into a folder:
|
||||
- 🌐 **HTML**/**Any websites** ➡️ `original HTML+CSS+JS`, `singlefile HTML`, `screenshot PNG`, `PDF`, `WARC`, ...
|
||||
- 🎥 **Social Media**/**News** ➡️ `post content TXT`, `comments`, `title`, `author`, `images`
|
||||
- 🎬 **YouTube**/**SoundCloud**/etc. ➡️ `MP3/MP4`s, `subtitles`, `metadata`, `thumbnail`, ...
|
||||
- 💾 **Github**/**Gitlab**/etc. links ➡️ `clone of GIT source code`, `README`, `images`, ...
|
||||
- ✨ *and more, see [Output Formats](#output-formats) below...*
|
||||
|
||||
It uses [standard tools](#dependencies) like Chrome, `wget`, & `yt-dlp`, and stores data in ordinary [files & folders](#archive-layout) (no complex proprietary formats).
|
||||
It uses [standard tools](#dependencies) like Chrome, `wget`, & `yt-dlp`, and stores data in ordinary [files & folders](#archive-layout).
|
||||
*(no complex proprietary formats)*
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -59,7 +61,7 @@ The goal is to sleep soundly knowing the part of the internet you care about wil
|
|||
|
||||
<br/>
|
||||
|
||||
**📦 Install ArchiveBox using your preferred method: `docker` / `pip` / `apt` / `brew` / etc. ([see full Quickstart below](#quickstart)).**
|
||||
**📦 Install ArchiveBox using your preferred method: `docker` / `pip` / `apt` / etc. ([see full Quickstart](#quickstart)).**
|
||||
|
||||
|
||||
<details>
|
||||
|
|
@ -116,7 +118,7 @@ curl -sSL 'https://get.archivebox.io' | sh
|
|||
- [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (yt-dlp), articles (readability), code (git), etc.](#output-formats)
|
||||
- [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
|
||||
- [**Uses standard, durable, long-term formats**](#output-formats) like HTML, JSON, PDF, PNG, MP4, TXT, and WARC
|
||||
- [**Usable as a oneshot CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/latest/modules.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox) (ALPHA)
|
||||
- [**Usable as a oneshot CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/latest/modules.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox)
|
||||
- [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_archive_dot_org) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) for local-only mode)
|
||||
- Advanced users: support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
|
||||
- Planned: support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [autoscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
|
||||
|
|
@ -128,13 +130,13 @@ curl -sSL 'https://get.archivebox.io' | sh
|
|||
ArchiveBox is free for everyone to self-host, but we also provide support, security review, and custom integrations to help NGOs, governments, and other organizations [run ArchiveBox professionally](https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102):
|
||||
|
||||
- 🗞️ **Journalists:**
|
||||
`crawling and collecting research`, `preserving quoted material`, `fact-checking and review`
|
||||
`crawling during research`, `preserving cited pages`, `fact-checking & review`
|
||||
- ⚖️ **Lawyers:**
|
||||
`collecting & preserving evidence`, `hashing / integrity checking / chain-of-custody`, `tagging & review`
|
||||
`collecting & preserving evidence`, `detecting changes`, `tagging & review`
|
||||
- 🔬 **Researchers:**
|
||||
`analyzing social media trends`, `collecting LLM training data`, `crawling to feed other pipelines`
|
||||
`analyzing social media trends`, `getting LLM training sets`, `crawling pipelines`
|
||||
- 👩🏽 **Individuals:**
|
||||
`saving legacy social media / memoirs`, `preserving portfolios / resume`, `backing up news articles`
|
||||
`saving bookmarks`, `preserving portfolio content`, `legacy / memoirs archival`
|
||||
|
||||
> ***[Contact our team](https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102)** if your institution/org wants to use ArchiveBox professionally.*
|
||||
>
|
||||
|
|
@ -154,7 +156,7 @@ ArchiveBox is free for everyone to self-host, but we also provide support, secur
|
|||
|
||||
# Quickstart
|
||||
|
||||
**🖥 Supported OSs:** Linux/BSD, macOS, Windows (Docker) **👾 CPUs:** `amd64` (`x86_64`), `arm64` (`arm8`), `arm7` <sup>(raspi>=3)</sup><br/>
|
||||
**🖥 Supported OSs:** Linux/BSD, macOS, Windows (Docker) **👾 CPUs:** `amd64` (`x86_64`), `arm64`, `arm7` <sup>(raspi>=3)</sup><br/>
|
||||
<sub>Note: On `arm7` the `playwright` package is not available, so `chromium` must be installed manually if needed.</sub>
|
||||
|
||||
<br/>
|
||||
|
|
@ -432,7 +434,7 @@ For more discussion on managed and paid hosting options see here: <a href="https
|
|||
#### ➡️ Next Steps
|
||||
|
||||
- Import URLs from some of the supported [Input Formats](#input-formats) or view the supported [Output Formats](#output-formats)...
|
||||
- Tweak your UI or archiving behavior [Configuration](#configuration) or read about some of the [Caveats](#caveats) and troubleshooting steps...
|
||||
- Tweak your UI or archiving behavior [Configuration](#configuration), read about some of the [Caveats](#caveats), or [Troubleshoot](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
|
||||
- Read about the [Dependencies](#dependencies) used for archiving, the [Upgrading Process](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives), or the [Archive Layout](#archive-layout) on disk...
|
||||
- Or check out our full [Documentation](#documentation) or [Community Wiki](#internet-archiving-ecosystem)...
|
||||
|
||||
|
|
@ -442,7 +444,8 @@ For more discussion on managed and paid hosting options see here: <a href="https
|
|||
|
||||
#### ⚡️ CLI Usage
|
||||
|
||||
ArchiveBox commands can be run in a terminal directly on your host, or via Docker/Docker Compose (depending on install method).
|
||||
ArchiveBox commands can be run in a terminal directly on your host, or via Docker/Docker Compose.
|
||||
<sup>(depending on how you chose to install it)</sup>
|
||||
|
||||
```bash
|
||||
mkdir -p ~/archivebox/data # create a new data dir anywhere
|
||||
|
|
@ -601,15 +604,20 @@ docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://exampl
|
|||
## Input Formats: How to pass URLs into ArchiveBox for saving
|
||||
|
||||
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>The official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a> (provides realtime archiving from Chrome/Chromium/Firefox browsers)</b>
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>The official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a>
|
||||
<sup>Provides realtime archiving from Chrome/Chromium/Firefox browsers</sup>
|
||||
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> Manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, or [any other text-based format...](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file)
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> Manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown
|
||||
<sup>ArchiveBox supports injecting URLs in [any other text-based format...](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file)</sup>
|
||||
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/4f7bd318-265c-4235-ad25-38be89946b12" height="22px"/> [MITM Proxy](https://mitmproxy.org/) archiving with [`archivebox-proxy`](https://github.com/ArchiveBox/archivebox-proxy) ([realtime archiving](https://github.com/ArchiveBox/ArchiveBox/issues/577) of all traffic from any device going through the proxy)
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/4f7bd318-265c-4235-ad25-38be89946b12" height="22px"/> [MITM Proxy](https://mitmproxy.org/) archiving with [`archivebox-proxy`](https://github.com/ArchiveBox/archivebox-proxy)
|
||||
<sup>Provides [realtime archiving](https://github.com/ArchiveBox/ArchiveBox/issues/577) of all traffic from any device going through the proxy.</sup>
|
||||
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/32b494e6-4de1-4984-8d88-dc02f18e5c34" height="22px"/> Exported [browser history](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) or [browser bookmarks](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) (see instructions for: [Chrome](https://support.google.com/chrome/answer/96816?hl=en), [Firefox](https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer), [Safari](https://github.com/ArchiveBox/ArchiveBox/assets/511499/24ad068e-0fa6-41f4-a7ff-4c26fc91f71a), [IE](https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows), [Opera](https://help.opera.com/en/latest/features/#bookmarks:~:text=Click%20the%20import/-,export%20button,-on%20the%20bottom), [and more...](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive))
|
||||
- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/32b494e6-4de1-4984-8d88-dc02f18e5c34" height="22px"/> Exported [browser history](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) or [browser bookmarks](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
|
||||
<sup>See instructions for: <a href="https://support.google.com/chrome/answer/96816?hl=en">Chrome</a>, <a href="https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer">Firefox</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/assets/511499/24ad068e-0fa6-41f4-a7ff-4c26fc91f71a">Safari</a>, <a href="https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows">IE</a>, <a href="https://help.opera.com/en/latest/features/#bookmarks:~:text=Click%20the%20import/-,export%20button,-on%20the%20bottom">Opera</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a></sup>
|
||||
|
||||
- <img src="https://getpocket.com/favicon.ico" height="22px"/> Links from [Pocket](https://getpocket.com/export), [Pinboard](https://pinboard.in/export/), [Instapaper](https://www.instapaper.com/user), [Shaarli](https://shaarli.readthedocs.io/en/master/Usage/#importexport), [Delicious](https://www.groovypost.com/howto/howto/export-delicious-bookmarks-xml/), [Reddit Saved](https://github.com/csu/export-saved-reddit), [Wallabag](https://doc.wallabag.org/en/user/import/wallabagv2.html), [Unmark.it](http://help.unmark.it/import-export), [OneTab](https://www.addictivetips.com/web/onetab-save-close-all-chrome-tabs-to-restore-export-or-import/), [Firefox Sync](https://github.com/ArchiveBox/ArchiveBox/issues/648), [and more...](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
|
||||
- <img src="https://getpocket.com/favicon.ico" height="22px"/> Links exported from Bookmarks on social media sites or Brookmarking services
|
||||
<sup>See instructions for: <a href="https://getpocket.com/export">Pocket</a>, <a href="https://pinboard.in/export/">Pinboard</a>, <a href="https://www.instapaper.com/user">Instapaper</a>, <a href="https://shaarli.readthedocs.io/en/master/Usage/#importexport">Shaarli</a>, <a href="https://www.groovypost.com/howto/howto/export-delicious-bookmarks-xml/">Delicious</a>, <a href="https://github.com/csu/export-saved-reddit">Reddit Saved</a>, <a href="https://doc.wallabag.org/en/user/import/wallabagv2.html">Wallabag</a>, <a href="http://help.unmark.it/import-export">Unmark.it</a>, <a href="https://www.addictivetips.com/web/onetab-save-close-all-chrome-tabs-to-restore-export-or-import/">OneTab</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/issues/648">Firefox Sync</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a></sup>
|
||||
|
||||
|
||||
|
||||
|
|
@ -650,7 +658,7 @@ It uses all available methods out-of-the-box, but you can disable extractors and
|
|||
|
||||
<br/>
|
||||
<details>
|
||||
<summary><i>Expand to see the full list of ways ArchiveBox saves each page...</i></summary>
|
||||
<summary><i>Expand to see the full list of ways it saves each page...</i></summary>
|
||||
|
||||
|
||||
<code>./archive/{Snapshot.id}/</code><br/>
|
||||
|
|
@ -677,7 +685,7 @@ It uses all available methods out-of-the-box, but you can disable extractors and
|
|||
|
||||
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ea672e6b-4df5-49d8-b550-7f450951fd27" width="330px" align="right" style="float: right"/>
|
||||
|
||||
ArchiveBox can be configured via environment variables, by using the `archivebox config` CLI, or by editing `./ArchiveBox.conf` directly.
|
||||
ArchiveBox can be configured via environment variables, by using the `archivebox config` CLI, or by editing `./ArchiveBox.conf`.
|
||||
<br/>
|
||||
<details>
|
||||
<summary><i>Expand to see examples...</i></summary>
|
||||
|
|
@ -722,8 +730,9 @@ CURL_USER_AGENT="Mozilla/5.0 ..."
|
|||
|
||||
To achieve high-fidelity archives in as many situations as possible, ArchiveBox depends on a variety of 3rd-party libraries and tools that specialize in extracting different types of content.
|
||||
|
||||
> Under-the-hood, ArchiveBox uses [Django](https://www.djangoproject.com/start/overview/) to power its [Web UI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage) and [SQlite](https://www.sqlite.org/locrsf.html) + the filesystem to provide [fast & durable metadata storage](https://www.sqlite.org/locrsf.html) w/ [determinisitc upgrades](https://stackoverflow.com/a/39976321/2156113). ArchiveBox bundles industry-standard tools like [Google Chrome](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install), [`wget`, `yt-dlp`, `readability`, etc.](#dependencies) internally, and its operation can be [tuned, secured, and extended](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) as-needed for many different applications.
|
||||
> Under-the-hood, ArchiveBox uses [Django](https://www.djangoproject.com/start/overview/) to power its [Web UI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage) and [SQlite](https://www.sqlite.org/locrsf.html) + the filesystem to provide [fast & durable metadata storage](https://www.sqlite.org/locrsf.html) w/ [determinisitc upgrades](https://stackoverflow.com/a/39976321/2156113).
|
||||
|
||||
For the actual archiving, ArchiveBox bundles industry-standard tools like [Google Chrome](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install), [`wget`, `yt-dlp`, `readability`, etc.](#dependencies) internally, and its operation can be [tuned, secured, and extended](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration) as-needed for many different applications.
|
||||
|
||||
<br/>
|
||||
<details>
|
||||
|
|
@ -774,7 +783,7 @@ Installing directly on **Windows without Docker or WSL/WSL2/Cygwin is not offici
|
|||
|
||||
## Archive Layout
|
||||
|
||||
All of ArchiveBox's state (SQLite DB, archived assets, config, logs, etc.) is stored in a single folder called the "ArchiveBox Data Folder".
|
||||
All of ArchiveBox's state (SQLite DB, archived assets, config, logs, etc.) is stored in a single folder (`data/`).
|
||||
|
||||
<br/>
|
||||
<details>
|
||||
|
|
@ -823,17 +832,17 @@ Each snapshot subfolder <code>./archive/TIMESTAMP/</code> includes a static <cod
|
|||
|
||||
## Static Archive Exporting
|
||||
|
||||
You can create one-off archives with `archivebox oneshot`, or export your index as static HTML with `archivebox list` (so you can view it without an ArchiveBox server).
|
||||
You can create one-off archives of individual URLs with `archivebox oneshot`, or export your index as static HTML using `archivebox list` (so you can view it without an ArchiveBox server).
|
||||
|
||||
<br/>
|
||||
<details>
|
||||
<summary><i>Expand to learn how to export your ArchiveBox collection...</i></summary><br/>
|
||||
|
||||
<blockquote>
|
||||
<p><em>NOTE: These exports are not paginated, exporting many URLs or the entire archive at once may be slow. Use the filtering CLI flags on the <code>archivebox list</code> command to export specific Snapshots or ranges.</em></p>
|
||||
</blockquote>
|
||||
|
||||
> *NOTE: These exports are not paginated, exporting many URLs or the entire archive at once may be slow. Use the filtering CLI flags on the `archivebox list` command to export specific Snapshots or ranges.*
|
||||
|
||||
```bash|
|
||||
# do a one-off single URL archive wihout needing a data dir initialized
|
||||
<pre lang="bash"><code style="white-space: pre-line"># do a one-off single URL archive wihout needing a data dir initialized
|
||||
archivebox oneshot 'https://example.com'
|
||||
|
||||
# archivebox list --help
|
||||
|
|
@ -843,16 +852,17 @@ archivebox list --csv=timestamp,url,title > index.csv # export to csv spreadshe
|
|||
|
||||
# (if using Docker Compose, add the -T flag when piping)
|
||||
# docker compose run -T archivebox list --html 'https://example.com' > index.json
|
||||
```
|
||||
</code></pre>
|
||||
|
||||
The paths in the static exports are relative, make sure to keep them next to your `./archive` folder when backing them up or viewing them.
|
||||
|
||||
#### Learn More
|
||||
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#public_index--public_snapshots--public_add_view
|
||||
<h4>Learn More</h4>
|
||||
|
||||
<ul>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html">https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing">https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#public_index--public_snapshots--public_add_view">https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#public_index--public_snapshots--public_add_view</a></li>
|
||||
</ul>
|
||||
|
||||
</details>
|
||||
<br/>
|
||||
|
|
@ -876,8 +886,7 @@ If you're importing pages with private content or URLs containing secret tokens
|
|||
<summary><i>Expand to learn about privacy, permissions, and user accounts...</i></summary>
|
||||
|
||||
|
||||
```bash
|
||||
# don't save private content to ArchiveBox, e.g.:
|
||||
<pre lang="bash"><code style="white-space: pre-line"># don't save private content to ArchiveBox, e.g.:
|
||||
archivebox add 'https://docs.google.com/document/d/12345somePrivateDocument'
|
||||
archivebox add 'https://vimeo.com/somePrivateVideo'
|
||||
|
||||
|
|
@ -893,19 +902,22 @@ archivebox manage createsuperuser
|
|||
# if extra paranoid or anti-Google:
|
||||
archivebox config --set SAVE_FAVICON=False # disable favicon fetching (it calls a Google API passing the URL's domain part only)
|
||||
archivebox config --set CHROME_BINARY=chromium # ensure it's using Chromium instead of Chrome
|
||||
```
|
||||
</code></pre>
|
||||
|
||||
> *CAUTION: Assume anyone *viewing* your archives will be able to see any cookies, session tokens, or private URLs passed to ArchiveBox during archiving.*
|
||||
> *Make sure to secure your ArchiveBox data and don't share snapshots with others without stripping out sensitive headers and content first.*
|
||||
<blockquote>
|
||||
<p><em>CAUTION: Assume anyone <em>viewing</em> your archives will be able to see any cookies, session tokens, or private URLs passed to ArchiveBox during archiving.</em>
|
||||
<em>Make sure to secure your ArchiveBox data and don't share snapshots with others without stripping out sensitive headers and content first.</em></p>
|
||||
</blockquote>
|
||||
|
||||
#### Learn More
|
||||
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#cookies_file
|
||||
<h4>Learn More</h4>
|
||||
|
||||
<ul>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir">https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#cookies_file">https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#cookies_file</a></li>
|
||||
</ul>
|
||||
|
||||
</details>
|
||||
<br/>
|
||||
|
|
@ -921,28 +933,27 @@ Be aware that malicious archived JS can access the contents of other pages in yo
|
|||
<summary><i>Expand to see risks and mitigations...</i></summary>
|
||||
|
||||
|
||||
```bash
|
||||
# visiting an archived page with malicious JS:
|
||||
<pre lang="bash"><code style="white-space: pre-line"># visiting an archived page with malicious JS:
|
||||
https://127.0.0.1:8000/archive/1602401954/example.com/index.html
|
||||
|
||||
# example.com/index.js can now make a request to read everything from:
|
||||
https://127.0.0.1:8000/index.html
|
||||
https://127.0.0.1:8000/archive/*
|
||||
# then example.com/index.js can send it off to some evil server
|
||||
```
|
||||
</code></pre>
|
||||
|
||||
The admin UI is also served from the same origin as replayed JS, so malicious pages could also potentially use your ArchiveBox login cookies to perform admin actions (e.g. adding/removing links, running extractors, etc.). We are planning to fix this security shortcoming in a future version by using separate ports/origins to serve the Admin UI and archived content (see [Issue #239](https://github.com/ArchiveBox/ArchiveBox/issues/239)).
|
||||
|
||||
> *NOTE: Only the `wget` & `dom` extractor methods execute archived JS when viewing snapshots, all other archive methods produce static output that does not execute JS on viewing.*
|
||||
> *If you are worried about these issues ^ you should disable these extractors using `archivebox config --set SAVE_WGET=False SAVE_DOM=False`.*
|
||||
|
||||
#### Learn More
|
||||
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview
|
||||
- https://github.com/ArchiveBox/ArchiveBox/issues/239
|
||||
- https://github.com/ArchiveBox/ArchiveBox/security/advisories/GHSA-cr45-98w9-gwqx (`CVE-2023-45815`)
|
||||
- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing
|
||||
<blockquote>
|
||||
<p><em>NOTE: Only the <code>wget</code> & <code>dom</code> extractor methods execute archived JS when viewing snapshots, all other archive methods produce static output that does not execute JS on viewing.</em>
|
||||
<em>If you are worried about these issues ^ you should disable these extractors using <code>archivebox config --set SAVE_WGET=False SAVE_DOM=False</code>.</em></p>
|
||||
</blockquote>
|
||||
|
||||
<h4>Learn More</h4>
|
||||
<ul>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/issues/239">https://github.com/ArchiveBox/ArchiveBox/issues/239</a></li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/security/advisories/GHSA-cr45-98w9-gwqx">https://github.com/ArchiveBox/ArchiveBox/security/advisories/GHSA-cr45-98w9-gwqx</a> (<code>CVE-2023-45815</code>)</li>
|
||||
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing">https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing</a></li>
|
||||
</ul>
|
||||
|
||||
</details>
|
||||
<br/>
|
||||
|
|
@ -958,13 +969,14 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
|
|||
<br/>
|
||||
|
||||
|
||||
- Set [`CHROME_USER_AGENT`, `WGET_USER_AGENT`, `CURL_USER_AGENT`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent) to impersonate a real browser (instead of an ArchiveBox bot)
|
||||
- Set up a logged-in browser session for archiving using [`CHROME_DATA_DIR` & `COOKIES_FILE`](https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile)
|
||||
- Rewrite your URLs before archiving to swap in an alternative frontend thats more bot-friendly e.g.
|
||||
`reddit.com/some/url` -> `teddit.net/some/url`: https://github.com/mendel5/alternative-front-ends
|
||||
<ul>
|
||||
<li>Set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent"><code>CHROME_USER_AGENT</code>, <code>WGET_USER_AGENT</code>, <code>CURL_USER_AGENT</code></a> to impersonate a real browser (instead of an ArchiveBox bot)</li>
|
||||
<li>Set up a logged-in browser session for archiving using <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile"><code>CHROME_DATA_DIR</code> & <code>COOKIES_FILE</code></a></li>
|
||||
<li>Rewrite your URLs before archiving to swap in an alternative frontend thats more bot-friendly e.g.<br>
|
||||
<code>reddit.com/some/url</code> -> <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
|
||||
</ul>
|
||||
|
||||
|
||||
In the future we plan on adding support for running JS scripts during archiving to block ads, cookie popups, modals, and fix other issues. Follow here for progress: [Issue #51](https://github.com/ArchiveBox/ArchiveBox/issues/51).
|
||||
In the future we plan on adding support for running JS scripts during archiving to block ads, cookie popups, modals, and fix other issues. Follow here for progress: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/51">Issue #51</a>.
|
||||
|
||||
</details>
|
||||
<br/>
|
||||
|
|
@ -983,11 +995,10 @@ ArchiveBox appends a hash with the current date `https://example.com#2020-10-24`
|
|||
|
||||
Because ArchiveBox uniquely identifies snapshots by URL, it must use a workaround to take multiple snapshots of the same URL (otherwise they would show up as a single Snapshot entry). It makes the URLs of repeated snapshots unique by adding a hash with the archive date at the end:
|
||||
|
||||
```bash
|
||||
archivebox add 'https://example.com#2020-10-24'
|
||||
<pre lang="bash"><code style="white-space: pre-line">archivebox add 'https://example.com#2020-10-24'
|
||||
...
|
||||
archivebox add 'https://example.com#2020-10-25'
|
||||
```
|
||||
</code></pre>
|
||||
|
||||
The <img src="https://user-images.githubusercontent.com/511499/115942091-73c02300-a476-11eb-958e-5c1fc04da488.png" alt="Re-Snapshot Button" height="24px"/> button in the Admin UI is a shortcut for this hash-date multi-snapshotting workaround.
|
||||
|
||||
|
|
@ -1164,7 +1175,7 @@ ArchiveBox is neither the highest fidelity nor the simplest tool available for s
|
|||
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/78d8a725-97f4-47f5-b983-1f62843ddc51" width="14%" align="right" style="float: right"/>
|
||||
|
||||
<details>
|
||||
<summary><i>Our <b><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community">Community Wiki</a></b> strives to be a comprehensive index of the broader web archiving community...</i></summary>
|
||||
<summary><i>Our <b><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community">Community Wiki</a></b> strives to be a comprehensive index of the web archiving industry...</i></summary>
|
||||
<br/>
|
||||
|
||||
- [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
|
||||
|
|
|
|||
Loading…
Reference in a new issue