homebr
- Arch:
yay -S archivebox (contributed by @imlonghao)
-- FreeBSD:
curl -sSL 'https://get.archivebox.io' | sh (uses pkg + pip3 under-the-hood)
+- FreeBSD:
curl -fsSL 'https://get.archivebox.io' | sh (uses pkg + pip3 under-the-hood)
- Nix:
nix-env --install archivebox (contributed by @siraben)
- Guix:
guix install archivebox (contributed by @rakino)
- More: contribute another distribution...!
@@ -458,14 +467,15 @@ ArchiveBox commands can be run in a terminal directly on your host, or via Docke
mkdir -p ~/archivebox/data # create a new data dir anywhere
cd ~/archivebox/data # IMPORTANT: cd into the directory
-# archivebox [subcommand] [--args]
+# archivebox [subcommand] [--help]
+archivebox version
archivebox help
-# equivalent: docker compose run archivebox [subcommand [--args]
+# equivalent: docker compose run archivebox [subcommand] [--help]
docker compose run archivebox help
-# equivalent: docker run -it -v $PWD:/data archivebox/archivebox [subcommand [--args]
- docker run -it -v $PWD:/data archivebox/archivebox help
+# equivalent: docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
+docker run -it -v $PWD:/data archivebox/archivebox help
```
#### ArchiveBox Subcommands
@@ -482,7 +492,7 @@ docker compose run archivebox help
# make sure you have pip-installed ArchiveBox and it's available in your $PATH first
-# archivebox [subcommand] [--args]
+# archivebox [subcommand] [--help]
archivebox init --setup # safe to run init multiple times (also how you update versions)
archivebox version # get archivebox version info + check dependencies
archivebox help # get list of archivebox subcommands that can be run
@@ -498,7 +508,7 @@ archivebox add --depth=1 'https://news.ycombinator.com'
# make sure you have `docker-compose.yml` from the Quickstart instructions first
-# docker compose run archivebox [subcommand [--args]
+# docker compose run archivebox [subcommand] [--help]
docker compose run archivebox init --setup
docker compose run archivebox version
docker compose run archivebox help
@@ -515,7 +525,7 @@ docker compose run archivebox add --depth=1 'https://news.ycombinator.com'
# make sure you create and cd into in a new empty directory first
-# docker run -it -v $PWD:/data archivebox/archivebox [subcommand [--args]
+# docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
docker run -v $PWD:/data -it archivebox/archivebox init --setup
docker run -v $PWD:/data -it archivebox/archivebox version
docker run -v $PWD:/data -it archivebox/archivebox help
@@ -675,7 +685,7 @@ It uses all available methods out-of-the-box, but you can disable extractors and
Expand to see the full list of ways it saves each page...
-./archive/{Snapshot.id}/
+data/archive/{Snapshot.id}/
- Index:
index.html & index.json HTML and JSON index files containing metadata and details
- Title, Favicon, Headers Response headers, site favicon, and parsed site title
@@ -806,18 +816,18 @@ All of ArchiveBox's state (SQLite DB, content, config, logs, etc.) is stored in
Expand to learn more about the layout of Archivebox's data on-disk...
-Data folders can be created anywhere (`~/archivebox` or `$PWD/data` as seen in our examples), and you can create as many data folders as you want to hold different collections.
+Data folders can be created anywhere (`~/archivebox/data` or `$PWD/data` as seen in our examples), and you can create as many data folders as you want to hold different collections.
All archivebox CLI commands are designed to be run from inside an ArchiveBox data folder, starting with archivebox init to initialize a new collection inside an empty directory.
-mkdir ~/archivebox && cd ~/archivebox # just an example, can be anywhere
+mkdir -p ~/archivebox/data && cd ~/archivebox/data # just an example, can be anywhere
archivebox init
-The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard index.sqlite3 database in the root of the data folder (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the ./archive/ subfolder.
+The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard index.sqlite3 database in the root of the data folder (it can also be exported as static JSON/HTML), and the archive snapshots are organized by date-added timestamp in the data/archive/ subfolder.
-/data/
+data/
index.sqlite3
ArchiveBox.conf
archive/
@@ -832,7 +842,7 @@ The on-disk layout is optimized to be easy to browse by hand and durable long-te
...
-Each snapshot subfolder ./archive/TIMESTAMP/ includes a static index.json and index.html describing its contents, and the snapshot extractor outputs are plain files within the folder.
+Each snapshot subfolder data/archive/TIMESTAMP/ includes a static index.json and index.html describing its contents, and the snapshot extractor outputs are plain files within the folder.
Learn More
@@ -1046,9 +1056,9 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
Disk usage can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by turning off extractors methods you don't need. You can also deduplicate content with a tool like [fdupes](https://github.com/adrianlopezroche/fdupes) or [rdfind](https://github.com/pauldreik/rdfind).
-**Don't store large collections on older filesystems like EXT3/FAT** as they may not be able to handle more than 50k directory entries in the `archive/` folder.
+**Don't store large collections on older filesystems like EXT3/FAT** as they may not be able to handle more than 50k directory entries in the `data/archive/` folder.
-**Try to keep the `index.sqlite3` file on local drive (not a network mount)** or SSD for maximum performance, however the `archive/` folder can be on a network mount or slower HDD.
+**Try to keep the `data/index.sqlite3` file on local drive (not a network mount)** or SSD for maximum performance, however the `data/archive/` folder can be on a network mount or slower HDD.
If using Docker or NFS/SMB/FUSE for the `data/archive/` folder, you may need to set [`PUID` & `PGID`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid) and [disable `root_squash`](https://github.com/ArchiveBox/ArchiveBox/issues/1304) on your fileshare server.
@@ -1439,7 +1449,7 @@ https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-dj
ArchiveBox [`extractors`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/extractors/media.py) are external binaries or Python/Node scripts that ArchiveBox runs to archive content on a page.
-Extractors take the URL of a page to archive, write their output to the filesystem `archive/TIMESTAMP/EXTRACTOR/...`, and return an [`ArchiveResult`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/models.py#:~:text=return%20qs-,class%20ArchiveResult,-(models.Model)%3A) entry which is saved to the database (visible on the `Log` page in the UI).
+Extractors take the URL of a page to archive, write their output to the filesystem `data/archive/TIMESTAMP/EXTRACTOR/...`, and return an [`ArchiveResult`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/core/models.py#:~:text=return%20qs-,class%20ArchiveResult,-(models.Model)%3A) entry which is saved to the database (visible on the `Log` page in the UI).
*Check out how we added **[`archivebox/extractors/singlefile.py`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/extractors/singlefile.py)** as an example of the process: [Issue #399](https://github.com/ArchiveBox/ArchiveBox/issues/399) + [PR #403](https://github.com/ArchiveBox/ArchiveBox/pull/403).*
@@ -25,7 +25,6 @@ curl -sSL 'https://get.archivebox.io' | sh # (or see pip/brew/Docker instruct
Without active preservation effort, everything on the internet eventually dissapears or degrades. Archive.org does a great job as a centralized service, but saved URLs have to be public, and they can't save every type of content.
*ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...*
-