1
0
Fork 0
mirror of synced 2024-06-20 19:30:15 +12:00
Commit graph

3136 commits

Author SHA1 Message Date
Nick Sweeting a680724367
Merge branch 'dev' into search_index_extract_html_text 2023-10-27 23:09:28 -07:00
Nick Sweeting 720061185c
Merge pull request #1195 from overhacked/method_allow_deny 2023-10-27 23:05:30 -07:00
Nick Sweeting 7a80b61bea
Merge pull request #1252 from c01o/patch-1 2023-10-25 03:28:42 -07:00
c01o 08cffb8742
Fix docker build
Lack of `fi` resulted in a syntax error.
Also, change `;` to `||` to ensure it builds successfully on architectures other than `linux/arm/v7`.


```
[2023-10-24T10:46:49.941Z] ------
 > [dev_container_auto_added_stage_label 11/18] RUN mkdir -p "/home/archivebox/.config/chromium/Crash Reports/pending/"     && chown -R archivebox "/home/archivebox/.config"     ; if [[ "linux/amd64" == "linux/arm/v7" ]]; then $exit 0; else exit 1:
0.401 /bin/bash: -c: line 2: syntax error: unexpected end of file
------
WARNING: buildx: git was not found in the system. Current commit information was not captured by the build
[2023-10-24T10:46:49.942Z] Dockerfile-with-features:135
--------------------
 134 |         && ln -s "$CHROME_BINARY" /usr/bin/chromium-browser
 135 | >>> RUN mkdir -p "/home/${ARCHIVEBOX_USER}/.config/chromium/Crash Reports/pending/" \
 136 | >>>     && chown -R $ARCHIVEBOX_USER "/home/${ARCHIVEBOX_USER}/.config" \
 137 | >>>     ; if [[ "$TARGETPLATFORM" == "linux/arm/v7" ]]; then $exit 0; else exit 1
 138 |         # ignore failure for architectures where no playwright release is available yet
--------------------
ERROR: failed to solve: process "/bin/bash -c mkdir -p \"/home/${ARCHIVEBOX_USER}/.config/chromium/Crash Reports/pending/\"     && chown -R $ARCHIVEBOX_USER \"/home/${ARCHIVEBOX_USER}/.config\"     ; if [[ \"$TARGETPLATFORM\" == \"linux/arm/v7\" ]]; then $exit 0; else exit 1" did not complete successfully: exit code: 2
```
2023-10-24 19:58:23 +09:00
Ross Williams 310b4d1242 Add htmltotext extractor
Saves HTML text nodes and selected element attributes in
`htmltotext.txt` for each Snapshot. Primarily intended to be used
for search indexing.
2023-10-23 21:42:32 -04:00
Nick Sweeting 2681f752f1
ignore playwright chrome install on arm v7 2023-10-21 14:17:50 -07:00
Nick Sweeting 2928c19119 fix multiarch chromium install 2023-10-20 05:45:08 -07:00
Nick Sweeting 988b6585c9 better multiarch builds 2023-10-20 05:30:45 -07:00
Nick Sweeting 25d2c17bb1 add dev build helper script 2023-10-20 05:10:18 -07:00
Nick Sweeting 015737b1e4 fix buildx helper script 2023-10-20 05:10:03 -07:00
Nick Sweeting bf7a6e4c09 add missing cpython packages needed for arm v8 docker build 2023-10-20 04:33:26 -07:00
Nick Sweeting 63ad43f46c
Merge branch 'dev' into method_allow_deny 2023-10-20 04:25:44 -07:00
Nick Sweeting a58535baff
Merge pull request #1249 from ArchiveBox/pdm 2023-10-20 04:23:25 -07:00
Nick Sweeting 82d8662c74 add more readability error output 2023-10-20 04:14:28 -07:00
Nick Sweeting f3e81c1487 bump npm lockfile 2023-10-20 04:09:03 -07:00
Nick Sweeting 76f9b91ed3 dockerfile fixes 2023-10-20 04:08:38 -07:00
Nick Sweeting 6736e63ef0 new Dockerfile layout with better layering 2023-10-20 02:47:34 -07:00
Nick Sweeting 86c662421b show FS_USER in version output debug string 2023-10-20 02:47:23 -07:00
Nick Sweeting 7910a5b34b fix pretty_path printing logic 2023-10-20 02:47:03 -07:00
Nick Sweeting d4ca2d1154 update mercury bin path to postlight parser 2023-10-20 02:46:41 -07:00
Nick Sweeting 75eeb12ebe dont install youtubedl anymore 2023-10-20 02:46:09 -07:00
Nick Sweeting 16796a63fd share PUID and PGID with child procs from entrypoint 2023-10-20 02:45:54 -07:00
Nick Sweeting e0e34e6377 update package.json and npm lockfile 2023-10-20 02:45:37 -07:00
Nick Sweeting 3b3bdab97d update ignore files 2023-10-20 02:45:20 -07:00
Nick Sweeting 63c276a93d redirect add page back to snapshots list automatically 2023-10-19 18:24:19 -07:00
Nick Sweeting 22bcffe5ee remove accidental duplicate template code 2023-10-19 18:24:05 -07:00
Nick Sweeting 85f8583d62 catch ldap configuration and packaging errors and make them non fatal 2023-10-19 18:05:59 -07:00
Nick Sweeting 53cff45ec0 switch from pipenv and setup.py to PDM for python packaging 2023-10-19 17:49:06 -07:00
Nick Sweeting 8caffc4b4e ignore .pdm-python file 2023-10-19 17:09:48 -07:00
Nick Sweeting b6113cc00c use pypa build instead of setup.py build and move ldap to extras 2023-10-19 16:44:48 -07:00
Nick Sweeting 4b26ab9d1f only load image into docker when building, and push on deploy 2023-10-19 16:29:51 -07:00
Nick Sweeting 9fa210515e
Bump required python version to 3.9 in stdeb.cfg 2023-10-19 16:22:06 -07:00
Nick Sweeting 748363e685
Bump required Python version to 3.9 2023-10-19 16:21:32 -07:00
Nick Sweeting 6619e8005e
Update pip.yml github action python version 2023-10-19 16:20:43 -07:00
Nick Sweeting a61f544547
Update SECURITY.md 2023-10-19 16:18:32 -07:00
Nick Sweeting edde40898b
fix markdown underlining everything as link 2023-10-19 12:58:38 -07:00
Nick Sweeting a8ce927937
fix markdown formatting 2023-10-19 12:57:05 -07:00
Nick Sweeting 0e3475a0eb
Create SECURITY.md policy 2023-10-18 18:40:58 -07:00
Nick Sweeting 4f655fc4a1
Replace chown of entire data directory when ownership mismatch detected 2023-10-18 18:24:43 -07:00
Nick Sweeting e9c85aa812
Update README.md 2023-10-18 17:53:49 -07:00
Nick Sweeting 6f4a7e4e68
Add ADMIN_USERNAME and ADMIN_PASSWORD to example options 2023-10-18 11:53:13 -07:00
Nick Sweeting 2a1104c76a
Merge pull request #1245 from benmuth/add-user-env-vars 2023-10-18 11:50:58 -07:00
Nick Sweeting d286dca925
better to log before doing a thing than after 2023-10-18 11:47:55 -07:00
Ben Muthalaly 9e6a87114b Fix formatting, logging, logic issues 2023-10-18 12:07:36 -05:00
Ben Muthalaly 521ea70e0c Add check for existing user, change varable names 2023-10-18 03:07:54 -05:00
Ross Williams 6555719489 Add space after tags when extracting text
Add space after any close tag to ensure that
tokens that would be rendered separate in HTML
get extracted as separate tokens in text.

Example:

`<p>First</p><p>Second</p>` --> `First Second`
NOT `FirstSecond`
2023-10-16 09:59:08 -04:00
Ben Muthalaly 44a94157be Remove logging of configured username and password 2023-10-15 23:36:47 -05:00
Ben Muthalaly 11d473e536 Add config options to add admin user on first run 2023-10-14 00:38:04 -05:00
Ross Williams d8aa84ac98 Make extracting text for indexing optional
Add a configuration option to enable/disable HTML text extraction
for indexing
2023-10-12 13:14:39 -04:00
Ross Williams b6a20c962a Extract text from singlefile.html when indexing
singlefile.html contains a lot of large strings in the form of `data:`
URLs, which can be unnecessarily stored in full-text indices. Also,
large chunks of JavaScript shouldn't be indexed, either, as they pollute
search results for searches about JS functions, etc.

This commit takes a blanket approach of parsing singlefile.html as it is
read and only outputting text and selected textual attributes (like
`alt`) for indexing.
2023-10-12 13:06:35 -04:00