Nick Sweeting
3093057675
bail out on sonic indexing after 5 errors
2021-04-10 05:18:03 -04:00
Nick Sweeting
32764347ce
add new SEARCH_BACKEND_TIMEOUT config option defaulted to 90sec
2021-04-09 00:27:12 -04:00
Nick Sweeting
21bb84f627
fix lint errors
2021-04-06 02:50:51 -04:00
Nick Sweeting
f67a5a215a
fix readability indexing process and implement a max total character length on indexed content
2021-04-06 02:01:38 -04:00
Nick Sweeting
bd6d9c165b
enforce utf8 on literally all file operations because windows sucks
2021-03-27 01:16:29 -04:00
Nick Sweeting
24e24934f7
add headers.json and fix relative singlefile path resolving for sonic
2021-01-30 21:59:34 -05:00
jdcaballerov
14df0cbb7c
Update sonic.py
...
Sonic buffer accepts 20.000 bytes not unicode characters, since the chunking here is on unicode characters, sending 20.000 characters will overflow sonic's buffer.
UTF-8 can take up to 6 bytes, so sending less than (20.000 / 6) rounded minus should be ok.
2021-01-20 14:51:46 -05:00
Nick Sweeting
326fe69eea
fix lint error
2020-12-12 12:35:32 -05:00
jdcaballerov
9b6afa36a3
Update archivebox/search/backends/ripgrep.py
...
Co-authored-by: Nick Sweeting <git@sweeting.me>
2020-12-12 08:36:08 -05:00
jdcaballerov
aa53f4f088
Update archivebox/search/backends/ripgrep.py
...
Co-authored-by: Nick Sweeting <git@sweeting.me>
2020-12-12 08:36:01 -05:00
jdcaballerov
24d4c44624
Add ripgrep configs
2020-12-12 07:36:31 -05:00
Cristian
e82161a768
refactor: Remove setup_django from search
2020-12-11 16:43:48 -05:00
Nick Sweeting
e90cf05141
fix lint errors
2020-12-11 16:51:11 +02:00
Cristian
9aac09a5e1
feat: Patch setup_django so we can use an inmemory db in specific commands
2020-12-08 18:42:25 -05:00
Cristian
8d22ebf988
feat: Remove walrus operator (we still need to support python3.7)
2020-12-06 12:23:02 -05:00
jdcaballerov
172197ae01
refactor: Remove if LENGTH and use text chunker for every input
2020-12-06 01:14:39 +02:00
jdcaballerov
5a6b814c79
Add exception handling for indexable content reader
2020-12-06 01:14:38 +02:00
JDC
15fbd81480
Change MAX_SONIC_TEXT_LENGTH
2020-12-06 01:14:38 +02:00
JDC
db9c2edccc
Add log print for url indexing
2020-12-06 01:14:38 +02:00
JDC
0acf479b70
Partition long strings in chunks for sonic
2020-12-06 01:14:38 +02:00
JDC
caf4660ac8
Add indexing to update command and utilities
2020-12-06 01:14:37 +02:00
JDC
23a9beb4e0
Add ignored extensions in ripgrep search
2020-12-06 01:13:39 +02:00
JDC
95382b3812
Add ripgrep rg search backend and set as default
2020-12-06 01:13:39 +02:00
JDC
4eeedae815
Exception handling for indexing and searching
2020-12-06 01:13:39 +02:00
JDC
fb67d6684c
fix: Return empty QuerySet instead of list
2020-12-06 01:12:47 +02:00
JDC
823df34080
Use QuerySets for search backend API instead of pks
2020-12-06 01:12:47 +02:00
JDC
f383648ffc
Use a generator for snapshot flush from index
2020-12-06 01:12:47 +02:00
JDC
47daa038eb
Implement flush for search backend after remove command
2020-12-06 01:12:47 +02:00
JDC
c2c01af3ad
Add config for search backend
2020-12-06 01:12:47 +02:00
JDC
5f6673c72c
Implement backend architecture for search engines
2020-12-06 01:12:46 +02:00
JDC
b1f70b2197
Initial implementation
2020-12-06 01:12:45 +02:00
