4224 Commits

Author SHA1 Message Date
Fanis Tharropoulos
0e1bcd65b5
chore(bench): add vitest 2024-12-30 11:14:34 +02:00
Fanis Tharropoulos
9446a65eb8
chore(bench): add fs-extra and type-fest 2024-12-30 11:06:35 +02:00
Fanis Tharropoulos
5cd38d7525
chore(bench): add types for node 2024-12-30 11:06:35 +02:00
Fanis Tharropoulos
2556763159
chore(bench): add commander 2024-12-30 11:06:35 +02:00
Fanis Tharropoulos
74b7e4cdda
chore(bench): add nvmrc 2024-12-30 11:06:35 +02:00
Fanis Tharropoulos
9ebf24f69f
chore(bench): config tsup 2024-12-30 11:06:35 +02:00
Fanis Tharropoulos
3593157c83
chore(bench): config prettier 2024-12-30 11:06:35 +02:00
Fanis Tharropoulos
5b177511c6
chore(bench): config typescript 2024-12-30 11:06:35 +02:00
Fanis Tharropoulos
035805d015
chore(bench): config eslint 2024-12-30 10:44:40 +02:00
Fanis Tharropoulos
4718e940ae
chore(bench): add gitignore 2024-12-30 10:39:19 +02:00
Fanis Tharropoulos
f2df252b1b
chore(bench): init benchmark repo 2024-12-30 10:37:15 +02:00
Krunal Gandhi
a8b7e93990
add param for text_match based sorting (#2120)
Some checks failed
tests / test (push) Has been cancelled
2024-12-27 18:19:12 +05:30
Harpreet Sangar
cd92c727ff
Support not left join like, !$User_Views(user_id: user_a). (#2119)
Some checks failed
tests / test (push) Has been cancelled
* Support not left join like, `!$User_Views(user_id: user_a)`.

* Fix `CollectionInfixSearchTest, InfixWithFiltering`.

* Add comment.

* Refactor `negate_left_join`.
2024-12-25 21:15:17 +05:30
Kishore Nallan
0399aba2ae Start on-demand snapshot copying only after internal snapshot closure fully returns.
Some checks failed
tests / test (push) Has been cancelled
2024-12-23 17:10:15 +05:30
Krunal Gandhi
c5cab62d0e
Field token_separators and symbols_to_index (#2118)
Some checks failed
tests / test (push) Has been cancelled
* initial commit for fieldwise_token_separators

* add basic field test

* add further test

* support fieldwise locale tokenization for query inference

* Revert "support fieldwise locale tokenization for query inference"

This reverts commit 00ce97c16b58a7cb964228ad84bf5f08aab34876.

* add reload test and multi-char string validation

* update error message
2024-12-20 12:03:21 +05:30
Harpreet Sangar
49d746deba
geopoint[] field inside an object[] should be treated as geopoint. (#2116)
Some checks are pending
tests / test (push) Waiting to run
2024-12-19 21:17:06 +05:30
Harpreet Sangar
2b1c4f7c24
Add more return parameters. (#2113)
Some checks failed
tests / test (push) Has been cancelled
2024-12-18 11:59:40 +05:30
Harpreet Sangar
0855aeff24
Union (#2051)
Some checks failed
tests / test (push) Has been cancelled
* Add `collection_search_args`.

* Remove `collection_name` argument. Use `Index::get_collection_name()` instead.

* Add `collection_search_args_t`, `Union_KV` and `Union_Topster`.

* Allow multiple sub-searches to the same collection in union.

* Fix union tests.

* Union pagination.

* Union sorting.

* Update test.

* Totals documents count of a particular collection should only be added once in `out_of`.

* Update error message.

* Make `Topster` generic.

* Refactor template declaration.

* Fix `CollectionFacetingTest, FacetCountsBool`.

* Rename parameter.

* Rename return parameters.

* Add `search_time_ms` and `page` return parameters.
2024-12-16 21:31:49 +05:30
Harpreet Sangar
ec19052db1
Fix unindexed fields of joined collection not being included with wildcard include. (#2107)
Some checks are pending
tests / test (push) Waiting to run
2024-12-16 11:56:55 +05:30
Kishore Nallan
a282aafe65 Remove ARM lg page size for jemalloc.
Some checks failed
tests / test (push) Has been cancelled
2024-12-11 21:07:23 +05:30
Kishore Nallan
57e1548acb Fix error: add flag for jemalloc for cache oblivious. 2024-12-11 20:49:07 +05:30
Kishore Nallan
61ebe244aa Add flag for jemalloc for cache oblivious. 2024-12-11 20:31:42 +05:30
Krunal Gandhi
9923a5455e
hybrid search flat_search_cutoff (#2099)
Some checks failed
tests / test (push) Has been cancelled
* hybrid search flat_search_cutoff

* refactor repeatative code

* make both approach exclusive
2024-12-10 19:03:20 +05:30
Kishore Nallan
225ed057cd Revert removing jemalloc unused memory as part of calculations.
Some checks failed
tests / test (push) Has been cancelled
2024-12-09 19:38:23 +05:30
Ozan Armağan
f6bdba15eb
Fix custom dimension initialization (#2092)
Some checks are pending
tests / test (push) Waiting to run
* Fix custom dimension initialization

* Add test
2024-12-06 17:53:46 +05:30
Kishore Nallan
04cb05aec3 Remove stray log. 2024-12-06 11:42:51 +05:30
Kishore Nallan
67d6f21991 Use explicit num_dim passed to auto embedding remote models. 2024-12-06 11:25:58 +05:30
Krunal Gandhi
adbfa7360e
Normalize plurals (#2062)
* initial commit

* allow multiple plurals set

* acess dictionary word from stemmerManager directly

* use find over count

* make dictionary post request async

* add endpoints to get dictionary

* update function names

* update core_api function names

* fix the test

* update end-point names

* reload dictionaries on restart

---------

Co-authored-by: Kishore Nallan <kishorenc@gmail.com>
2024-12-05 14:06:25 +05:30
Ozan Armağan
d126f91aa1
Refactor get_history_collection by overloading (#2090)
Some checks failed
tests / test (push) Has been cancelled
* Refactor `get_history_collection` by overloading

* Return error when model doesn't have history collection

* Refactoring

* Remove unnecessary function

* Refactor add_conversation

* Make conversation model id necessary for deleting conversation
2024-12-03 19:06:52 +05:30
Ozan Armağan
83f1cad4b9
Fix conversation not found error (#2087)
Some checks are pending
tests / test (push) Waiting to run
* Fix `conversation not found` error

* Add test and refactor

* Move getting last n messages logic to a functing
2024-12-03 15:36:25 +05:30
Ozan Armağan
8ec85d7dba
Move public models to ts_ prefix directories (#2070)
Some checks are pending
tests / test (push) Waiting to run
* Move public models to ts_ prefix directories

* Use `butil::Move` to rename

* Refactor saving model config

* Delete the dest folder before move for model prefix migration

---------

Co-authored-by: Kishore Nallan <kishorenc@gmail.com>
2024-12-02 18:53:03 +05:30
Harisaran G
d06de6f14d
add: GET /events endpoint (#2084)
Some checks failed
tests / test (push) Has been cancelled
* add: get_events endpoint

* fix: query_compaction_interval for test build

* fix: tests

* fix: analytics_manager.h

* fix: tests
2024-11-29 15:15:33 +05:30
Kishore Nallan
84a25e2aa5 Add env flag for disabling embedding model memory check.
Some checks are pending
tests / test (push) Waiting to run
Useful for local dev where swap usage is ok.
2024-11-29 09:55:21 +05:30
Kishore Nallan
f829d4f922 Access guard for tokenizer. 2024-11-29 09:54:43 +05:30
Kishore Nallan
c6ed3c9be1 Consider all fields for aux text match score.
Some checks are pending
tests / test (push) Waiting to run
2024-11-28 11:21:06 +05:30
Kishore Nallan
5ac477a13b Refactor on-demand snapshot validation logic.
Some checks failed
tests / test (push) Has been cancelled
2024-11-26 13:08:35 +05:30
Kishore Nallan
bf5172fe40 Add test case for validation of group by field name.
Some checks are pending
tests / test (push) Waiting to run
2024-11-25 21:17:53 +05:30
Kishore Nallan
4752c09227 Implement validate field names for group by. 2024-11-25 20:46:51 +05:30
Harpreet Sangar
0379072083
When validate_field_names is false, ignore the filter whose field name is not found in the schema. (#2077)
* When `validate_field_names` is false, ignore the filter whose field name is not found in the schema.

* Add test.

* Ignored filter should match 0 documents.

* Respect `validate_field_names` in export, update, delete, overrides, _eval, and joins.

* Refactor `validate_field_names` initialization.

* Update comment.
2024-11-25 18:30:01 +05:30
Harisaran G
67ae222c96
fix: analytics_manager validation and tsconfig (#2079)
* fix: analytics_manager validation and tsconfig

* fix: review comments
2024-11-25 15:43:01 +05:30
Kishore Nallan
b26a856827 Refactor memory stats calculations.
Some checks failed
tests / test (push) Has been cancelled
2024-11-23 11:19:07 +05:30
Kishore Nallan
1f590e443e Fix facet value intersection galloping. 2024-11-23 08:41:32 +05:30
Kishore Nallan
a103d97487 Fix updates to empty vector record.
Some checks are pending
tests / test (push) Waiting to run
2024-11-21 18:51:40 +05:30
Kishore Nallan
30c442ee0d Fix test.
Some checks are pending
tests / test (push) Waiting to run
2024-11-21 11:57:10 +05:30
Kishore Nallan
6fa4296392 Add index for resolving facet string value. 2024-11-21 11:40:18 +05:30
Kishore Nallan
5b900be858 Move get_proc_meminfo to system metrics. 2024-11-16 16:42:07 +05:30
Kishore Nallan
ce276e1606 Add flag for skipping validation of field names.
Some checks failed
tests / test (push) Has been cancelled
2024-11-15 14:13:22 +05:30
Ozan Armağan
05be6d1eef
Fine grain embedding locks (#2067)
Some checks are pending
tests / test (push) Waiting to run
* Fine grain  embedding locks

* Fix unique lock bug
2024-11-14 19:23:31 +05:30
Harisaran G
4373b2b6fd
add: personalization model APIs (#2001)
Some checks are pending
tests / test (push) Waiting to run
* add: basic recommendations endpoint

* add: POST `/recommendations/models` recommendations endpoint

* add: get specific model endpoint

* add: DELETE `/recommendations/models/:id` and GET all models recommendations endpoint

* add: ONNX model upload endpoint

* add: save the file as a .tar.gz instead of ONNX

* add: verify_tar_gz function

* add: support for model upload

* add: PUT `/recommendations/model/:id` to update the model, name, collection

* refactor: recommendations -> personalization

* add: personalization tests

* add: personalization_model_manager_tests

* fix: review comments

* fix: review comments
2024-11-14 13:08:42 +05:30
Ozan Armağan
6a762d637a
Fix race condition for image embedding models (#2065) 2024-11-14 11:31:40 +05:30