c9bc70f272
fix: [AddField] Use shared_ptr of schema in plan fixing dangling ref ( #42693 )
...
Related to #42640
The search/query plan holded a reference to schema, which could be
destructed after schema change. This PR make plan hold a shared ptr to
it fixing dangling reference problem under concurrent read & schema
change.
This PR also remove field binlog check for loading index for old segment
with old schema may have binlog lack.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-06-12 20:46:36 +08:00
3de904c7ea
feat: add cachinglayer to sealed segment ( #41436 )
...
issue: https://github.com/milvus-io/milvus/issues/41435
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-04-28 10:52:40 +08:00
bf617115ca
enhance: Remove single chunk segment related codes ( #39249 )
...
https://github.com/milvus-io/milvus/issues/39112
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-04-11 18:56:29 +08:00
cb1e86e17c
enhance: support add field ( #39800 )
...
after the pr merged, we can support to insert, upsert, build index,
query, search in the added field.
can only do the above operates in added field after add field request
complete, which is a sync operate.
compact will be supported in the next pr.
#39718
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-04-02 14:24:31 +08:00
2a02bbe3ee
enhance: Use template to remove unittest duplication ( #39144 )
...
Issue: #38666
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2025-01-13 09:58:57 +08:00
489087d18b
enhance: refactor executor framework V2 ( #35251 )
...
#32636
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-13 20:57:09 +08:00
5616b7e8d2
enhance: support null in c data_datacodec and load null value ( #32183 )
...
1. support read and write null in segcore
will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3. retrieve valid_data.
parse valid_data in search/query.
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-23 16:07:51 +08:00
8a219e0102
feat: Support knowhere trace using OpenTelemetry ( #30750 )
...
Issue: #21508
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
60e88fb833
fix: Restore the MVCC functionality. ( #29749 )
...
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:
1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.
issue: #29656
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-09 11:38:48 +08:00
9b52cb6417
enhance: improve reducing results when many segments are filtered ( #29073 )
...
Do not fill the invalid ids for the empty results, it will incur useless
memory overhead and reduce overhead when nq and topk is large.
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-20 12:56:42 +08:00
f7d2ab6677
enhance: reduce 1x copy for variable length field while retrieving ( #28345 )
...
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-15 18:08:20 +08:00
c3f15c6b95
Refactor duplicate error class into one place ( #26985 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-11 20:43:17 +08:00
ca1349708b
Remove time travel ralted testcase ( #26119 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-10 18:53:17 +08:00
816158e4af
Remove outdated searchplan ( #25282 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-04 18:30:25 +08:00
26a6e1b946
Fix BinaryRange expression on integer overflow case ( #24920 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-06-16 09:56:39 +08:00
f60b839127
Support element in json array in segcore part( #24677 ) ( #24829 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-06-14 14:38:37 +08:00
60fdd7e4f4
Introduce simdjson ( #23644 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-26 10:30:34 +08:00
6f4ed517de
add growing segment index ( #23615 )
...
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-04-26 10:14:41 +08:00
a36fefb009
Fix cpplint ( #22657 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-10 09:47:54 +08:00
089912baa7
Fix segcore unit test name ( #19783 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-14 14:29:24 +08:00
87d78a4a85
Ignore cases when comparing metric type in segcore ( #19437 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-09-26 17:58:52 +08:00
a001412e12
Replace faiss::MetricType with knowhere::MetricType ( #17891 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-29 14:20:19 +08:00
f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore ( #17389 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-06 21:34:05 +08:00
56778787be
Reverse data from scalar index ( #17145 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-26 14:58:01 +08:00
08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks ( #17022 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-23 16:41:58 +08:00
515d0369de
Support string type in segcore ( #16546 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
58ea38142f
Use boost dynamic_bitset in segcore ( #16476 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-14 22:37:34 +08:00
27b4cbc098
Cherry pick remove translateHits commit to mater ( #16436 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2022-04-08 20:27:31 +08:00
a37479d728
Upgrade to knowhere-v1.1.2 to support all index types for mac ( #16416 )
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2022-04-08 15:29:31 +08:00
cbb01051f0
Update Search return type ( #12578 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-02 11:45:32 +08:00
cb952d6036
Rename SearchResult fields for better readability ( #12327 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-11-29 14:57:18 +08:00
356b7e36ba
Reorder header files for test_query ( #9932 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-10-15 16:56:51 +08:00
d4232f88a2
For search funtion adding a round_decimal paramter to precision control ( #8574 )
...
Signed-off-by: elfisworking <zymustb@126.com>
2021-10-08 17:39:55 +08:00
af1bdaf202
Reduce cpp unittest time cost ( #8409 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-09-23 17:55:54 +08:00
724f10b9a0
Unify the usage of query and search ( #6467 )
...
Unify the usage of query and search
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-07-13 22:20:33 +08:00
97b1426666
fix benchmark, add support for primary key in schemaHelper ( #6281 )
...
* fix benchmark
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
* fix tests
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-07-06 09:50:01 +08:00
bf8b2be4a7
Deprecate num_groups to simplify search API ( #6230 )
...
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-07-01 10:32:15 +08:00
c2734fa55f
Fix bug and enchance system
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-03-22 16:36:10 +08:00
f3649f0419
Refactor interface and proto
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-12 14:22:09 +08:00
66146223ca
Support flat
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-02-27 12:46:37 +08:00
2ca53fa668
Fix msgstream deadlock when loadCollection
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-02-07 15:47:10 +08:00
eba6bb288c
Refactor the sructure of proto file
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-01-22 09:36:18 +08:00
7f044fff82
Enable search for sealed
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-21 15:29:52 +08:00
63ca5f8031
Add span and segment interface, rename segment_growing and use interface as base
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-13 18:46:25 +08:00
1104f059ee
Update doc:module interfaces
...
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2021-01-13 11:08:03 +08:00
8e9d8e36e1
Update run_go_unittest.sh
...
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-01-13 10:40:46 +08:00
4cd42c553f
Rename field_name, make field_id strongly typed, skip multithread test
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-12 18:31:52 +08:00
3bf205d9a8
Fix inner product
...
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-07 09:32:17 +08:00
4015d7245d
Merge operation
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-06 14:45:50 +08:00
da6eeddbf8
Add flush timestamp
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-06 13:14:38 +08:00
congqixia
Buqian Zheng
Bingyi Sun
smellthemoon
Cai Yudong
zhagnlu
zhenshan.cao
Gao
yah01
Enwei Jiao
Jiquan Long
yah01
foxspy
bigsheeper
xige-16
elfisworking
groot
FluorineDog
godchen
FluorineDog
GuoRentong
quicksilver
neza2017
sunby