489087d18b
enhance: refactor executor framework V2 ( #35251 )
...
#32636
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-13 20:57:09 +08:00
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
4b553b0333
enhance: revert remove duplicated pk function ( #35103 )
...
issue: #34778
Revert "fix: fix query count(*) concurrently"
Revert "enhance: mark duplicated pk as deleted "
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-05 10:48:17 +08:00
5616b7e8d2
enhance: support null in c data_datacodec and load null value ( #32183 )
...
1. support read and write null in segcore
will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3. retrieve valid_data.
parse valid_data in search/query.
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-23 16:07:51 +08:00
804dd5409a
enhance: mark duplicated pk as deleted ( #34586 )
...
fix #34247
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-16 14:25:39 +08:00
589d4dfd82
enhance: optimize bitmap index ( #33358 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-30 13:09:43 +08:00
246586be27
enhance: Unify data type check APIs under internal/core ( #31800 )
...
Issue: #22837
Move and rename following C++ APIs:
datatype_sizeof() ==> GetDataTypeSize()
datatype_name() ==> GetDataTypeName()
datatype_is_vector() / IsVectorType() ==> IsVectorDataType()
datatype_is_variable() ==> IsVariableDataType()
datatype_is_sparse_vector() ==> IsSparseFloatVectorDataType()
datatype_is_string() / IsString() ==> IsDataTypeString()
datatype_is_floating() / IsFloat() ==> IsDataTypeFloat()
datatype_is_binary() ==> IsDataTypeBinary()
datatype_is_json() ==> IsDataTypeJson()
datatype_is_array() ==> IsDataTypeArray()
datatype_is_variable() == IsDataTypeVariable()
datatype_is_integer() / IsIntegral() ==> IsDataTypeInteger()
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-02 19:15:14 +08:00
659ad81ab7
fix: remove deprecated ut test ( #31499 )
...
#31498
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-26 14:01:07 +08:00
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id ( #30629 )
...
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-12 09:16:30 -07:00
1aa97a5c21
enhance: Support more relational operators for binary expressions ( #30902 )
...
issue: #30677
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 16:57:00 +08:00
8a219e0102
feat: Support knowhere trace using OpenTelemetry ( #30750 )
...
Issue: #21508
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
601a8b801b
fix: add move cursor function to physical expr ( #29603 )
...
#29570
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-09 17:08:48 +08:00
79c417b14e
fix: pass active count to query context instead of timestamp ( #29541 )
...
#29319
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-31 16:08:48 +08:00
a6eb7e5f9a
enhance: skip segment when using pk in (..) expr ( #29394 )
...
#29293
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-21 20:06:42 +08:00
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
b80a3e19d3
Add code for PanicInfo ( #27364 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-27 12:01:28 +08:00
5db4a0489e
dynamic index version control ( #27335 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
a362bb1457
Support array datatype ( #26369 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-19 14:23:23 +08:00
0afdfdb9af
Remove other Exceptions, keeps SegcoreError only ( #27017 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-14 14:05:20 +08:00
c073aa0dc3
Fix bug for json_contains_all has multiple array elements ( #26446 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-08-18 22:36:19 +08:00
709352f96c
Repeat multi times for performace compare test because of cpu cache ( #26394 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-17 15:18:17 +08:00
a0198ce8ae
Support json contains feature ( #25384 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-08-11 17:09:30 +08:00
9489e14000
Optimize multi logical exprs performance when meet some situations ( #26265 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-11 15:31:29 +08:00
65cb52d06b
Support dynamic simd framework and using term expr as example ( #25260 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-07-13 16:22:30 +08:00
dd5f896dc8
Load batch by batch ( #25212 )
...
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-06 13:58:27 +08:00
816158e4af
Remove outdated searchplan ( #25282 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-04 18:30:25 +08:00
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
f60b839127
Support element in json array in segcore part( #24677 ) ( #24829 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-06-14 14:38:37 +08:00
ceda0ed598
Optimize the performance of filter by JSON field ( #24268 )
...
- Construct JSON pointer only once
- Avoid copying nested path for each row
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-22 00:47:25 +08:00
c75e7a5d05
Fix failed to compare int value with double value ( #24229 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-05-19 12:57:23 +08:00
113f9a0ebc
Support SIMD of several Expr ( #23715 ) ( #23717 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-05-12 14:11:20 +08:00
9715a850fa
Support expr with json field ( #23804 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-05-10 10:19:19 +08:00
62eea5286f
Support to filter with json expr ( #23739 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-30 20:36:39 +08:00
60fdd7e4f4
Introduce simdjson ( #23644 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-26 10:30:34 +08:00
6f4ed517de
add growing segment index ( #23615 )
...
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-04-26 10:14:41 +08:00
a36fefb009
Fix cpplint ( #22657 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-10 09:47:54 +08:00
8c9c1672ae
Assign different storage config for indexes ( #19517 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 14:45:23 +08:00
428840178c
Support diskann index for vector field ( #19093 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-21 20:16:51 +08:00
a001412e12
Replace faiss::MetricType with knowhere::MetricType ( #17891 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-06-29 14:20:19 +08:00
56778787be
Reverse data from scalar index ( #17145 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-26 14:58:01 +08:00
21a1311f66
Merge utils/Utils.h into common/Utils.h ( #16762 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-05-03 12:05:50 +08:00
515d0369de
Support string type in segcore ( #16546 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
04fffb08ef
Support arithmetic operations on numerical fields for scalar filtering ( #16520 )
...
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
2022-04-24 16:43:45 +08:00
1227b9ebae
Rename utils/tools.h to utils/Utils.h ( #10665 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-10-26 17:28:23 +08:00
2d8e86692a
Format test_expr.cpp ( #9698 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-10-12 12:48:34 +08:00
6f7031111a
Update header file ( #9696 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-10-12 12:42:33 +08:00
d4232f88a2
For search funtion adding a round_decimal paramter to precision control ( #8574 )
...
Signed-off-by: elfisworking <zymustb@126.com>
2021-10-08 17:39:55 +08:00
8b81ceb5d7
Reduce cpp unittest time cost ( #8403 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-09-23 16:55:54 +08:00
aba21baf82
refactor expr executor to use single bitset ( #6667 )
...
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-07-23 15:30:12 +08:00
724f10b9a0
Unify the usage of query and search ( #6467 )
...
Unify the usage of query and search
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-07-13 22:20:33 +08:00
zhagnlu
cai.zhang
smellthemoon
Cai Yudong
Buqian Zheng
Enwei Jiao
foxspy
yah01
xige-16
yah01
Jiquan Long
Aivin V. Solatorio
elfisworking
groot
FluorineDog