milvus-io
/
milvus
mirror of https://github.com/milvus-io/milvus.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
22,860 Commits
174 Branches
147 Tags
595 MiB
Commit Graph

716 Commits (master)

Author SHA1 Message Date
Jiquan Long aa36f9feed
fix: [ut] regex query under unsupported index (#34087) 
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/29988

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-24 14:08:03 +08:00
Patrick Weizhi Xu b961767005
enhance: support integral type for MV and skip MV if there is only one category (#33161) 
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 10:20:01 +08:00
zhagnlu 03a3f50892
enhance: add skip using array index when some situation (#33947) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-23 21:26:02 +08:00
chyezh 259a682673
enhance: async search and retrieve in cgo (#33228) 
issue: #30926, #33132
related pr: #33133

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-22 09:38:02 +08:00
cqy123456 dc4437ff82
enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk (#33993) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-20 17:42:02 +08:00
Gao 0d20303e54
fix: fix binary vector data size (#33750) 
issue: https://github.com/milvus-io/milvus/issues/22837

- fix byte size wrong for binary vectors
- fix the expect/actual error msg

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-18 21:39:59 +08:00
cqy123456 32f685ff12
enhance: growing segment support mmap (#32633) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
zhagnlu d43ec4db0b
enhance: support array bitmap index (#33527) 
#32900

---------

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:51:58 +08:00
Buqian Zheng 8cb350598c
enhance: Improve GetVectorById of Sparse Float Vector (#33209) 
issue: #29419

* sparse float vector to support raw data mmap

For get vector from chunk cache, I added a unit test but marking it as
skipped due to a known issue. I have tested it locally.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-12 10:09:55 +08:00
Jiquan Long ecf2bcee42
enhance: speed up array-equal operator via inverted index (#33633) 
fix: #33632

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-11 14:13:54 +08:00
chyezh f53ab54c5d
enhance: async cgo utility (#33133) 
issue: #30926, #33132

- implement future-based cgo utility.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-09 22:55:53 +08:00
cai.zhang 27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
Gao 545d4725fb
fix: correct get vector data size for bf16/fp16/binary vector (#33377) 
related #22837

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-05 14:31:57 +08:00
congqixia 597f4c5e03
enhance: Make hasMoreResult accurate when hit number larger than limit (#33609) 
See also milvus-io/milvus-sdk-go#756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-05 11:51:51 +08:00
Jiquan Long 0c5d8660aa
feat: support inverted index for array (#33452) 
issue: https://github.com/milvus-io/milvus/issues/27704

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-31 09:47:47 +08:00
Chun Han 416a2cf507
fix: query iterator lack results(#33137) (#33422) 
related: #33137 
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-05-30 17:51:44 +08:00
zhagnlu 589d4dfd82
enhance: optimize bitmap index (#33358) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-05-30 13:09:43 +08:00
Buqian Zheng c5918ffbdb
enhance: mark sparse inverted index as mmap-able (#33281) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-23 14:11:42 +08:00
cai.zhang be77ceba84
enhance: Use proto for passing info in cgo (#33184) 
issue: #33183

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-23 10:31:40 +08:00
zhagnlu d669fbcf46
enhance: support bitmap index for scalar type (#32902) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-05-19 21:49:38 +08:00
Buqian Zheng bb7765cbd6
fix: fix Indexing.Iterator ut: build index with all data at once (#32844) 
issue: #32843

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-10 11:31:30 +08:00
Chun Han 01c2684355
enhance: [skip e2e] disable unstable ut temporarily (#32836) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-05-08 12:17:29 +08:00
Jiquan Long 9837ad6a8d
enhance: remove deprecated api (#32808) 
issue: #32728

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-07 20:03:30 +08:00
Jiquan Long 1f58cda957
enhance: add more trace for search & query (#32734) 
issue: https://github.com/milvus-io/milvus/issues/32728

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-07 13:03:29 +08:00
Bingyi Sun fecd9c21ba
feat: LRU cache implementation (#32567) 
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
 2024-05-06 20:29:30 +08:00
Buqian Zheng 858599d831
enhance: sparse float vector to support brute force iterator and range search (#32635) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-04-29 14:35:26 +08:00
smellthemoon 46d7298407
fix: file not find test_parquet.cpp (#32644) 
#32646

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-04-26 16:57:27 +08:00
smellthemoon 4fb8044a27
enhance: delete some no lint code (#32182) 
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-04-26 14:15:26 +08:00
Jiquan Long ccce1e928a
fix: regex query can't handle text with newline (#32569) 
issue: https://github.com/milvus-io/milvus/issues/32482

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-04-26 12:01:26 +08:00
Jiquan Long c002745902
enhance: retrieve output fields after local reduce (#32346) 
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-04-25 09:49:26 +08:00
chyezh e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData (#32012) 
issue: #30361

- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.

- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`

- Remove repeat load delta log operation in lru.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-16 11:17:20 +08:00
Chun Han 337cc0756d
fix: lack good results for insufficient ef(#29883) (#32080) 
related: #29883

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-04-13 22:13:23 +08:00
Patrick Weizhi Xu 52ae47c850
enhance: gather materialized view search info once per request (#31996) 
issue: #29892 

This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-04-11 15:21:19 +08:00
cai.zhang 1b767669a4
enhance: Throw error instead of crash when index cannot be built (#31844) 
issue: #27589

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-04-09 11:51:18 +08:00
cqy123456 aba4993c6c
fix: fix some fp16/bf16 code miss in segcore. (#31771) 
issue：https://github.com/milvus-io/milvus/issues/22837

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-04-07 14:13:16 +08:00
Alexander Guzhva cae5722229
enhance: performance improvements for the bitset (#31753) 
Issue: #31752

This PR improves the performance for bitset utilities (introduced in PR
#30454), including varchar filtering

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-04-06 05:19:22 +08:00
zhagnlu b2669e26dc
fix:reduce thread pool test time (#31893) 
#31877

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-04-05 10:05:12 +08:00
zhagnlu d6d3b01a04
fix:remove thread pool timeout test because of high load cpu (#31879) 
#31877

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-04-03 15:55:38 +08:00
Cai Yudong 246586be27
enhance: Unify data type check APIs under internal/core (#31800) 
Issue: #22837 

Move and rename following C++ APIs:
datatype_sizeof() ==> GetDataTypeSize()
datatype_name() ==> GetDataTypeName()
datatype_is_vector() / IsVectorType() ==> IsVectorDataType()
datatype_is_variable() ==> IsVariableDataType()
datatype_is_sparse_vector() ==> IsSparseFloatVectorDataType()
datatype_is_string() / IsString() ==> IsDataTypeString()
datatype_is_floating() / IsFloat() ==> IsDataTypeFloat()
datatype_is_binary() ==> IsDataTypeBinary()
datatype_is_json() ==> IsDataTypeJson()
datatype_is_array() ==> IsDataTypeArray()
datatype_is_variable() == IsDataTypeVariable()
datatype_is_integer() / IsIntegral() ==> IsDataTypeInteger()

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-04-02 19:15:14 +08:00
chyezh 5655ec4fc0
enhance: add mmap usage metrics (#31708) 
issue: #31707

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-01 11:35:12 +08:00
Cai Yudong 675a5dc822
fix: Save traceID and spanID as std::vector into search config (#31278) 
Issue: #30961

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2024-03-29 14:29:11 +08:00
Jiquan Long 4eb4df1e81
fix: predict inverted index resource usage more reasonably (#31615) 
/kind improvement
issue: #31617

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-03-27 19:33:09 +08:00
zhagnlu 659ad81ab7
fix: remove deprecated ut test (#31499) 
#31498

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-03-26 14:01:07 +08:00
Alexander Guzhva c4b37fb285
enhance: Custom bitset and bitsetview prototypes (#30454) 
Issue: #31285 

Basically, I've replaced `FixedVector<bool>` and `boost::dynamic_bitset`
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.

This PR is for internal use only. 

Current progress (numbers are for GCC 9.5.0 on Ubuntu 22.04 LTS;
clang-17 produces better performance numbers):
Baseline:
```
[ RUN      ] CApiTest.AssembeChunkPerfTest
start test
cost: 17903us
[       OK ] CApiTest.AssembeChunkPerfTest (183 ms)

[ RUN      ] Expr.TestMultiLogicalExprsOptimization
cost: 1391us
cost: 5us
cost: 4us
cost: 4us
cost: 6us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
143
cost: 10us
cost: 8us
cost: 10us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 9us
8
/home/ubuntu/zilliz/milvus4/milvus/internal/core/unittest/test_expr.cpp:1561: Failure
Expected: (cost_op) < (cost_no_op), actual: 143 vs 8
[  FAILED  ] Expr.TestMultiLogicalExprsOptimization (7 ms)
[ RUN      ] Expr.TestExprs
start test
3cost: 889us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 3us
start test
100cost: 7us
start test
200cost: 16us
[       OK ] Expr.TestExprs (9 ms)

[ RUN      ] Expr.TestUnaryBenchTest
start test type:2
 cost: 124.8us
start test type:3
 cost: 163.1us
start test type:4
 cost: 275.9us
start test type:5
 cost: 590.9us
start test type:10
 cost: 62.7us
start test type:11
 cost: 65.9us
[       OK ] Expr.TestUnaryBenchTest (1153 ms)
[ RUN      ] Expr.TestBinaryRangeBenchTest
start test type:2
 cost: 151.4us
start test type:3
 cost: 198.4us
start test type:4
 cost: 361.9us
start test type:5
 cost: 753.9us
start test type:10
 cost: 64.6us
start test type:11
 cost: 62.2us
[       OK ] Expr.TestBinaryRangeBenchTest (1151 ms)
[ RUN      ] Expr.TestLogicalUnaryBenchTest
start test type:2
 cost: 121.14us
start test type:3
 cost: 156.84us
start test type:4
 cost: 249.76us
start test type:5
 cost: 534.44us
start test type:10
 cost: 82.2us
start test type:11
 cost: 83.52us
[       OK ] Expr.TestLogicalUnaryBenchTest (1202 ms)
[ RUN      ] Expr.TestBinaryLogicalBenchTest
start test type:2
 cost: 80.64us
start test type:3
 cost: 78.22us
start test type:4
 cost: 255.76us
start test type:5
 cost: 532.04us
start test type:10
 cost: 89.26us
start test type:11
 cost: 90us
[       OK ] Expr.TestBinaryLogicalBenchTest (1198 ms)
[ RUN      ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
 cost: 401.7us
start test type:3
 cost: 420.96us
start test type:4
 cost: 418.04us
start test type:5
 cost: 470.54us
start test type:10
 cost: 250.32us
start test type:11
 cost: 850.08us
[       OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1273 ms)
[ RUN      ] Expr.TestCompareExprBenchTest
start test type:2
 cost: 162us
start test type:3
 cost: 142us
start test type:4
 cost: 374us
start test type:5
 cost: 674us
start test type:10
 cost: 366us
start test type:11
 cost: 645us
[       OK ] Expr.TestCompareExprBenchTest (1214 ms)
[ RUN      ] Expr.TestRefactorExprs
start test
3cost: 1253us
start test
10cost: 1060us
start test
20cost: 681us
start test
30cost: 522us
start test
50cost: 511us
start test
100cost: 506us
start test
200cost: 497us
[       OK ] Expr.TestRefactorExprs (1142 ms)

```

Candidate:
```
[ RUN      ] CApiTest.AssembeChunkPerfTest
start test
cost: 6099us
[       OK ] CApiTest.AssembeChunkPerfTest (153 ms)

[ RUN      ] Expr.TestMultiLogicalExprsOptimization
cost: 42us
cost: 15us
cost: 15us
cost: 14us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
17
cost: 41us
cost: 39us
cost: 33us
cost: 33us
cost: 33us
cost: 33us
cost: 34us
cost: 41us
cost: 34us
cost: 34us
35
[       OK ] Expr.TestMultiLogicalExprsOptimization (6 ms)
[ RUN      ] Expr.TestExprs
start test
3cost: 20us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 4us
start test
100cost: 8us
start test
200cost: 15us
[       OK ] Expr.TestExprs (8 ms)

[ RUN      ] Expr.TestUnaryBenchTest
start test type:2
 cost: 55.7us
start test type:3
 cost: 79.8us
start test type:4
 cost: 177.6us
start test type:5
 cost: 337.2us
start test type:10
 cost: 16.9us
start test type:11
 cost: 15.7us
[       OK ] Expr.TestUnaryBenchTest (1140 ms)
[ RUN      ] Expr.TestBinaryRangeBenchTest
start test type:2
 cost: 57.1us
start test type:3
 cost: 87us
start test type:4
 cost: 177.5us
start test type:5
 cost: 342.7us
start test type:10
 cost: 17.9us
start test type:11
 cost: 16.7us
[       OK ] Expr.TestBinaryRangeBenchTest (1152 ms)
[ RUN      ] Expr.TestLogicalUnaryBenchTest
start test type:2
 cost: 34.58us
start test type:3
 cost: 68.86us
start test type:4
 cost: 151.38us
start test type:5
 cost: 286.8us
start test type:10
 cost: 16.54us
start test type:11
 cost: 16.7us
[       OK ] Expr.TestLogicalUnaryBenchTest (1165 ms)
[ RUN      ] Expr.TestBinaryLogicalBenchTest
start test type:2
 cost: 20us
start test type:3
 cost: 17.1us
start test type:4
 cost: 154.12us
start test type:5
 cost: 286.1us
start test type:10
 cost: 19.6us
start test type:11
 cost: 19.24us
[       OK ] Expr.TestBinaryLogicalBenchTest (1188 ms)
[ RUN      ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
 cost: 125.7us
start test type:3
 cost: 111.34us
start test type:4
 cost: 148.02us
start test type:5
 cost: 306.7us
start test type:10
 cost: 149.3us
start test type:11
 cost: 282.94us
[       OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1221 ms)
[ RUN      ] Expr.TestCompareExprBenchTest
start test type:2
 cost: 89us
start test type:3
 cost: 79us
start test type:4
 cost: 323us
start test type:5
 cost: 629us
start test type:10
 cost: 313us
start test type:11
 cost: 591us
[       OK ] Expr.TestCompareExprBenchTest (1228 ms)
[ RUN      ] Expr.TestRefactorExprs
start test
3cost: 874us
start test
10cost: 611us
start test
20cost: 290us
start test
30cost: 294us
start test
50cost: 272us
start test
100cost: 278us
start test
200cost: 279us
[       OK ] Expr.TestRefactorExprs (1149 ms)

```

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-03-24 21:49:07 +08:00
Patrick Weizhi Xu 982dd2834b
enhance: add materialized view search info (#30888) 
issue: #29892 

This PR
1. Pass Materialized View (MV) search information obtained from the
expression parsing planning procedure to Knowhere. It only performs when
MV is enabled and the partition key is involved in the expression. The
search information includes:
1. Touched field_id and the count of related categories in the
expression. E.g., `color == red && color == blue` yields `field_id ->
2`.
2. Whether the expression only includes AND (&&) logical operator,
default `true`.
    3. Whether the expression has NOT (!) operator, default `false`.
4. Store if turning on MV on the proxy to eliminate reading from
paramtable for every search request.
5. Renames to MV.

## Rebuttals
1. Did not write in `ExtractInfoPlanNodeVisitor` since the new scalar
framework was introduced and this part might be removed in the future.
2. Currently only interested in `==` and `in` expression, `string` data
type, anything else is a bonus.
3. Leave handling expressions like `F == A || F == A` for future works
of the optimizer.

## Detailed MV Info

![image](https://github.com/milvus-io/milvus/assets/6563846/b27c08a0-9fd3-4474-8897-30a3d6d6b36f)

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-03-21 11:19:07 +08:00
groot c81909bfab
enhance: Support MinIO TLS connection (#31311) 
issue: https://github.com/milvus-io/milvus/issues/30709
pr: #31292

Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
 2024-03-21 11:15:20 +08:00
Chun Han 6939ad15f2
fix:possible out-of-bound due to groupby when reduing(#30711) (#31200) 
related: #30711

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-03-14 13:07:03 +08:00
Buqian Zheng 96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id (#30629) 
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-03-12 09:16:30 -07:00
Buqian Zheng 070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building (#30357) 
This commit adds sparse float vector support to segcore with the
following:

1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built

This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-03-11 14:45:02 +08:00
Cai Yudong a99143dd52
fix: Save traceID and spanID as hex string into search config (#31071) 
Issue: #30961

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2024-03-11 14:21:01 +08:00
cai.zhang 1aa97a5c21
enhance: Support more relational operators for binary expressions (#30902) 
issue: #30677

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-03-01 16:57:00 +08:00
Jiquan Long e2f35954d4
enhance: support pattern matching on json field (#30779) 
issue: https://github.com/milvus-io/milvus/issues/30714

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-02-28 18:31:00 +08:00
Jiquan Long 16b785e149
enhance: optimize the memory usage and speed up loading variable length data (#30787) 
/kind improvement
this removes the 1x copying while loading variable length data, also
avoids constructing std::string, which could lead to memory
fragmentation

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: yah01 <yah2er0ne@outlook.com>
 2024-02-28 16:45:00 +08:00
congqixia a115b731ed
enhance: fix old pr cpp format issue (#30894) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-28 16:28:20 +08:00
Cai Yudong 8a219e0102
feat: Support knowhere trace using OpenTelemetry (#30750) 
Issue: #21508

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2024-02-28 12:29:00 +08:00
yah01 57397b1307
enhance: add new LRU cache impl (#30360) 
- remove  the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache

related #30361

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-02-27 20:58:40 +08:00
Jiquan Long e2330f02f8
fix: pattern match use incorrect raw data (#30764) 
issue: https://github.com/milvus-io/milvus/issues/30687
We store all the varchar datas in an continuous address and use
string_view to quickly find them. In this case, using string_view.data()
directly will point to all rest varchar datas.

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-02-22 19:56:52 +08:00
MrPresent-Han 77eb6defb1
feat: support groupby on growing and non-indexed sealed egment(#30307) (#30644) 
related: #30308

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-02-21 14:02:53 +08:00
zhagnlu 18aac076de
fix: move test from NEON to X86 (#30324) 
#26137

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-02-21 11:58:53 +08:00
zhagnlu 976b6fc0e4
enhance: change opendal as compile configurable (#30384) 
#30373

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-02-20 19:16:52 +08:00
congqixia 18c351efa6
fix: Prevent ChunkCache use absolute path in All-in-one mode (#30666) 
See also #30651

Append operator of `std::filesystem::path` will replace whole path when
the param of "/" operation is an absolute path.

In "All-in-one" mode, this shall cause ChunkCache removing the original
vector data file when building chunk cache during/after load procedure.

This PR changes the ChunkCache path generation logic to a separate
function in which will check whether the file path is absolute or not.
If the file path is absolute, it removes the root path prefix and return
concatenated file path.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-02-19 20:58:51 +08:00
zhagnlu e8a6f1ea2b
fix: erase pk empty check when pk index replace raw data (#30432) 
#30350

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-02-07 14:56:47 +08:00
Jiquan Long a587450e56
enhance: [skip-e2e] disable asan (#30498) 
fix: #30511 
/kind improvement

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-02-04 21:25:05 +08:00
Jiquan Long e549148a19
enhance: full-support for wildcard pattern matching (#30288) 
issue: #29988 
This pr adds full-support for wildcard pattern matching from end to end.
Before this pr, the users can only use prefix match in their expression,
for example, "like 'prefix%'". With this pr, more flexible syntax can be
combined.

To do so, this pr makes these changes:
- 1. support regex query both on index and raw data;
- 2. translate the pattern matching to regex query, so that it can be
handled by the regex query logic;
- 3. loose the limit of the expression parsing, which allows general
pattern matching syntax;

With the support of regex query in segcore backend, we can also add
mysql-like `REGEXP` syntax later easily.

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-02-01 12:37:04 +08:00
xige-16 e9fdd2475d
fix: fix searchPlan metricType modified concurrently (#30227) 
issue: #30225
/kind bug
Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-26 14:03:09 +08:00
yihao.dai c02fb64ad6
enhance: Allows proactive warming up of chunk cache (#30182) 
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.

issue: https://github.com/milvus-io/milvus/issues/30181

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-25 19:55:39 +08:00
zhagnlu 8c58d9af67
enhance: optimize marisa trie range search for performance (#30079) 
#30078
#29986

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-01-25 10:07:00 +08:00
Patrick Weizhi Xu 0907d76253
enhance: pass partition key scalar info if enabled when build vector index (#29931) 
issue: #29892 

Pass optional scalar IVF offsets to Cardinal

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-01-24 00:04:55 +08:00
cai.zhang 6cf2f09b60
feat: Support tencent cloud object storage for milvus (#30163) 
issue: #30162

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-01-23 11:28:56 +08:00
MrPresent-Han 4436effdc3
enhance: support groupby based on scalar-index(#29965) (#30091) 
related: #29965

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-01-22 10:50:54 +08:00
Xu Tong e429965f32
Add float16 approve for multi-type part (#28427) 
issue：https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
 2024-01-11 15:48:51 +08:00
congqixia d6429933a7
enhance: make Load process traceable in querynode & segcore (#29858) 
See also #29803

This PR:
- Add trace span for `LoadIndex` & `LoadFieldData` in segment loader
- Add `TraceCtx` parameter for `Index.Load` in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-10 21:58:51 +08:00
zhagnlu 601a8b801b
fix: add move cursor function to physical expr (#29603) 
#29570

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-01-09 17:08:48 +08:00
zhenshan.cao 60e88fb833
fix: Restore the MVCC functionality. (#29749) 
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-01-09 11:38:48 +08:00
Jiquan Long e9f3df3626
fix: inverted index file not found (#29695) 
issue: https://github.com/milvus-io/milvus/issues/29654

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-01-07 20:26:49 +08:00
zhagnlu d07197ab1a
enhance: add compare simd function (#29432) 
#26137

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-01-07 20:20:57 +08:00
MrPresent-Han 9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983) 
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-01-05 15:50:47 +08:00
PowderLi c8db36a63a
enhance: get a blob to check object storage config (#29703) 
issue: #29672
the storage account need privileges of actions
`Microsoft.Storage/storageAccounts/blobServices/containers/blobs/*` at
least

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-01-05 14:50:46 +08:00
yah01 99e0f1e65a
enhance: unable to compile C++ tests (#29616) 
The tests need to call a private method, Milvus uses `#define` to
replace private with public, the hack trick works but would be broken if
the including order changed.

This uses friend to make all things work well

Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-04 13:20:46 +08:00
Jiquan Long 3f46c6d459
feat: support inverted index (#28783) 
issue: https://github.com/milvus-io/milvus/issues/27704

Add inverted index for some data types in Milvus. This index type can
save a lot of memory compared to loading all data into RAM and speed up
the term query and range query.

Supported: `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BOOL`
and `VARCHAR`.

Not supported: `ARRAY` and `JSON`.

Note:
- The inverted index for `VARCHAR` is not designed to serve full-text
search now. We will treat every row as a whole keyword instead of
tokenizing it into multiple terms.
- The inverted index don't support retrieval well, so if you create
inverted index for field, those operations which depend on the raw data
will fallback to use chunk storage, which will bring some performance
loss. For example, comparisons between two columns and retrieval of
output fields.

The inverted index is very easy to be used.

Taking below collection as an example:

```python
fields = [
		FieldSchema(name="pk", dtype=DataType.VARCHAR, is_primary=True, auto_id=False, max_length=100),
		FieldSchema(name="int8", dtype=DataType.INT8),
		FieldSchema(name="int16", dtype=DataType.INT16),
		FieldSchema(name="int32", dtype=DataType.INT32),
		FieldSchema(name="int64", dtype=DataType.INT64),
		FieldSchema(name="float", dtype=DataType.FLOAT),
		FieldSchema(name="double", dtype=DataType.DOUBLE),
		FieldSchema(name="bool", dtype=DataType.BOOL),
		FieldSchema(name="varchar", dtype=DataType.VARCHAR, max_length=1000),
		FieldSchema(name="random", dtype=DataType.DOUBLE),
		FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=dim),
]
schema = CollectionSchema(fields)
collection = Collection("demo", schema)
```

Then we can simply create inverted index for field via:

```python
index_type = "INVERTED"
collection.create_index("int8", {"index_type": index_type})
collection.create_index("int16", {"index_type": index_type})
collection.create_index("int32", {"index_type": index_type})
collection.create_index("int64", {"index_type": index_type})
collection.create_index("float", {"index_type": index_type})
collection.create_index("double", {"index_type": index_type})
collection.create_index("bool", {"index_type": index_type})
collection.create_index("varchar", {"index_type": index_type})
```

Then, term query and range query on the field can be speed up
automatically by the inverted index:

```python
result = collection.query(expr='int64 in [1, 2, 3]', output_fields=["pk"])
result = collection.query(expr='int64 < 5', output_fields=["pk"])
result = collection.query(expr='int64 > 2997', output_fields=["pk"])
result = collection.query(expr='1 < int64 < 5', output_fields=["pk"])
```

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-31 19:50:47 +08:00
zhagnlu 79c417b14e
fix: pass active count to query context instead of timestamp (#29541) 
#29319

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-31 16:08:48 +08:00
Jiquan Long 6f4791da0b
fix: panic in concurrent insert/query scenario (#29408) 
issue: https://github.com/milvus-io/milvus/issues/29405

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-26 15:10:48 +08:00
cqy123456 4c979538a4
enhance: update cagra index params in config and add params check (#29045) 
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
 a.milvus yaml config support gpu memory settings

 b.add cagra-params check

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
 2023-12-26 11:04:47 +08:00
zhagnlu a6eb7e5f9a
enhance: skip segment when using pk in (..) expr (#29394) 
#29293

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-21 20:06:42 +08:00
Gao 9b52cb6417
enhance: improve reducing results when many segments are filtered (#29073) 
Do not fill the invalid ids for the empty results, it will incur useless
memory overhead and reduce overhead when nq and topk is large.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-20 12:56:42 +08:00
zhagnlu a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
congqixia dcb662d9ed
enhance: Refine C.NewSegment response and handle exception (#28952) 
See also #28795

Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-07 13:34:35 +08:00
cai.zhang fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data (#28888) 
issue: #28886

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-06 20:36:36 +08:00
Bingyi Sun 36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
PowderLi 20fc90c591
enhance: find collection schema from cache (#28782) 
issue: #28781 #28329

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-03 19:22:33 +08:00
yah01 d69440524b
fix: bypass growing index if no index meta (#28791) 
we shouldn't panic if no index meta, just skip building it
fix #28022

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-30 14:10:27 +08:00
congqixia 1dc086496f
fix: schema->size() check logic with system field (#28802) 
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-29 22:40:28 +08:00
zhagnlu 0d9d098186
enhance: Add precheck when chunk manager init (#28330) 
#28329

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 19:56:32 +08:00
Bingyi Sun e8c5e75786
enhance: ignore proto generated files (#28537) 
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-22 10:20:22 +08:00
PowderLi a1c505dbd5
add internal storage metrics (#28278) 
/kind improvement
issue: #28277

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-19 17:22:25 +08:00
yah01 f7d2ab6677
enhance: reduce 1x copy for variable length field while retrieving (#28345) 
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-15 18:08:20 +08:00
MrPresent-Han 836f300536
support skip-index based on chunk-metrics to accelerate expr filter(#27925) (#28297) 
related: #27925

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-15 11:20:19 +08:00
Xu Tong 8ec85f5f4c
Add template for VectorMemIndex (#28324) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-11-11 13:20:22 +08:00
yah01 267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment (#28323) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-10 15:44:22 +08:00
cai.zhang 01c5484817
No hit when the index exceeds the array length (#28301) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-10 10:04:19 +08:00
PowderLi 9f9726f8b7
print azure sdk log (#28240) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-08 17:50:18 +08:00
cai.zhang 2b5f632fa4
Fix bug for constructing ArrayView with fixed-length type (#28185) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-07 23:38:21 +08:00
PowderLi 7bb0fa9c70
reduce useless ObjectExists (#28156) 
replace ListBlobs() with GetProperties()
unified style std::string& / char*
config azure requestTimeoutMs

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-07 16:32:20 +08:00
yah01 863e26969a
Reduce 1x memory copy for retrieving data (#28106) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-06 17:48:19 +08:00
yah01 dc89730a50
Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
Enwei Jiao f8dd589755
Refactor collection's cgo call (#28055) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-02 13:02:13 +08:00
Enwei Jiao 8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
cqy123456 4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00
yihao.dai ab6b0103a3
Get vector concurrently (#27838) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-30 15:44:12 +08:00
Enwei Jiao 4a33391b8f
rename createindex (#27903) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-27 10:12:14 +08:00
Alexander Guzhva fbaec4c1a2
Fix non-removed 2 GB temporary file in test_utils.cpp (#27651) 
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2023-10-18 02:24:09 +08:00
PowderLi 09d8b76048
[restful] new context with grpc metadata (#27668) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-10-17 20:00:14 +08:00
Xiaofan d83869aaeb
Refine minio chunks manager (#27510) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-13 14:15:35 +08:00
Gao 7a65b6fb85
Limit faiss ivf index build thread num and fix ut (#27567) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-10-11 10:33:33 +08:00
yihao.dai 106c17f304
Make read ahead policy in ChunkCache configurable (#27291) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 15:47:27 +08:00
Enwei Jiao b80a3e19d3
Add code for PanicInfo (#27364) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-27 12:01:28 +08:00
foxspy 5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
foxspy 370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
yah01 93e2eb78c9
Delete only if primary keys exist (#25292) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-20 19:03:25 +08:00
cai.zhang a362bb1457
Support array datatype (#26369) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-19 14:23:23 +08:00
PowderLi 4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
yihao.dai bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
Enwei Jiao 0afdfdb9af
Remove other Exceptions, keeps SegcoreError only (#27017) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-14 14:05:20 +08:00
yah01 3d05ddf505
Reduce cpp test time (#27043) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-13 15:41:18 +08:00
cqy123456 0ff4ddc76c
remove VectorMemNMIndex (#27000) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-09-12 17:13:18 +08:00
Enwei Jiao c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
Xu Tong 9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
Cai Yudong 8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-08-31 20:07:00 +08:00
yah01 ba882b49b6
Optimize query/search on growing segment while output vector field (#26542) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-24 09:46:24 +08:00
zhagnlu 0cb3f88e73
Fix unit test case of thread pool (#26501) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-21 21:50:22 +08:00
zhagnlu 79ca04b995
fix unstable test case for thread pool (#26465) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-18 22:42:20 +08:00
cai.zhang c073aa0dc3
Fix bug for json_contains_all has multiple array elements (#26446) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-08-18 22:36:19 +08:00
cqy123456 fd37860e57
init knowhere build/search thread pool; (#26449) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-08-18 14:00:19 +08:00
zhagnlu bdc8c507ea
Refactor thread pool to dynamic threads mode (#26407) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-18 11:58:19 +08:00
xige-16 50a77ef1f7
Fix serialize string index failed when greater than 2G (#26392) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-17 21:16:19 +08:00
zhagnlu 709352f96c
Repeat multi times for performace compare test because of cpu cache (#26394) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-17 15:18:17 +08:00
Jiquan Long 95bcf3e900
Fix error message of proto parser not clear (#26365) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-16 20:20:17 +08:00
Patrick Weizhi Xu 09da953a19
Use Knowhere AIO Context Init Defalut Value and Panic when Fail (#26286) 
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2023-08-13 20:53:30 +08:00
xige-16 5b8d716cbc
Add ut for growing segment load binlog (#26268) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-13 20:41:31 +08:00
cai.zhang a0198ce8ae
Support json contains feature (#25384) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-08-11 17:09:30 +08:00
zhagnlu 9489e14000
Optimize multi logical exprs performance when meet some situations (#26265) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 15:31:29 +08:00
zhagnlu 411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 10:37:36 +08:00
Enwei Jiao ca1349708b
Remove time travel ralted testcase (#26119) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-10 18:53:17 +08:00
xige-16 1055c90456
Add default retrieve limit (#24782) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-10 14:11:15 +08:00
yah01 300fef446b
Enable mmap for vector index (#25877) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-10 13:59:15 +08:00
Jiquan Long 6578a27fe7
Support empty expression (#26153) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-10 08:37:15 +08:00
Jiquan Long bafb183a2b
Optimize bitset usage (#26096) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-03 15:25:09 +08:00
MrPresent-Han 5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 09:31:07 +08:00
Jiquan Long 5c1f79dc54
Push down the limit operator to segcore (#25959) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-01 20:29:05 +08:00
yah01 9618bd9b42
Set channel capacity before consuming it (#25895) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-26 17:35:01 +08:00
foxspy 5437fcce8e
fix range search (#25880) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-07-25 10:07:01 +08:00
yah01 53c3bf053e
Fix unstable sealed segment bruteforce unittest (#25867) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-07-25 09:05:00 +08:00
Cai Yudong 9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-19 16:16:58 +08:00
zhagnlu 65cb52d06b
Support dynamic simd framework and using term expr as example (#25260) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-13 16:22:30 +08:00
yah01 dd5f896dc8
Load batch by batch (#25212) 
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 13:58:27 +08:00
Jiquan Long c2a2ece551
Fix expression on integer overflow case (#25320) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-07-05 10:18:24 +08:00
foxspy 31173727b2
growing segment index memory opt & get vector bugfix (#25272) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-07-05 00:04:25 +08:00
Enwei Jiao 816158e4af
Remove outdated searchplan (#25282) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-04 18:30:25 +08:00
xige-16 04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
PowderLi 3f4356df10
fix the spelling of `field` (#25008) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-06-21 14:00:42 +08:00
yah01 a413842e38
Fix deleted data is still visible (#24849) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-16 17:16:41 +08:00
zhagnlu c5b1533fdc
Change segcore search_id from travelling all bits to select true bits(#24659) (#24800) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-06-16 16:48:44 +08:00
cqy123456 a519213316
Update knowhere version, update diskann api and generate cache nodes in build process (#24898) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-06-16 14:20:39 +08:00
Jiquan Long 26a6e1b946
Fix BinaryRange expression on integer overflow case (#24920) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-16 09:56:39 +08:00
zhagnlu f60b839127
Support element in json array in segcore part(#24677) (#24829) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-06-14 14:38:37 +08:00
Jiquan Long c6a6998ba7
Fix term expression on interger overflow case (#24867) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-14 11:44:38 +08:00
Cai Yudong 1b3c4b26f1
Optimize range search result sort in segcore (#24837) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-06-13 19:22:38 +08:00
yah01 ceda0ed598
Optimize the performance of filter by JSON field (#24268) 
- Construct JSON pointer only once
- Avoid copying nested path for each row

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-22 00:47:25 +08:00
yah01 c75e7a5d05
Fix failed to compare int value with double value (#24229) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-05-19 12:57:23 +08:00
zhagnlu 113f9a0ebc
Support SIMD of several Expr (#23715) (#23717) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-05-12 14:11:20 +08:00
cai.zhang 9715a850fa
Support expr with json field (#23804) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-05-10 10:19:19 +08:00
yah01 62eea5286f
Support to filter with json expr (#23739) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-30 20:36:39 +08:00
yah01 60fdd7e4f4
Introduce simdjson (#23644) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-26 10:30:34 +08:00
foxspy 6f4ed517de
add growing segment index (#23615) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-04-26 10:14:41 +08:00
yihao.dai 092d743917
Add support for getting vectors by ids (#23450) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-23 09:00:32 +08:00
yah01 546080dcdd
Support to retrieve json (#23563) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-21 11:46:32 +08:00
Cai Yudong 5f4673fd16
Optimize unittest to save runtime (#23248) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-04-07 14:20:29 +08:00
Cai Yudong ef63e64ded
Remove ANNOY index type (#23189) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-04-04 16:30:27 +08:00
Enwei Jiao 50dab2d394
Add tracing for retrieve (#23033) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-29 18:10:02 +08:00
Enwei Jiao ef57ba3bac
Remove easylogging and replace by GLog (#22979) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-27 18:22:02 +08:00
yah01 081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Cai Yudong 0e9a4478e3
Remove useless index mode (#22934) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-23 21:39:59 +08:00
Enwei Jiao fd974df09f
OTEL integration in cpp (#22175) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-21 16:31:57 +08:00
zhagnlu 7c633e9b9d
Support aliyun OSS for chunk manager (#22663) (#22842) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-03-20 14:45:57 +08:00
xige-16 9aa99aedbb
[Cherry-Pick] Remove arrow uasge in FieldData (#22726) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-03-20 10:41:56 +08:00
Jiquan Long 8139106b51
Feat: count entities by expression (#22765) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-16 19:31:55 +08:00
Cai Yudong ab3cbdfc61
Partial change to prepare for GPU index type support (#22591) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-03-14 23:21:56 +08:00
Jiquan Long a36fefb009
Fix cpplint (#22657) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-10 09:47:54 +08:00
XuanYang-cn 955bc06165
Fix search binary pagination failure (#22477) 
See also: #22168

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-08 11:03:51 +08:00
smellthemoon 0dae20c938
Layer range search results (#22539) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-03-03 15:19:48 +08:00
yah01 7478e44911
Support using mmap to load data (#22052) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-01 18:07:49 +08:00
smellthemoon 8d05874ff9
Fix get index mode error (#22364) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-27 14:45:47 +08:00
smellthemoon 820d0bc060
Fix range search error (#22352) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-02-23 14:21:46 +08:00
smellthemoon 9e0ec15436
Support range search (#21652) 
Signed-off-by: smellthemoon <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
 2023-02-21 09:48:32 +08:00
xige-16 8ec5ba13dc
Fix search failed on disk index when search_list equal to limit (#21113) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-02-16 11:02:34 +08:00
presburger 9950cacd10
support knowhere 2.0 (#21857) 
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
 2023-02-10 14:24:32 +08:00
yah01 73ce87dfe5
Fix reduce decreasing recall (#21981) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-02-06 11:23:53 +08:00
yah01 836773f1a0
Optimize performance of reducing segments (#21722) 
- Improve the performance of reducing from O(knlogn) to O(nlogk)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-16 15:25:42 +08:00
smellthemoon bf3c02155a
Support update (#20875) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-01-04 17:21:36 +08:00
Jiquan Long d7156812c1
Try using ASAN in ci ut (#21089) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-12-29 15:29:31 +08:00
smellthemoon 1e5cd9dc6e
Modify comment (#21362) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-12-26 15:27:30 +08:00
xige-16 a20770c172
Delete logs that print sensitive information (#20889) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-12-01 10:35:16 +08:00
Enwei Jiao 958e94f6f0
Use Conan as c++ package manager (#19920) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-23 10:39:11 +08:00
xige-16 34c88cea32
Fix GetDiskUsedSize not work (#20474) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 17:47:17 +08:00
smellthemoon f5ab719f21
timestamp decided if the pks were the same (#20166) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-21 10:55:10 +08:00
xige-16 4a66965df4
Delete RAW_DATA copy when load IVF_FLAT index data (#20274) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-05 17:33:05 +08:00
zhagnlu cb2591d1fe
Parallel operations of disk-ann file manager (#20120) (#20122) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-11-03 14:39:40 +08:00
zhagnlu 02a8a07322
Fix BucketExists bug in minio chunk manager (#19548) (#19854) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-10-20 10:23:27 +08:00
xige-16 158787811e
Move assemble/disassemble func to core (#19420) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-16 21:17:25 +08:00
xige-16 a1db9038fb
Move disk index params to config file (#19714) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 17:51:24 +08:00
xige-16 8c9c1672ae
Assign different storage config for indexes (#19517) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 14:45:23 +08:00
congqixia 089912baa7
Fix segcore unit test name (#19783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-14 14:29:24 +08:00
Cai Yudong 87d78a4a85
Ignore cases when comparing metric type in segcore (#19437) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-26 17:58:52 +08:00
aoiasd 2b58bd5c0a
Optimize large memory usage of InsertRecord by using vector instead of unordered_map if InsertRecord used in sealed segment (#19245) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-09-23 17:08:51 +08:00
xige-16 428840178c
Support diskann index for vector field (#19093) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-21 20:16:51 +08:00
aoiasd 58d3a49b62
Optimize large memory usage of Variant used in InsertRecord (#19197) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-09-16 10:52:47 +08:00
xige-16 4de1bfe5bc
Add cpp data codec (#18538) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-09 22:12:34 +08:00
Cai Yudong 7b5eb85205
Remove redundant parameter for segcore Search() (#19025) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-06 12:05:11 +08:00
Cai Yudong 765907ab77
Optimize segcore Reduce (#18902) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-06 10:55:12 +08:00
Cai Yudong 686b0ce796
Upgrade to knowhere-v1.3.0, remove following index support: (#18935) 
- IVF_SQ8H
- RHNSW_FLAT/RHNSW_PQ/RHNSW_SQ
- NGT
- NSG
- SPTAG

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-05 10:41:11 +08:00
Cai Yudong da96659569
[skip e2e] Add more testcases with different parameter combinations in test_reduce (#18967) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-01 16:59:02 +08:00
Cai Yudong 4ded45394a
[skip e2e] Optimize test_reduce (#18957) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-01 11:09:00 +08:00
Cai Yudong 9dc3bbecbd
Change the type of slice_nqs and slice_topks from int32_t[] to int64_t[] (#18867) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-29 11:36:56 +08:00
Letian Jiang c6d9d0df5e
Fix segcore benchmark (#18675) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-08-17 20:24:49 +08:00
Cai Yudong d878cc2ec0
Rename real_topK_per_nq_ to topks_per_nq_prefix_sum_ in SearchResult struct (#18634) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-16 00:32:48 +08:00
zhenshan.cao a287a2b3fd
Return empty result in advance if all data filtered out (#18329) (#18438) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-28 12:36:30 +08:00
bigsheeper cef8b1e7cc
Enable jemalloc (#18349) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-20 22:22:31 +08:00
Jeng.Gwan 638f6c36e9
Support to get real row count of segment (#18115) 
Signed-off-by: xaxys <zheng.guan@zilliz.com>
 2022-07-18 09:58:28 +08:00
Cai Yudong 015a2f0866
Let growing segment call knowhere brute search API (#18227) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-07-12 11:58:25 +08:00
bigsheeper 22508f36d3
Get search field id from search plan and log if loaded index when search segments (#18183) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-08 20:18:22 +08:00
zhagnlu 257da153ce
Fix core dump when nq has no topk result (#17923) (#18051) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-07-05 19:48:20 +08:00
xige-16 0702993106
Fix query not empty after delete string (#17953) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-07-04 16:54:20 +08:00
Jiquan Long 6954a5ba3e
Fix search successfully with invalid metric type (#17977) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-07-01 22:28:23 +08:00
bigsheeper 641fb96958
Return malloc_trim result and log if malloc_trim successfully (#17959) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-07-01 10:44:18 +08:00
Cai Yudong a001412e12
Replace faiss::MetricType with knowhere::MetricType (#17891) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-29 14:20:19 +08:00
Letian Jiang 4ae1ca2cac
Fix arrow builder nullptr check in FinishPayloadWriter (#17873) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-06-28 20:04:17 +08:00
Enwei Jiao 16c3aedc15
refine complie configuration (#17502) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-24 21:12:15 +08:00
Cai Yudong 7385770014
Upgrade to knowhere-v1.1.12 (#17692) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-24 10:34:18 +08:00
bigsheeper 1722431699
Checking for existence in map in segcore (#17700) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-22 18:16:14 +08:00
Jiquan Long fdccfb4273
Fix ut heap overflow (#17706) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-22 16:56:13 +08:00
bigsheeper 92d06b2e30
Purge memory by the memory state and try to purge after each search (#17565) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-17 17:46:10 +08:00
xige-16 36ad989590
Fix segOffset grater than insert barrier when mark delete (#17444) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-10 20:02:08 +08:00
bigsheeper f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore (#17389) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 21:34:05 +08:00
Jiquan Long e88ffb8a57
Enable marisa trie ut on MacOS (#17316) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-06-02 10:48:03 +08:00
xige-16 56778787be
Reverse data from scalar index (#17145) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 14:58:01 +08:00
xige-16 7d810ac159
Fix query empty when insert same pk after deletion (#17222) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-25 22:06:00 +08:00
xige-16 a8829554eb
Merge multiple sets of expr's optypes in diff dirs (#17192) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-24 21:56:00 +08:00
zhenshan.cao 08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-23 16:41:58 +08:00
Jiquan Long 77b0f290a8
Fix prefix match on sealed segment (#17050) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-05-17 20:15:55 +08:00
xige-16 935b729a0c
Fix multiple deletes not taking effect (#17033) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-16 18:39:56 +08:00