66d679ecbb
fix: clear binlog files in CleanData ( #31039 )
...
issue: https://github.com/milvus-io/milvus/issues/31042
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-20 11:11:07 +08:00
6edd06083f
chore: remove repetitive words ( #31153 )
...
Signed-off-by: gcmutator <329964069@qq.com>
2024-03-20 10:17:07 +08:00
b35ecebcc3
enhance: Update Knowhere version ( #31392 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-03-19 15:11:07 +08:00
d7727dd087
enhance: fetch simdjson directly in the format of targz ( #31369 )
...
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-03-18 18:55:11 +08:00
1c930e560c
enhance: Update Knowhere version ( #31312 )
...
/kind branch-feature
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-03-18 11:29:04 +08:00
038c570ef3
enhance: upgrade folly to run on arm ( #31284 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-03-15 15:39:03 +08:00
6939ad15f2
fix:possible out-of-bound due to groupby when reduing( #30711 ) ( #31200 )
...
related: #30711
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-14 13:07:03 +08:00
7fc3094a42
fix: fix growing index data race and properly handle build error ( #31170 )
...
issue: https://github.com/milvus-io/milvus/issues/31169
also properly handling index build error by re-create a new index so
that nothing will be left in the previous failed index build attempt.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 20:19:04 +08:00
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id ( #30629 )
...
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-12 09:16:30 -07:00
c8b54f321a
fix:restrict pk in [...] optimization situations ( #31184 )
...
#31154
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-12 14:49:03 +08:00
6a83f16871
feat: Support for multiple forms of JSON ( #31052 )
...
issue: #31051
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-11 19:55:02 +08:00
070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building ( #30357 )
...
This commit adds sparse float vector support to segcore with the
following:
1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built
This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.
https://github.com/milvus-io/milvus/issues/29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-11 14:45:02 +08:00
a99143dd52
fix: Save traceID and spanID as hex string into search config ( #31071 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-11 14:21:01 +08:00
53af6d8c59
[automated] Update Knowhere Commit ( #31151 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-09 01:55:02 +08:00
122981aeb9
fix: Disable knowhere trace as a quick fix ( #31055 )
...
Issue: #30961
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-08 15:27:01 +08:00
3574bdf858
enhance: ban range-search iteration for search-group-by ( #30824 )
...
related: #30033
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-08 14:17:00 +08:00
19c64067af
enhance: jemalloc aarch64 platform use 64k pagesize. ( #29522 )
...
enhance: jemalloc aarch64 platform use 64k pagesize.
issue: #28843
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2024-03-07 21:01:01 +08:00
2d9de233fc
[automated] Update Knowhere Commit ( #31089 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-07 12:05:02 +08:00
c047f09110
[automated] Update Knowhere Commit ( #31015 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-05 16:31:00 +08:00
4bda6c33ad
fix: binary vector should not limit dimension to 32768 ( #30676 )
...
all the vector dimension check should happen on collection creation but
not index build
fix #30285
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-03-05 14:21:00 +08:00
3dc5e38240
[automated] Update Knowhere Commit ( #30989 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-04 16:34:59 +08:00
29f44f840a
enhance: refine groupBy error msg( #29968 ) ( #30920 )
...
related: #29968
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-01 18:53:03 +08:00
1aa97a5c21
enhance: Support more relational operators for binary expressions ( #30902 )
...
issue: #30677
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 16:57:00 +08:00
e2f35954d4
enhance: support pattern matching on json field ( #30779 )
...
issue: https://github.com/milvus-io/milvus/issues/30714
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 18:31:00 +08:00
16b785e149
enhance: optimize the memory usage and speed up loading variable length data ( #30787 )
...
/kind improvement
this removes the 1x copying while loading variable length data, also
avoids constructing std::string, which could lead to memory
fragmentation
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: yah01 <yah2er0ne@outlook.com>
2024-02-28 16:45:00 +08:00
4459078e0b
fix: wrong num_entities used when mmap variable length data ( #30848 )
...
https://github.com/milvus-io/milvus/issues/30728
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 16:38:56 +08:00
a115b731ed
enhance: fix old pr cpp format issue ( #30894 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 16:28:20 +08:00
f658dd5faa
enhance: update knowhere version to 60a5c9c ( #30788 )
...
/kind improvement
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-02-28 14:18:55 +08:00
8a219e0102
feat: Support knowhere trace using OpenTelemetry ( #30750 )
...
Issue: #21508
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
6e9f3ea531
[automated] Update Knowhere Commit ( #30744 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-28 10:50:57 +08:00
57397b1307
enhance: add new LRU cache impl ( #30360 )
...
- remove the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache
related #30361
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
3e82d21ca1
enhance: reduce 1x memory copy when loading json ( #30753 )
...
/kind improvement
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-27 10:18:55 +08:00
e2330f02f8
fix: pattern match use incorrect raw data ( #30764 )
...
issue: https://github.com/milvus-io/milvus/issues/30687
We store all the varchar datas in an continuous address and use
string_view to quickly find them. In this case, using string_view.data()
directly will point to all rest varchar datas.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:56:52 +08:00
77eb6defb1
feat: support groupby on growing and non-indexed sealed egment( #30307 ) ( #30644 )
...
related: #30308
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-21 14:02:53 +08:00
18aac076de
fix: move test from NEON to X86 ( #30324 )
...
#26137
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-21 11:58:53 +08:00
0118bef2a2
fix: replace sse2 simd interface with older version ( #30668 )
...
#30667
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-21 10:04:54 +08:00
976b6fc0e4
enhance: change opendal as compile configurable ( #30384 )
...
#30373
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-20 19:16:52 +08:00
b74673c147
enhance: calculate the accuracy memory usage while loading segment ( #30473 )
...
the old version Knowhere would copy the index data while loading, we
need to consider this to avoid OOM.
Knowhere provides a util function to indicate whether it will load the
index with disk, if not, we need to double the memory usage prediction
for index data
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-20 14:52:51 +08:00
43e8cd531d
enhance: Update Knowhere version ( #30675 )
...
issue: #30669
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2024-02-19 22:04:51 +08:00
18c351efa6
fix: Prevent ChunkCache use absolute path in All-in-one mode ( #30666 )
...
See also #30651
Append operator of `std::filesystem::path` will replace whole path when
the param of "/" operation is an absolute path.
In "All-in-one" mode, this shall cause ChunkCache removing the original
vector data file when building chunk cache during/after load procedure.
This PR changes the ChunkCache path generation logic to a separate
function in which will check whether the file path is absolute or not.
If the file path is absolute, it removes the root path prefix and return
concatenated file path.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-19 20:58:51 +08:00
5bb28a9ea4
enhance: Print out range_filter and radius when range search param check fail ( #30623 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-18 15:40:48 +08:00
a297baae9d
enhance: remove unused code ( #30601 )
...
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-02-13 10:26:47 +08:00
e8a6f1ea2b
fix: erase pk empty check when pk index replace raw data ( #30432 )
...
#30350
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-07 14:56:47 +08:00
92d1d744ae
fix: groupby results lack good results( #29883 ) ( #30428 )
...
related: #29883
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-06 17:08:34 +08:00
5449e862d5
fix: safety access unordered_map and remove some useless code excute ( #30504 )
...
issue: https://github.com/milvus-io/milvus/issues/30358 and
https://github.com/milvus-io/milvus/issues/30491
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-02-05 22:03:09 +08:00
ebbe32df9a
[automated] Update Knowhere Commit ( #30515 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-05 01:32:44 +08:00
a587450e56
enhance: [skip-e2e] disable asan ( #30498 )
...
fix : #30511
/kind improvement
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-04 21:25:05 +08:00
20c9cfc587
[automated] Update Knowhere Commit ( #30487 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-04 01:23:04 +08:00
e549148a19
enhance: full-support for wildcard pattern matching ( #30288 )
...
issue: #29988
This pr adds full-support for wildcard pattern matching from end to end.
Before this pr, the users can only use prefix match in their expression,
for example, "like 'prefix%'". With this pr, more flexible syntax can be
combined.
To do so, this pr makes these changes:
- 1. support regex query both on index and raw data;
- 2. translate the pattern matching to regex query, so that it can be
handled by the regex query logic;
- 3. loose the limit of the expression parsing, which allows general
pattern matching syntax;
With the support of regex query in segcore backend, we can also add
mysql-like `REGEXP` syntax later easily.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-01 12:37:04 +08:00
5cf9bb236e
enhance: restful support import jobs ( #30343 )
...
issue: #28521 #29732
include
1. list collection's import jobs
2. create a new import job
3. get the progress of an import job
fix:
1. mix the order of dbName & collectionName #29728
2. trace log keep the same as v1
3. support traceID
4. azure precheck, blob name cannot end with / #29703
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-31 17:57:04 +08:00
878c4c9463
enhance: limit the max pool size to 16 ( #30371 )
...
according to our benchmark, concurrency level 16 is enough to fully
utilize the object storage network bandwidth
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-31 14:13:06 +08:00
74cfba0249
enhance:limit binlog index rows num ( #30173 )
...
issue: https://github.com/milvus-io/milvus/issues/27678
also relate issue: https://github.com/milvus-io/milvus/issues/30065
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-29 19:49:02 +08:00
0542a0e7dc
[automated] Update Knowhere Commit ( #30332 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-29 01:05:01 +08:00
aeb1e36f00
enhance: change plan desc log from info to debug ( #30304 )
...
#30172
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-28 16:04:38 +08:00
e9fdd2475d
fix: fix searchPlan metricType modified concurrently ( #30227 )
...
issue: #30225
/kind bug
Signed-off-by: xige-16 <xi.ge@zilliz.com>
---------
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-26 14:03:09 +08:00
116d0f20b8
fix: groupby bug for ut ( #30272 )
...
related: #29965
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-25 20:57:00 +08:00
c02fb64ad6
enhance: Allows proactive warming up of chunk cache ( #30182 )
...
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.
issue: https://github.com/milvus-io/milvus/issues/30181
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-25 19:55:39 +08:00
a27c0e86fd
enhance: reduce many I/O operations while loading disk index ( #30189 )
...
before this, every time writting the index chunk data into the disk,
there are 4 I/O operations:
- open the file
- seek to the offset
- write the data
- close the file
this optimized this to open only once and continiously write all data.
This also makes it concurrent to load the files from object storage
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-25 15:23:02 +08:00
8c58d9af67
enhance: optimize marisa trie range search for performance ( #30079 )
...
#30078
#29986
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-25 10:07:00 +08:00
0907d76253
enhance: pass partition key scalar info if enabled when build vector index ( #29931 )
...
issue: #29892
Pass optional scalar IVF offsets to Cardinal
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
42bb4e37e5
fix:diskann search crash when search list = 9999999999 ( #30185 )
...
issue: https://github.com/milvus-io/milvus/issues/29020
Json can't not pass a max_int32 value to int32_t, so let knowhere check
value range by itself.
After fix this, pymilvus will report:
pymilvus.exceptions.MilvusException: <MilvusException: (code=65535,
message=fail to search on QueryNode 6: worker(6) query failed: => failed
to search: arithmetic overflow: param search_list_size should be at most
2147483647)>
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-23 14:46:55 +08:00
6cf2f09b60
feat: Support tencent cloud object storage for milvus ( #30163 )
...
issue: #30162
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-23 11:28:56 +08:00
a77693aa19
enhance: convert the `GetObject` util to async ( #30166 )
...
This makes it much easier to use
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 19:20:57 +08:00
e967949cc5
[automated] Update Knowhere Commit ( #30120 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-22 18:40:54 +08:00
4436effdc3
enhance: support groupby based on scalar-index( #29965 ) ( #30091 )
...
related: #29965
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-22 10:50:54 +08:00
aee19dcd6b
enhance: Opt vector dimension mismatch error message ( #29928 )
...
issue: https://github.com/milvus-io/milvus/issues/29791
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>
---------
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-19 17:52:54 +08:00
f542bdbf3c
enhance: calc the accurate mem size of segment ( #30093 )
...
this stats the real memory size of segment, also reduces the memory
usage in mmap mode
resolve #30095
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-19 12:32:53 +08:00
fa7cf587b0
enhance: Opt metric type does not match error message ( #29927 )
...
issue: #29791
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-17 20:25:03 +08:00
1185e4dcd5
fix: written file size is over the int32 range and raises error ( #30057 )
...
we sum the total data size in int32, which could lead to an overflow
error
related #30056
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 16:42:54 +08:00
8030b90891
fix: correct file name when loading index ( #29985 )
...
issue: #29973
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-16 10:24:52 +08:00
c31e68446e
enhance: refine groupby-performance ( #29933 )
...
related: #29844
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-15 14:12:52 +08:00
def717af55
fix: SealedIndexingEntry in SealedIndexingRecord may leak without smart pointer protect. ( #29932 )
...
may related issue: #29828
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:28:51 +08:00
e1258b8cad
feat: integrate storagev2 into loading segment ( #29336 )
...
issue: #29335
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
f2e36db488
enhance: optimize the loading index performance ( #29894 )
...
this utilizes concurrent loading
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 17:44:51 +08:00
6c477ce3a7
enhance: optimize the loading strategy ( #29910 )
...
as we have the pool size limit so we don't need to limit the concurrency
manually
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 14:26:50 +08:00
aba2656e68
fix: missing field data after appending scalar index to loaded segment ( #29912 )
...
related #29843
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 14:04:54 +08:00
4d11525f55
[automated] Update Knowhere Commit ( #29904 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-12 14:00:50 +08:00
e429965f32
Add float16 approve for multi-type part ( #28427 )
...
issue：https://github.com/milvus-io/milvus/issues/22837
Add bfloat16 vector, add the index part of float16 vector.
Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
67ab5be15a
enhance: optimize search performance of inverted index ( #29794 )
...
issue: #29793
Use `DocSetCollector` instead of `TopDocsCollector`, which will avoid
scoring and sorting.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-01-11 11:12:49 +08:00
5164d30287
fix: increase expr recursion depth to avoid parse failed ( #29860 )
...
#29759
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-11 10:26:50 +08:00
031243fee7
feat: support mmap for marisa trie ( #29613 )
...
this supports mmap for marisa trie index
related https://github.com/milvus-io/milvus/issues/21866
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-11 10:22:50 +08:00
d6429933a7
enhance: make Load process traceable in querynode & segcore ( #29858 )
...
See also #29803
This PR:
- Add trace span for `LoadIndex` & `LoadFieldData` in segment loader
- Add `TraceCtx` parameter for `Index.Load` in segcore
- Add span for ReadFiles & Engine Load for Memory/Disk Vector index
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-10 21:58:51 +08:00
cb9d9ec0f0
enhance: Correct sampleFraction's type to float ( #29810 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-01-10 13:18:50 +08:00
600f6eff06
enhance: Upgrade gtest to 1.13.0 ( #29805 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-01-10 13:16:57 +08:00
601a8b801b
fix: add move cursor function to physical expr ( #29603 )
...
#29570
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-09 17:08:48 +08:00
60e88fb833
fix: Restore the MVCC functionality. ( #29749 )
...
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:
1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.
issue: #29656
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-09 11:38:48 +08:00
9702cef2b5
feat: Support multiple vector search ( #29433 )
...
issue #25639
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
e9f3df3626
fix: inverted index file not found ( #29695 )
...
issue: https://github.com/milvus-io/milvus/issues/29654
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-01-07 20:26:49 +08:00
d07197ab1a
enhance: add compare simd function ( #29432 )
...
#26137
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-07 20:20:57 +08:00
271edc6669
fix: throw exception when upload file failed for DiskIndex ( #29627 )
...
related to : #29417
cardinal indexes upload index files in `Serialize` interface, and throw
exception when the `Serialize` failed.
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2024-01-07 20:03:13 +08:00
5dc300c4a9
fix: Fix bug for pk index doesn't have raw data ( #29711 )
...
issue: #29697
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-07 19:36:48 +08:00
9e2e7157e9
feat: support search_group_by for milvus( #25324 ) ( #28983 )
...
related: #25324
Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:
1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-05 15:50:47 +08:00
22bb84fa9d
feat:add new gpu index:GPU_BRUTE_FORCE and limit gpu index metric type ( #29590 )
...
issue: https://github.com/milvus-io/milvus/issues/29230
this pr do these things:
1. add gpu brute force;
2. limit gpu index only support l2 / ip;
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-05 15:24:48 +08:00
c8db36a63a
enhance: get a blob to check object storage config ( #29703 )
...
issue: #29672
the storage account need privileges of actions
`Microsoft.Storage/storageAccounts/blobServices/containers/blobs/*` at
least
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-05 14:50:46 +08:00
0ae90443ba
enhance: fill missed info for segcore error ( #29610 )
...
- fill missed error info
- format the error message directly
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-04 17:54:46 +08:00
99e0f1e65a
enhance: unable to compile C++ tests ( #29616 )
...
The tests need to call a private method, Milvus uses `#define` to
replace private with public, the hack trick works but would be broken if
the including order changed.
This uses friend to make all things work well
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-04 13:20:46 +08:00
5f00bad4b8
fix: link with install path's libblob-chunk-manager ( #29496 )
...
issue: #29494
1. link with install path's libblob-chunk-manager
2. performance of `ShouldBindWith` is better than `ShouldBindBodyWith`
3. the middleware shouldn't read the unrefreshed parameter repeatly
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-31 20:02:48 +08:00
3f46c6d459
feat: support inverted index ( #28783 )
...
issue: https://github.com/milvus-io/milvus/issues/27704
Add inverted index for some data types in Milvus. This index type can
save a lot of memory compared to loading all data into RAM and speed up
the term query and range query.
Supported: `INT8`, `INT16`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BOOL`
and `VARCHAR`.
Not supported: `ARRAY` and `JSON`.
Note:
- The inverted index for `VARCHAR` is not designed to serve full-text
search now. We will treat every row as a whole keyword instead of
tokenizing it into multiple terms.
- The inverted index don't support retrieval well, so if you create
inverted index for field, those operations which depend on the raw data
will fallback to use chunk storage, which will bring some performance
loss. For example, comparisons between two columns and retrieval of
output fields.
The inverted index is very easy to be used.
Taking below collection as an example:
```python
fields = [
FieldSchema(name="pk", dtype=DataType.VARCHAR, is_primary=True, auto_id=False, max_length=100),
FieldSchema(name="int8", dtype=DataType.INT8),
FieldSchema(name="int16", dtype=DataType.INT16),
FieldSchema(name="int32", dtype=DataType.INT32),
FieldSchema(name="int64", dtype=DataType.INT64),
FieldSchema(name="float", dtype=DataType.FLOAT),
FieldSchema(name="double", dtype=DataType.DOUBLE),
FieldSchema(name="bool", dtype=DataType.BOOL),
FieldSchema(name="varchar", dtype=DataType.VARCHAR, max_length=1000),
FieldSchema(name="random", dtype=DataType.DOUBLE),
FieldSchema(name="embeddings", dtype=DataType.FLOAT_VECTOR, dim=dim),
]
schema = CollectionSchema(fields)
collection = Collection("demo", schema)
```
Then we can simply create inverted index for field via:
```python
index_type = "INVERTED"
collection.create_index("int8", {"index_type": index_type})
collection.create_index("int16", {"index_type": index_type})
collection.create_index("int32", {"index_type": index_type})
collection.create_index("int64", {"index_type": index_type})
collection.create_index("float", {"index_type": index_type})
collection.create_index("double", {"index_type": index_type})
collection.create_index("bool", {"index_type": index_type})
collection.create_index("varchar", {"index_type": index_type})
```
Then, term query and range query on the field can be speed up
automatically by the inverted index:
```python
result = collection.query(expr='int64 in [1, 2, 3]', output_fields=["pk"])
result = collection.query(expr='int64 < 5', output_fields=["pk"])
result = collection.query(expr='int64 > 2997', output_fields=["pk"])
result = collection.query(expr='1 < int64 < 5', output_fields=["pk"])
```
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-31 19:50:47 +08:00
79c417b14e
fix: pass active count to query context instead of timestamp ( #29541 )
...
#29319
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-31 16:08:48 +08:00
c2345daf3a
[automated] Update Knowhere Commit ( #29578 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-29 18:56:46 +08:00
6f4791da0b
fix: panic in concurrent insert/query scenario ( #29408 )
...
issue: https://github.com/milvus-io/milvus/issues/29405
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-26 15:10:48 +08:00
b8318fcd7d
enhance: improve the handling for segcore error ( #29471 )
...
- fix lost exception details in segcore
- improve the logs of handling errors from segcore
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-26 14:06:46 +08:00
4c979538a4
enhance: update cagra index params in config and add params check ( #29045 )
...
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
a.milvus yaml config support gpu memory settings
b.add cagra-params check
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
2023-12-26 11:04:47 +08:00
fce1a8dafb
[automated] Update Knowhere Commit ( #29412 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-25 17:58:46 +08:00
aef483806d
enhance: improve the segcore logs ( #29372 )
...
- remove the streaming logging
- refine existing logs
fix #29366
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-23 21:52:43 +08:00
1b7f1d7067
enhance: mmap data corrupted after seal the column ( #29422 )
...
this bug was introduced in recent changes
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-23 15:20:43 +08:00
1cbe3cd5fc
fix: fix memory leak when cancel segcore task ( #29431 )
...
#29430
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-22 20:28:43 +08:00
a6eb7e5f9a
enhance: skip segment when using pk in (..) expr ( #29394 )
...
#29293
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-21 20:06:42 +08:00
7a2374e698
enhance: reduce the memory usage of variable length data ( #29387 )
...
add all loading data into a buffer and then copy them into the a
fit-in-size memory
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:02:42 +08:00
be87c18b44
fix: fixup data race at generate binlog index ( #29370 )
...
issue: #29339
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-12-21 14:58:49 +08:00
04b2518ae7
enhance: fix the incorrect init parameter ( #29357 )
...
as the `driver_` field is not used so this doesn't matter for now
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 20:50:43 +08:00
9b52cb6417
enhance: improve reducing results when many segments are filtered ( #29073 )
...
Do not fill the invalid ids for the empty results, it will incur useless
memory overhead and reduce overhead when nq and topk is large.
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-20 12:56:42 +08:00
8f89e9cf75
enhance: remove all unnecessary string formatting ( #29323 )
...
done by two regex expressions:
- `PanicInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
- `AssertInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
related: #28811
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 10:04:43 +08:00
89b208d27a
enhance: Fix format message ( #29159 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-20 09:30:44 +08:00
bfca0a7926
fix: refine skipIndex to resolve cyclic dependcy( #29132 ) ( #29189 )
...
related: #29132
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-19 10:26:40 +08:00
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
26409d801e
enhance: Remove omp from segcore ( #29207 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-12-15 14:00:39 +08:00
3e66e78508
[automated] Update Knowhere Commit ( #29178 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-14 17:16:39 +08:00
49b8657f95
enhance: Support implicit type conversion for parquet ( #29046 )
...
issue: #29019
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-12 16:14:44 +08:00
0e65e90338
enhance: Support otlp with insecure ( #29115 )
...
issue: https://github.com/milvus-io/milvus/issues/28914
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-12-12 11:14:37 +08:00
9d54d6f590
fix: change Abseil to shared library to solve macos compilation issue ( #28986 )
...
fix the compilation error on macos 14.0 with x86 arch processor
related to #28985
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2023-12-11 21:54:37 +08:00
464bc9e8f4
fix: fix reduce precision for search( #27325 ) ( #29031 )
...
related: #27325
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 10:04:37 +08:00
dcb662d9ed
enhance: Refine C.NewSegment response and handle exception ( #28952 )
...
See also #28795
Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-07 13:34:35 +08:00
fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data ( #28888 )
...
issue: #28886
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-06 20:36:36 +08:00
36f69ea031
feat: integrate storagev2 in building index of segcore ( #28768 )
...
issue: https://github.com/milvus-io/milvus/issues/28655
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-05 16:48:54 +08:00
f01e507b15
[automated] Update Knowhere Commit ( #28965 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-05 15:56:35 +08:00
9b6cbe956a
[automated] Update Knowhere Commit ( #28917 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-04 15:42:34 +08:00
c8b1a4618a
enhance: Resolve libunwind requirement conflict using 1.7.2 ( #28929 )
...
Try to resolve libunwind dependency requirement conflict between glog &
folly
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 14:06:40 +08:00
20fc90c591
enhance: find collection schema from cache ( #28782 )
...
issue: #28781 #28329
1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-03 19:22:33 +08:00
342635ed61
enhance: enable assert method to format arguments ( #28812 )
...
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.
make it happy
related #28811
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-01 18:04:33 +08:00
f5856812a2
fix: Fix get binary vector from chunk cache ( #28866 )
...
The way of getting binary vector size is wrong. This PR will fix it.
issue: https://github.com/milvus-io/milvus/issues/28865
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-01 14:40:32 +08:00
7206795e91
fix: update folly to resolve simd issue ( #28878 )
...
related #27552 , after this, milvus could run successfully on sse4.2
only machine
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-01 13:50:32 +08:00
8036ee13fa
feat: avoid dereferencing nullptr ( #28862 )
...
issue: #28793
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-01 10:20:32 +08:00
ecc3ca374c
[automated] Update Knowhere Commit ( #28882 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-01 02:28:31 +08:00
cac802ef7f
enhance: use already installed vcpkg ( #28703 )
...
issue #28686
1. Update Builder gpu image changes, see changes #28505
2. update azure-identity-cpp from beta to release
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-30 15:58:32 +08:00
d69440524b
fix: bypass growing index if no index meta ( #28791 )
...
we shouldn't panic if no index meta, just skip building it
fix #28022
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-30 14:10:27 +08:00
1dc086496f
fix: schema->size() check logic with system field ( #28802 )
...
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-29 22:40:28 +08:00
3b1b14dd78
fix: update binlog index memory uasge before loading segments ( #28528 )
...
issue: #27678
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-29 16:42:27 +08:00
86ccb8e146
[automated] Update Knowhere Commit ( #28704 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-24 16:56:24 +08:00
6f7a9264d5
enhance: Handle knowhere error for creare diskann index ( #28690 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-24 11:58:23 +08:00
0d9d098186
enhance: Add precheck when chunk manager init ( #28330 )
...
#28329
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:56:32 +08:00
e9e9b30e8e
enhance: Disable stdout buffer ( #28391 )
...
#28390
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:26:25 +08:00
e5ce385ffd
enhance: remove -inl.h files ( #28674 )
...
issue: https://github.com/milvus-io/milvus/issues/28673
Move template implementations from -inl.h to .cpp file and make explicit
instantiation
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:20:25 +08:00
c96d07682e
enhance: enhance the accuracy of memory usage ( #28554 )
...
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.
we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage
resolve #28553
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-23 15:12:23 +08:00
e8c5e75786
enhance: ignore proto generated files ( #28537 )
...
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-22 10:20:22 +08:00
bfccfcd0ca
enhance: refine error messages ( #28424 )
...
- Split the simple reason and full detail
- Refine existing error messages
related: #28422
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 17:02:24 +08:00
02c5a649cf
enhance: store system fields in segcore ( #28524 )
...
we need the system fields info for some usacase
fix : #28523
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 09:28:22 +08:00
a1c505dbd5
add internal storage metrics ( #28278 )
...
/kind improvement
issue: #28277
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-19 17:22:25 +08:00
cc713931e9
fix: Fix rocksdb compile failed ( #28484 )
...
Fix : #27675
/kind improvement
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-16 16:36:19 +08:00
f7d2ab6677
enhance: reduce 1x copy for variable length field while retrieving ( #28345 )
...
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-15 18:08:20 +08:00
836f300536
support skip-index based on chunk-metrics to accelerate expr filter( #27925 ) ( #28297 )
...
related: #27925
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-11-15 11:20:19 +08:00
8ec85f5f4c
Add template for VectorMemIndex ( #28324 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-11-11 13:20:22 +08:00
267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment ( #28323 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-10 15:44:22 +08:00
3920bbc55f
Force set aliyun use_virtual_host to true for all ( #28158 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-10 10:32:20 +08:00
01c5484817
No hit when the index exceeds the array length ( #28301 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-10 10:04:19 +08:00
30847cad3e
Handle exception while loading ( #28304 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-09 17:59:12 +08:00
b7b31ce0bc
Update knowhere commit ( #28285 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-09 14:24:19 +08:00
9f9726f8b7
print azure sdk log ( #28240 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-08 17:50:18 +08:00
2b5f632fa4
Fix bug for constructing ArrayView with fixed-length type ( #28185 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-07 23:38:21 +08:00
7bb0fa9c70
reduce useless ObjectExists ( #28156 )
...
replace ListBlobs() with GetProperties()
unified style std::string& / char*
config azure requestTimeoutMs
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-07 16:32:20 +08:00
b1df3ead0e
Update knowhere commit ( #28176 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-06 22:44:19 +08:00
863e26969a
Reduce 1x memory copy for retrieving data ( #28106 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-06 17:48:19 +08:00
8011054a2a
Check length before comparing strings ( #28110 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-04 10:04:29 +08:00
501cba90ce
Use IVFFLAT_CC instead of IVFFLAT for sealed segment tmp index ( #28164 )
...
See also #28152
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-03 23:50:15 +08:00
7f28e9d2f3
Update knowhere commit ( #28087 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-03 10:24:16 +08:00
fc2df9514f
Refine code for fixed-length types array ( #28108 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-03 00:40:14 +08:00
dc89730a50
Support collection-level mmap control ( #26901 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-02 23:52:16 +08:00
f8dd589755
Refactor collection's cgo call ( #28055 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-02 13:02:13 +08:00
873b29e226
Fix unstable cpp ut ( #28083 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-02 00:58:16 +08:00
854c0e8a16
fix precision for segcore reduce( #27325 ) ( #28062 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-11-01 21:52:13 +08:00
0c0f012e03
add a custom http header: Accept-Type-Allow-Int64 ( #27901 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-01 11:42:16 +08:00
8ae9c947ae
Use OpenDAL to access object store ( #25642 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-01 09:00:14 +08:00
4fbe3c9142
replace loaded binlog with binlog index for search performance ( #27673 )
...
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-01 02:20:15 +08:00
1b9b8a31a8
Fix QueryNode panic while upgrading ( #28034 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 15:50:13 +08:00
ab6dbf7659
Limit max thread num for pool ( #28018 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 13:00:13 +08:00
2af46d7333
Increase the ChunkManager request timeout ( #28015 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 09:06:13 +08:00
ab6b0103a3
Get vector concurrently ( #27838 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-30 15:44:12 +08:00
1ae6e5d8c8
Update knowhere commit ( #27993 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-10-28 04:24:10 +08:00
f212158d61
Fix delete records timestamp may be reordered ( #27941 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-27 10:24:10 +08:00
4a33391b8f
rename createindex ( #27903 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-10-27 10:12:14 +08:00
a36a57bfec
Enable mmap for IVF index ( #27904 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-26 01:48:12 +08:00
4faba61647
Upgrade conan to 1.61.0 ( #27870 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-10-24 17:50:10 +08:00
8c605ca858
Update knowhere commit ( #27865 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-10-24 02:34:08 +08:00
6060dd7ea8
Add chunk manager request timeout ( #27692 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-23 20:08:08 +08:00
9b0ecbdca7
Support to replicate the mq message ( #27240 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
b6e07d6fe3
Update knowhere commit ( #27812 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-10-20 03:54:09 +08:00
9dd369dd99
Update knowhere version to v2.2.2 ( #27810 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-10-19 21:34:07 +08:00
6b79d2b7d6
Update knowhere commit ( #27752 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-10-18 07:14:09 +08:00
fbaec4c1a2
Fix non-removed 2 GB temporary file in test_utils.cpp ( #27651 )
...
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2023-10-18 02:24:09 +08:00
e98e56f75d
Fix SIGSEGV if run within gdb ( #27736 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-10-18 02:16:11 +08:00
09d8b76048
[restful] new context with grpc metadata ( #27668 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-17 20:00:14 +08:00
5785756d31
Add some log to print size of search result ( #27601 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-10-16 10:48:08 +08:00
c5ea31316d
Fix glog thread-safe bug for standalone mode ( #27703 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-14 12:10:05 +08:00
75343b2cb4
Update knowhere commit ( #27706 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-10-14 03:03:37 +08:00
2efc4ec703
Remove deprecated cmake parameter ( #27695 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-14 02:57:35 +08:00
c882e9c87b
Fix compatibility of Flush ( #27543 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-13 18:01:39 +08:00
d83869aaeb
Refine minio chunks manager ( #27510 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-10-13 14:15:35 +08:00
0f2f4a0a75
Remove useless parameters for Makefile ( #27622 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-10-11 20:45:35 +08:00
7a65b6fb85
Limit faiss ivf index build thread num and fix ut ( #27567 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-10-11 10:33:33 +08:00
8d3069b1db
update openssl to 3.1.2 ( #27399 )
...
deal with root path's normalization
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-08 19:17:31 +08:00
5ba158a3f5
fix knowhere version-changing ( #27508 )
...
Update KNOWHERE_VERSION for the first occurrence
Signed-off-by: Sheldon <chuanfeng.liu@zilliz.com>
2023-10-08 08:35:32 +08:00
dbdb9e15d8
Update Knowhere version ( #27445 )
...
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Li Liu <li.liu@zilliz.com>
2023-09-29 14:23:28 +08:00
106c17f304
Make read ahead policy in ChunkCache configurable ( #27291 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-28 15:47:27 +08:00
b80a3e19d3
Add code for PanicInfo ( #27364 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-27 12:01:28 +08:00
e02228b5ad
Update knowhere commit ( #27357 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-26 18:57:37 +08:00
5db4a0489e
dynamic index version control ( #27335 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
fa033e586a
disable growing index for flat ( #27309 )
...
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-09-22 14:19:24 +08:00
370b6fde58
milvus support multi index engine ( #27178 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
93e2eb78c9
Delete only if primary keys exist ( #25292 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-09-20 19:03:25 +08:00
a362bb1457
Support array datatype ( #26369 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-19 14:23:23 +08:00
fc694bd56d
Update knowhere commit ( #27190 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-19 10:35:22 +08:00
4feb3fa7c6
support azure ( #26398 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-09-19 10:01:23 +08:00
0459a662e4
use MADV_WILLNEED for scalar column data ( #27170 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-18 18:05:22 +08:00
a11136b158
Update knowhere commit ( #27159 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-18 09:41:22 +08:00
060d3563ba
Fix compile error at core/storage ( #27121 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 14:41:21 +08:00
c85c255eb1
Update knowhere commit ( #27109 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-15 13:59:19 +08:00
bb6711f28c
Add ChunkCache: support get vector from storage ( #26142 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00
18cbe26876
fix growing hasRawData ( #27105 )
...
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-09-14 21:11:18 +08:00
0afdfdb9af
Remove other Exceptions, keeps SegcoreError only ( #27017 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-14 14:05:20 +08:00
3d05ddf505
Reduce cpp test time ( #27043 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-13 15:41:18 +08:00
0ff4ddc76c
remove VectorMemNMIndex ( #27000 )
...
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-09-12 17:13:18 +08:00
e107d0794c
support complex delete expression ( #25752 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00
dde3cd2f93
Update knowhere commit ( #26998 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-12 09:49:18 +08:00
3203ce1654
Reduce copy while retrieving primary keys ( #26616 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-09-11 21:31:18 +08:00
c3f15c6b95
Refactor duplicate error class into one place ( #26985 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-11 20:43:17 +08:00
9166011c4a
Add float16 vector ( #25852 )
...
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
56a6559fe7
Update knowhere commit ( #26888 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-07 09:19:16 +08:00
4a2377e669
Fix retrieving may return more result than limit ( #26670 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-06 17:47:15 +08:00
9b3e744b24
Fix embedded milvus log bug ( #26864 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-09-06 14:27:14 +08:00
78a2638fd4
Update knowhere commit ( #26861 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-06 09:21:49 +08:00
c132c53b1a
Update knowhere commit ( #26840 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-05 09:05:49 +08:00
a34a9d606c
fix panic due to empty traceID( #26754 ) ( #26808 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-02 16:13:01 +08:00
09218bfd3d
Optimize loading by reduce 1x copy while reading data ( #26746 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-01 10:15:01 +08:00
b47da91f3c
Update knowhere commit ( #26792 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-09-01 09:23:01 +08:00
8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back ( #26766 )
...
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-08-31 20:07:00 +08:00
9004601817
Enable vector index mmap ( #26750 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-31 17:31:11 +08:00
8330c18dc9
add log for loading segment( #26564 ) ( #26640 )
...
/kind improvement
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-31 12:03:00 +08:00
41d37409e9
Remove unused search conf ( #26689 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-08-31 10:23:07 +08:00
7d5a4b2994
add more event for segcore search( #26277 ) ( #26688 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-30 14:15:01 +08:00
b475f25042
Remove invalid offset check while filling data ( #26666 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-30 09:52:27 +08:00
dc88ef0399
Fix failure while connect kafka with SASL_SSL ( #26617 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-29 13:58:25 +08:00
a73eb1c831
Sync pb files ( #26643 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-29 12:36:26 +08:00
bc6b376c13
Fix arrow compile error ( #26626 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-28 18:04:26 +08:00
1dbe1e63a4
Update knowhere commit ( #26604 )
...
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-08-27 17:04:25 +08:00
ba882b49b6
Optimize query/search on growing segment while output vector field ( #26542 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-24 09:46:24 +08:00
f625e3beb5
Fill trailing \0 for c metrics value ( #26578 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-23 18:42:24 +08:00
d6473028d1
Fix bug for erase while traversing ( #26540 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-08-23 09:34:22 +08:00
e17cda23f4
update knowhere's verison to 2.2.0 ( #26553 )
...
Signed-off-by: Li Liu <li.liu@zilliz.com>
2023-08-23 00:52:21 +08:00
0cb3f88e73
Fix unit test case of thread pool ( #26501 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-21 21:50:22 +08:00
79ca04b995
fix unstable test case for thread pool ( #26465 )
...
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-18 22:42:20 +08:00
Bingyi Sun
gcmutator
foxspy
sammy.huang
Gao
Chun Han
Buqian Zheng
zhagnlu
cai.zhang
Cai Yudong
sre-ci-robot
presburger
Xiaofan
Jiquan Long
congqixia
yah01
Alexander Guzhva
cqy123456
PowderLi
xige-16
yihao.dai
Patrick Weizhi Xu
chyezh
Xu Tong
zhenshan.cao
yah01
Enwei Jiao
SimFG
smellthemoon
Sheldon
aoiasd
jaime
liliu-z