05df70973c
enhance: retry etcd server start with 3 default retries ( #34208 )
...
Added retry method and unit test cases for retrying etcd server start.
New to open source and Go language. Please reject if this is not the
requirement/ specify the modifications needed in code.
issue : #17569
Signed-off-by: Charles Kakumanu <charles_kakumanu@apple.com>
Co-authored-by: Charles Kakumanu <charles_kakumanu@apple.com>
2024-07-05 20:24:11 +08:00
60be454db0
enhance: add disk quota and max collections into db properties ( #34368 )
...
issue: #34385
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
6b4d977a10
fix: Write padding into mmap file in case of SIGBUS ( #34443 )
...
See also #34442
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-05 17:44:09 +08:00
233b8486ec
fix: generate gin logs failed cause access logs panic ( #34142 )
...
relate: https://github.com/milvus-io/milvus/issues/34141
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-05 16:30:10 +08:00
18f2458385
fix: fix bitmap supported type ( #34350 )
...
#34314
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-05 15:50:10 +08:00
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation ( #34421 )
...
issue: #34420
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00
ba04981a43
enhance: implement wal managerment on streaming node ( #34153 )
...
issue: #33285
- add lifetime control for wal.
- implement distributed-safe wal manager on streaming node.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-05 14:29:42 +08:00
d65b6895d7
[automated] Update Knowhere Commit ( #34432 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-05 01:18:09 +08:00
c16a704934
fix: [restful v2] count(*) & hook ( #34369 )
...
issue: #31224 #34374
for query api:
1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]
add hook about grpc call
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-04 19:38:09 +08:00
439fb24b32
fix: Skip l0 segments when syncing segments to datanodes ( #34388 )
...
See also #34387
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-04 18:02:14 +08:00
fcafdb6d5f
enhance: reconstruct scalar part's code for segment-pruner( #30376 ) ( #34346 )
...
related: #30376
1. support more complex expr
2. add more ut test for unrelated fields
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-04 16:36:09 +08:00
0b404bff22
fix: Correct the update logic of timerecorder ( #34339 )
...
Correct the update logic of timerecorder in the flowgraph to avoid false
failure: "some node(s) haven't received input".
issue: https://github.com/milvus-io/milvus/issues/34337
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-04 16:34:17 +08:00
7611128e57
enhance: wal adaptor implementation ( #34122 )
...
issue: #33285
- add adaptor to implement walimpls into wal interface.
- implement timetick sorted and filtering scanner.
- add test for wal.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-04 15:23:08 +08:00
10b3ce24ba
enhance: use configed max topk for iterator when input topk exceeds ( #34290 )
...
related: #34290
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-04 14:24:19 +08:00
d51d0954bd
enhance: Continue loop when reassign channel fails ( #34331 )
...
Log will be confusing when `Reassign` channel operation failed for both
success & failure log will be printed in row. This PR continue the loop
to avoid this output.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-04 14:20:10 +08:00
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize ( #34201 )
...
See also: #34136
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-04 10:16:17 +08:00
23e2a0e78b
fix: Remove unused code to fix compiling ( #34406 )
...
issue: #34407
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-04 08:48:08 +08:00
d1f57aa4ba
enhance: remove deprecated code within channel manager ( #34340 )
...
issue: https://github.com/milvus-io/milvus/issues/33994
only remove deprecated code, no additional changes.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-03 19:46:09 +08:00
0fd0fcfe1d
enhance: Fix lint issues & sdk testcase ( #34399 )
...
Some lint issue is not detect due to recent static check pipeline issue.
This PR fixes these problem and Go milvusclient testcases.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-03 19:42:10 +08:00
da56880d0f
fix: Avoid datarace in clustering compaction ( #34288 )
...
#34289
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-03 19:08:09 +08:00
a501fa11c0
fix: remove isFull check in compaction.enqueue ( #34338 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-03 18:10:10 +08:00
2e434e4453
fix: Only load or release Flushed segment in datanode meta ( #34390 )
...
issue: #34376 , #34379 , #34375
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 17:52:09 +08:00
feb13cdf07
enhance: Skip pick worker when task doesn't need to execute actually ( #34348 )
...
issue: #34347
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:52:09 +08:00
8165961223
enhance: Optimeize clustering compaction ( #34313 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:14:08 +08:00
fb61344dc9
fix: Revert a optimize in clustering compaction ( #34299 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-03 10:32:08 +08:00
34bec2ea5e
enhance: add metrics for segment prune latnecy( #30376 ) ( #34094 )
...
related: #30376
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-03 10:04:07 +08:00
fa8d641ce6
fix: update check for sparse hnsw index ( #33713 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-02 21:56:09 +08:00
ec5db7a57d
enhance: update knowhere version ( #34361 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-02 19:18:24 +08:00
3563136c2a
enhance: timetick interceptor implementation ( #34238 )
...
issue: #33285
- optimize the message package
- add interceptor package to achieve append operation intercepting.
- add timetick interceptor to attach timetick properties for message.
- add timetick background task to send timetick message.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-02 14:42:08 +08:00
a5be322ab2
enhance: Don't set the timeout for SyncSegments ( #34226 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-02 14:28:07 +08:00
4cf1a358ba
fix: Sync the sealed and flushed segments to datanode ( #34301 )
...
issue: #33696
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-01 22:42:08 +08:00
b32dae1883
enhance: Refine max length exceeded error message ( #34300 )
...
This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 21:16:07 +08:00
e083d99dd9
enhance: Use channel name map finding channel watcher ( #34294 )
...
See also #33235
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 18:46:07 +08:00
734415b8a2
fix: Reduce duplicate PKs in segcore ( #34267 )
...
issue: https://github.com/milvus-io/milvus/issues/34247
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-01 17:42:06 +08:00
b284b81a47
fix: Check partition in current target when observing partition load status ( #34282 )
...
See also #34234
`LoadPartitions` does not guarantee the current target has loading
partitions if there are some partitions already loaded before.
This PR check current target contains the partition to load when
advancing loading percentage to 100.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 17:40:07 +08:00
ef3ced8138
fix: descriptor event in previous version not has nullable to parse error ( #34235 )
...
#34176
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-01 16:38:06 +08:00
b49862d4f3
enhance: Optimize grow slice cost during query ( #34253 )
...
issue: #32252
This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 15:18:11 +08:00
186757e622
enhance: support mark error as user error ( #33498 )
...
relate: https://github.com/milvus-io/milvus/issues/33492
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
14e827dc6c
fix: Implement singleflight for segcore ChunkCache ( #34250 )
...
See also #34249
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 11:46:06 +08:00
f7ecafe77d
enhance: Skip update index for L0 segment ( #34099 )
...
try to update index for l0 segment, will failed by `index not found`
This PR skip update index for l0 segment
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 10:26:06 +08:00
0426390f06
enhance: improve check health ( #33800 )
...
issue: #34264
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
e5d691d854
Use new stream segment reader in clustering compaction ( #34232 )
...
#32939
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:26:07 +08:00
73ffc1b424
Refine clustering_compaction_task retry mechanism ( #34194 )
...
#32939
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:22:09 +08:00
3030e4625e
enhance: refactor variable column to reduce memory cost ( #33875 )
...
#33874
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-30 20:16:06 +08:00
cc1bc07bfd
enhance: add log to bitmap index ( #34197 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-30 20:02:06 +08:00
96dcee5dff
fix:load major compaction partial result( #34051 ) ( #34052 )
...
related: #34051
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-28 14:04:05 +08:00
8123bea1ae
enhance: Avoid assign too much segment/channels to new querynode ( #34096 )
...
issue: #34095
When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.
This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-27 19:06:05 +08:00
ad90360162
enhance: Update knowhere commit ( #34223 )
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-06-27 18:20:06 +08:00
d2bc4a53be
enhance: implement rmq and pulsar as wal ( #34046 )
...
issue: #33285
- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-27 15:11:05 +08:00
9c2eeff427
enhance: refine error message for invalid limit parameter ( #34178 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-26 22:14:04 +08:00
51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call ( #34102 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-26 21:13:44 +08:00
948bfba93a
fix: Pass `otlpSecure` config when setup segcore tracing ( #34193 )
...
See also #34192
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-26 19:18:04 +08:00
e04f1f9748
enhance: Add unittest for `storage.DeleteLog` ( #34190 )
...
See also #33787
Backport unit test part in #34188
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-26 17:14:04 +08:00
f8bf6909c6
fix: Fix datacoord's properties overwriten ( #34146 )
...
issue: #34144
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-06-26 11:56:04 +08:00
45203425fd
enhance: Avoid search querynode return nil status in response ( #34100 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-26 11:50:11 +08:00
a933f6731b
fix: centroids file not removed when data skew in major compaction ( #34050 )
...
issue: https://github.com/milvus-io/milvus/issues/30633
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-26 10:48:04 +08:00
fbe3231b1f
fix: fix error ignore in compactor ( #34169 )
...
#34170
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-26 10:24:03 +08:00
cc54ab2e47
[automated] Update Knowhere Commit ( #34172 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-26 02:20:03 +08:00
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
c65f41dc60
fix: Only sync flushed segments to datanode ( #34156 )
...
issue: #33540
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-25 20:46:07 +08:00
f9a0f7bb25
Add an option to enable/disable vector field clustering key ( #34097 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
940a0ac0a3
fix: Prevent import from generating orphaned files ( #34070 )
...
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.
issue: https://github.com/milvus-io/milvus/issues/34068
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 16:58:02 +08:00
962a5446f8
enhance: Add ctx in `SyncTask.Run` to be cancellable ( #34042 )
...
Related to #33716
This PR add context param in SyncTask.Run execution functions to make it
cancellable from the caller.
This make it possible to cancel task when datanode/data sync service is
beeing shut down.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-25 14:22:04 +08:00
d18c49013b
enhance: Refine compaction ( #33982 )
...
issue : https://github.com/milvus-io/milvus/issues/32939
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-25 10:08:03 +08:00
506a915272
fix: Deep copy ImportTask.segmentsInfo to prevent data race ( #34090 )
...
See also #34089
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-25 10:06:02 +08:00
254b555d53
[automated] Update Knowhere Commit ( #34119 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-25 01:22:02 +08:00
07c25a19d9
fix: Make querycoord panick when rg metastore sync fail ( #34106 )
...
See also #34047
When `unassignNode` sync resource group with node removed failed
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 21:38:02 +08:00
fd922d921a
enhance: Add nilness linter and fix some small issues ( #34049 )
...
Add `nilness` for govet linter and fixed some detected issues
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 14:52:03 +08:00
aa36f9feed
fix: [ut] regex query under unsupported index ( #34087 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/29988
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-24 14:08:03 +08:00
d08cb885ca
enhance: enable flush rate limiter of collection level ( #33837 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-24 10:52:03 +08:00
b9237280c2
enhance: wal interface definition ( #33745 )
...
issue: #33285
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-24 10:34:12 +08:00
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
b5c9a7364b
fix: Prevent remove new growing L1 segment when SyncSegments ( #34056 )
...
Related to #34018
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 10:18:01 +08:00
ca7ef26e4b
fix: sync part stats task cannot be finished( #30376 ) ( #34027 )
...
related: #30376
also: refine log output for query_coord task by rephrasing action string
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-24 10:16:02 +08:00
78885a44c4
fix: turn on compression on stream writers ( #34067 )
...
See #31679
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-24 10:08:02 +08:00
03a3f50892
enhance: add skip using array index when some situation ( #33947 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:26:02 +08:00
0d7ea8ec42
enhance: Enhance and correct exception module ( #33705 )
...
#33704
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:22:01 +08:00
6c1d815894
enhance: Remove the unused compaction logic from shard ( #33932 )
...
1. Remove the `compactTo` field in `SegmentInfo`.
2. Remove the target segment not match and its retry logic in
`SyncManager`.
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-23 21:12:01 +08:00
fb870d2426
fix: Do compressBinlog to fix logID 0 ( #34060 )
...
issue: https://github.com/milvus-io/milvus/issues/34059
Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-22 16:14:01 +08:00
259a682673
enhance: async search and retrieve in cgo ( #33228 )
...
issue: #30926 , #33132
related pr: #33133
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-22 09:38:02 +08:00
380d3f4469
fix: Fix memory buffer error & some renaming ( #33850 )
...
#30633
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-21 17:30:01 +08:00
2f691f1e67
enhance: Unify DeleteLog parsing code ( #34009 )
...
See also #33787
The parsing delete log is distributed in lots of places, which is not
recommended and hard to maintain.
This PR abstract common parsing logic into `DeleteLog.Parse` method to
unify implementation and make it easier to replace json parsing lib.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-21 16:54:01 +08:00
622be36ad7
fix: fix l0 compaction may miss some sealed segments ( #33980 )
...
issue: https://github.com/milvus-io/milvus/issues/33955
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-06-21 16:52:15 +08:00
e653ad27e2
fix: metrics database_num is 0 after restarting rootcoord ( #34011 )
...
issue: #34041
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-21 11:36:00 +08:00
d9f986439d
[automated] Update Knowhere Commit ( #34040 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-21 01:50:01 +08:00
dac20d41c0
fix: fix milvus gpu compile error ( #33995 )
...
NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: https://github.com/milvus-io/milvus/issues/33991
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-06-20 20:30:01 +08:00
04edb07d82
enhance: Add deltaRowCount in l0 compaction ( #33997 )
...
See also: #33998
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-20 17:46:01 +08:00
dc4437ff82
enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk ( #33993 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-20 17:42:02 +08:00
935bc1fb71
fix: Fix GetReplicas API return nil status ( #33715 )
...
issue: #33702
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-20 14:40:15 +08:00
5f02e52561
enhance: Refactor data codec deserialize ( #33923 )
...
#33922
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-06-20 11:17:59 +08:00
31ef0a1fe8
enhance: Add trace for bf cost in l0 compactor ( #33860 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-20 10:10:05 +08:00
6a15b6e2be
fix: Remove loopclosure issue in ChannelManagerImplV2 ( #33989 )
...
See also #33987
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-19 21:25:59 +08:00
2a1356985d
enhance: support null in go payload ( #32296 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-19 17:08:00 +08:00
6d5747cb3e
feat: adding deltalog stream reader and writer ( #33844 )
...
See #31679
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-19 14:42:01 +08:00
7b9462c0d3
enhance: fix copying hits of inverted index twice ( #33968 )
...
issue: https://github.com/milvus-io/milvus/issues/29793
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-19 12:40:01 +08:00
5fac2fa1d2
fix: Panic if ProcessActiveStandBy returns error ( #33369 )
...
#33368
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-19 11:16:00 +08:00
02945959d9
enhance: Avoid to iterate whole segment list for each task's process ( #33943 )
...
when querycoord process segment task, it will try to iterate whole
segment list to checke whether segment is loaded, which cost too much
cpu if there has thousands of segments.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-19 10:19:58 +08:00
298e50b834
enhance: check index with data type ( #33880 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-19 10:15:59 +08:00
0d20303e54
fix: fix binary vector data size ( #33750 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
- fix byte size wrong for binary vectors
- fix the expect/actual error msg
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-18 21:39:59 +08:00
a789c60380
enhance: autoindex for multi data type ( #33868 )
...
issue: #22837
contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
2024-06-18 21:34:01 +08:00
b460862537
fix: can't find Chunk struct after growing support mmap ( #33951 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 18:37:58 +08:00
5cb0760187
fix: Small fixs of major compaction ( #33929 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-18 16:53:58 +08:00
3fdaae8792
fix: Return record with largest timestamp for entires with same PK ( #33936 )
...
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 15:55:59 +08:00
5772123ca2
fix: Check the correct return error in MultiRemove ( #33926 )
...
See also #33925
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 15:23:59 +08:00
5e906fd17f
enhance: Force to reset coord connection for unavailable error ( #33908 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-18 14:53:59 +08:00
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
ec64499536
fix: Check nodeID wildcard when removing pkOracle ( #33895 )
...
See also #33894
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 14:11:58 +08:00
8cdc0e6233
fix: fix data codec writer close ( #33818 )
...
issue:#33813
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-06-18 13:59:57 +08:00
e83ecd5074
enhance: fix not find arrow::parquet lib ( #33928 )
...
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-06-18 00:22:01 +08:00
2a04b0929a
fix: Prevent use captured iteration variable partitionID ( #33906 )
...
See also #33902
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-17 19:11:59 +08:00
f993b2913b
enhance: Reserve space of payload writer when serialize data ( #33817 )
...
See also #33561 #33562
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-17 12:06:04 +08:00
188ee433b9
enhance: Refine name rule check error msg ( #33815 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 10:37:58 +08:00
95148866ed
fix: Don't remove growing L0 segment in datanode metacache ( #33829 )
...
issue: #33540
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-17 10:09:57 +08:00
1a9ab52f66
enhance: Ensure the idempotency of compaction task ( #33872 )
...
/kind enhancement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-16 22:09:57 +08:00
8537f3daeb
enhance: Rename Compaction to CompactionV2 ( #33858 )
...
Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.
issue: https://github.com/milvus-io/milvus/issues/32809
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-16 22:07:57 +08:00
d43ec4db0b
enhance: support array bitmap index ( #33527 )
...
#32900
---------
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-16 21:51:58 +08:00
e422168f09
fix: readd timestamp index because segment timestamp not ordered ( #33856 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-16 21:48:01 +08:00
f67b6dc2b0
fix: DeleteData merge wrong data casuing data loss ( #33820 )
...
See also: #33819
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-14 17:57:56 +08:00
9b69601dfb
fix: load operation when segment is on releasing ( #31340 )
...
issue: #30857
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-14 15:35:56 +08:00
86a36b105a
enhance: Tidy compaction executor ( #33778 )
...
Move compaction executor to compaction pacakge.
issue: https://github.com/milvus-io/milvus/issues/32451
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-14 14:34:01 +08:00
0ecd694305
enhance: legacy code clean up ( #33838 )
...
issue: #33839
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-06-14 14:25:56 +08:00
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33792 )
...
issue: #33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-14 11:37:56 +08:00
b69e9093c8
fix: Fallback field type when it isn't in request ( #33832 )
...
issue: #33432
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-14 09:55:56 +08:00
08fcf3f62b
fix: Fix meta prefix overlap bug ( #33830 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-13 19:27:57 +08:00
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost ( #33611 )
...
issue:#33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 17:57:56 +08:00
2dfa752527
doc: [skip-e2e] add comments for event manager ( #33444 )
...
Add comments for event manager(internal/datanode/event_manager.go).
Signed-off-by: ArenaSu <704427617@qq.com>
2024-06-13 17:56:06 +08:00
db59249776
doc: [skip-e2e] add comments for meta table ( #33403 )
...
Add commnets for meta table in internal/rootcoord/meta_table.go.
Signed-off-by: ArenaSu <704427617@qq.com>
2024-06-13 17:54:09 +08:00
7d74edd6dd
fix: update clang-tidy and clang-format from 10 to 12 ( #33141 )
...
Default llvm toolchain version in Ubuntu 20.04 is 10, while Ubuntu 22.04
does not have `clang-tidy-10` or `clang-format-10` by default.
issue: #33142
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-06-13 15:27:58 +08:00
512ea6be5f
enhance: Avoid merging insert data when buffering insert msgs ( #33562 )
...
See also #33561
This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-13 11:15:56 +08:00
9ab3058da2
fix: Prevent restart timetick sender creating ut datanode ( #33790 )
...
See also #33789
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-12 22:29:58 +08:00
b39dfc25dc
enhance: Use fastjson lib for unmarshal delete log ( #33787 )
...
```
goos: linux
goarch: amd64
GOMAXPROC=1
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkJsonSerdeStd 343872 3568 ns/op 1335 B/op 25 allocs/op
BenchmarkJsonSerdeFastjson 5124177 234.9 ns/op 16 B/op 1 allocs/op
```
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-12 20:41:57 +08:00
b3ee1a4bd5
fix: lost DroppedAt field when DropCollection ( #33766 )
...
issue: #33097
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-12 20:21:56 +08:00
9a3e4080f1
enhance: Add comment for channel cp updater ( #33759 )
...
/kind enhancement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-12 20:01:55 +08:00
9d4535ce0b
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector ( #33760 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-06-12 17:17:55 +08:00
47b04ea167
enhance: support sparse cardinal hnsw index ( #33656 )
...
issue: #29419
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-12 16:57:55 +08:00
8cb350598c
enhance: Improve GetVectorById of Sparse Float Vector ( #33209 )
...
issue: #29419
* sparse float vector to support raw data mmap
For get vector from chunk cache, I added a unit test but marking it as
skipped due to a known issue. I have tested it locally.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-12 10:09:55 +08:00
6b9901c59f
enhance: add a semaphore for CompactionExecutor ( #33558 )
...
#33182
---------
Signed-off-by: coldWater <254244460@qq.com>
2024-06-11 17:25:55 +08:00
f7af323d1e
fix: sync partitiion stats blocking balance task( #33741 ) ( #33742 )
...
related: #33741
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-11 14:21:56 +08:00
ecf2bcee42
enhance: speed up array-equal operator via inverted index ( #33633 )
...
fix : #33632
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-11 14:13:54 +08:00
2b7ee1968f
enhance: new messsage interface for log service ( #33286 )
...
issue: #33285
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-11 10:38:01 +08:00
8ca5ced821
fix: async warmup will be blocked by state lock ( #33686 )
...
issue: #33685
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-10 21:59:53 +08:00
b1d46eb34b
fix: Fix multiple vector fields import ( #33723 )
...
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.
issue: https://github.com/milvus-io/milvus/issues/33681 ,
https://github.com/milvus-io/milvus/issues/33682
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-10 21:57:54 +08:00
eb5d4de390
fix: Check if the import job exists ( #33672 )
...
issue: https://github.com/milvus-io/milvus/issues/33671
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-10 21:51:55 +08:00
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
f53ab54c5d
enhance: async cgo utility ( #33133 )
...
issue: #30926 , #33132
- implement future-based cgo utility.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-09 22:55:53 +08:00
3540eee977
enhance: Support L0 import ( #33514 )
...
issue: https://github.com/milvus-io/milvus/issues/33157
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
6ce9df913f
fix: clean vector memory ( #33692 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-07 10:37:54 +08:00
0a3d456688
enhance: add restful api to trigger component stop ( #32076 )
...
issue: #32698
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-07 10:35:54 +08:00
c61fb1eff5
enhance: do check when add not empty logpath ( #33640 )
...
meta only store logid
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-07 10:19:51 +08:00
charleskakumanu
jaime
congqixia
aoiasd
zhagnlu
chyezh
sre-ci-robot
PowderLi
Chun Han
yihao.dai
XuanYang-cn
cai.zhang
wayblink
Buqian Zheng
Gao
smellthemoon
wei liu
Cai Yudong
cqy123456
xige-16
zhenshan.cao
Jiquan Long
Patrick Weizhi Xu
Ted Xu
Bingyi Sun
presburger
shaoting-huang
ArenaSu
Yinzuo Jiang
coldWater