a5be322ab2
enhance: Don't set the timeout for SyncSegments ( #34226 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-02 14:28:07 +08:00
4cf1a358ba
fix: Sync the sealed and flushed segments to datanode ( #34301 )
...
issue: #33696
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-01 22:42:08 +08:00
b32dae1883
enhance: Refine max length exceeded error message ( #34300 )
...
This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 21:16:07 +08:00
e083d99dd9
enhance: Use channel name map finding channel watcher ( #34294 )
...
See also #33235
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 18:46:07 +08:00
734415b8a2
fix: Reduce duplicate PKs in segcore ( #34267 )
...
issue: https://github.com/milvus-io/milvus/issues/34247
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-01 17:42:06 +08:00
b284b81a47
fix: Check partition in current target when observing partition load status ( #34282 )
...
See also #34234
`LoadPartitions` does not guarantee the current target has loading
partitions if there are some partitions already loaded before.
This PR check current target contains the partition to load when
advancing loading percentage to 100.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 17:40:07 +08:00
ef3ced8138
fix: descriptor event in previous version not has nullable to parse error ( #34235 )
...
#34176
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-01 16:38:06 +08:00
b49862d4f3
enhance: Optimize grow slice cost during query ( #34253 )
...
issue: #32252
This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 15:18:11 +08:00
186757e622
enhance: support mark error as user error ( #33498 )
...
relate: https://github.com/milvus-io/milvus/issues/33492
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
14e827dc6c
fix: Implement singleflight for segcore ChunkCache ( #34250 )
...
See also #34249
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 11:46:06 +08:00
f7ecafe77d
enhance: Skip update index for L0 segment ( #34099 )
...
try to update index for l0 segment, will failed by `index not found`
This PR skip update index for l0 segment
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-01 10:26:06 +08:00
0426390f06
enhance: improve check health ( #33800 )
...
issue: #34264
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
e5d691d854
Use new stream segment reader in clustering compaction ( #34232 )
...
#32939
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:26:07 +08:00
73ffc1b424
Refine clustering_compaction_task retry mechanism ( #34194 )
...
#32939
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:22:09 +08:00
3030e4625e
enhance: refactor variable column to reduce memory cost ( #33875 )
...
#33874
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-30 20:16:06 +08:00
cc1bc07bfd
enhance: add log to bitmap index ( #34197 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-30 20:02:06 +08:00
96dcee5dff
fix:load major compaction partial result( #34051 ) ( #34052 )
...
related: #34051
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-28 14:04:05 +08:00
8123bea1ae
enhance: Avoid assign too much segment/channels to new querynode ( #34096 )
...
issue: #34095
When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.
This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-27 19:06:05 +08:00
ad90360162
enhance: Update knowhere commit ( #34223 )
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-06-27 18:20:06 +08:00
d2bc4a53be
enhance: implement rmq and pulsar as wal ( #34046 )
...
issue: #33285
- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-27 15:11:05 +08:00
9c2eeff427
enhance: refine error message for invalid limit parameter ( #34178 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-26 22:14:04 +08:00
51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call ( #34102 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-26 21:13:44 +08:00
948bfba93a
fix: Pass `otlpSecure` config when setup segcore tracing ( #34193 )
...
See also #34192
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-26 19:18:04 +08:00
e04f1f9748
enhance: Add unittest for `storage.DeleteLog` ( #34190 )
...
See also #33787
Backport unit test part in #34188
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-26 17:14:04 +08:00
f8bf6909c6
fix: Fix datacoord's properties overwriten ( #34146 )
...
issue: #34144
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-06-26 11:56:04 +08:00
45203425fd
enhance: Avoid search querynode return nil status in response ( #34100 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-26 11:50:11 +08:00
a933f6731b
fix: centroids file not removed when data skew in major compaction ( #34050 )
...
issue: https://github.com/milvus-io/milvus/issues/30633
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-26 10:48:04 +08:00
fbe3231b1f
fix: fix error ignore in compactor ( #34169 )
...
#34170
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-26 10:24:03 +08:00
cc54ab2e47
[automated] Update Knowhere Commit ( #34172 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-26 02:20:03 +08:00
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
c65f41dc60
fix: Only sync flushed segments to datanode ( #34156 )
...
issue: #33540
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-25 20:46:07 +08:00
f9a0f7bb25
Add an option to enable/disable vector field clustering key ( #34097 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
940a0ac0a3
fix: Prevent import from generating orphaned files ( #34070 )
...
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.
issue: https://github.com/milvus-io/milvus/issues/34068
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 16:58:02 +08:00
962a5446f8
enhance: Add ctx in `SyncTask.Run` to be cancellable ( #34042 )
...
Related to #33716
This PR add context param in SyncTask.Run execution functions to make it
cancellable from the caller.
This make it possible to cancel task when datanode/data sync service is
beeing shut down.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-25 14:22:04 +08:00
d18c49013b
enhance: Refine compaction ( #33982 )
...
issue : https://github.com/milvus-io/milvus/issues/32939
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-25 10:08:03 +08:00
506a915272
fix: Deep copy ImportTask.segmentsInfo to prevent data race ( #34090 )
...
See also #34089
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-25 10:06:02 +08:00
254b555d53
[automated] Update Knowhere Commit ( #34119 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-25 01:22:02 +08:00
07c25a19d9
fix: Make querycoord panick when rg metastore sync fail ( #34106 )
...
See also #34047
When `unassignNode` sync resource group with node removed failed
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 21:38:02 +08:00
fd922d921a
enhance: Add nilness linter and fix some small issues ( #34049 )
...
Add `nilness` for govet linter and fixed some detected issues
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 14:52:03 +08:00
aa36f9feed
fix: [ut] regex query under unsupported index ( #34087 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/29988
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-24 14:08:03 +08:00
d08cb885ca
enhance: enable flush rate limiter of collection level ( #33837 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-24 10:52:03 +08:00
b9237280c2
enhance: wal interface definition ( #33745 )
...
issue: #33285
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-24 10:34:12 +08:00
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
b5c9a7364b
fix: Prevent remove new growing L1 segment when SyncSegments ( #34056 )
...
Related to #34018
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 10:18:01 +08:00
ca7ef26e4b
fix: sync part stats task cannot be finished( #30376 ) ( #34027 )
...
related: #30376
also: refine log output for query_coord task by rephrasing action string
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-24 10:16:02 +08:00
78885a44c4
fix: turn on compression on stream writers ( #34067 )
...
See #31679
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-24 10:08:02 +08:00
03a3f50892
enhance: add skip using array index when some situation ( #33947 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:26:02 +08:00
0d7ea8ec42
enhance: Enhance and correct exception module ( #33705 )
...
#33704
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:22:01 +08:00
6c1d815894
enhance: Remove the unused compaction logic from shard ( #33932 )
...
1. Remove the `compactTo` field in `SegmentInfo`.
2. Remove the target segment not match and its retry logic in
`SyncManager`.
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-23 21:12:01 +08:00
cai.zhang
congqixia
yihao.dai
smellthemoon
wei liu
aoiasd
jaime
wayblink
zhagnlu
Chun Han
Cai Yudong
chyezh
cqy123456
xige-16
Gao
sre-ci-robot
zhenshan.cao
Jiquan Long
Patrick Weizhi Xu
Ted Xu